Response format mis-match between OpenSocial REST and RPC specs

[goosemanjack]

There is a mismatch between the REST and RPC specs with regards to
response formats.

The REST spec treats all responses identically, wrapping all in a
Response JSON object as defined under section 3.1 of the spec.

http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/REST-API.html#rfc.section.3.1

The actual data, be it a single object or a list of objects always is
placed under the "entry" label.

(REST data format)
{
"startIndex" : 1,
"itemsPerPage" : 10,
"totalResults" : 100,
"entry" : [
{...first item...},
{...second item...}
...
]
}


The RPC spec, in contrast, treats single items and lists differently.
Single elements have their data placed under the "result" label, while
multiple-item responses have their results placed under the label
"list" as a child of "result".

http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RPC-Protocol.html#rfc.section.3.1

http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RPC-Protocol.html#rfc.section.6


(RPC data format for single items)
{
"id" : "myself"
"result" : {
"id" : "example.org:34KJDCSKJN2HHF0DW20394",
"name" : { "unstructured" : "Jane Doe"},
"gender" : "female"
}
}

(RPC data format for list items)
{
"result" : {
"list" : [ ... ]
}
}


I would propose we align the response format for OpenSocial methods in
the RPC spec with that of the REST spec. The REST spec is more
consistent across endpoints and treatment of data.

[Lane LiaBraaten]

I'm unclear on the actual change you're proposing, as RPC treats all responses identically as well.  It takes the actual data, be it a single object or a list of objects, returned and wraps it in a JSON-RPC Response object [1], which has an 'id' and 'result' property.

If the response is a collection, extra information (startIndex, totalResults, etc.) is returned in either case.

REST Response to request for single person
HTTP/1.x 200 OK
<OpenSocial-Person>

RPC Response to request for single person
HTTP/1.x 207 Multi-Status
{
   "id" : "myself"
   "result" : <OpenSocial-Person>
}

REST Response to request for list of people
HTTP/1.x 200 OK
<OpenSocial-Collection<OpenSocial-Person>>

RPC Response to request for single person
HTTP/1.x 207 Multi-Status
{
   "id" : "myfriends"
   "result" : <OpenSocial-Collection<OpenSocial-Person>>
}

Digging a bit deeper, there is an inconsistency in the OpenSocial Collection object.
In REST it is defined as [2]:

{

   "startIndex" : 1,
   "itemsPerPage" : 10,
   "totalResults" : 100,


   "entry" : [
      {...first item...},
      {...second item...}
      ...
   ]
}

while in RPC it is defined as [3]

{
    "totalResults" : <total number of elements in the full list>,


    "startIndex" : <0 based offset in the full-list of the returned sublist>,
    "itemsPerPage" : <total number of elements in the returned sublist>,
    "isFiltered" : <boolean indicating if the result honors filter params in the request>,


    "isUpdatedSince" : <boolean indicating if the result honors the updatedSince param in the request>,
    "itemsPerPage" : <total number of elements in the returned sublist>,
    "list" : [ ... the sublist ...]


}

In REST we call it "entry", in RPC we call it "list". 

I think "entry" is more appropriate based on the existing XML <--> JSON mapping rules (RPC spec's Overview section [4] has a more detailed version than REST for this point):

Plural fields are encoded as arrays in JSON and repeated tags in XML, e.g. "fields": [ "value1", "value2" ] and<fields>value1</field><fields>value2</field> respectively.

(n.b. the typo: <fields></field> (open tag is plural, close tag is singular))

Otherwise we'd end up with { list: ["foo", "bar"] } being equivalent to

<list>foo</list>
<list>bar</list>

-Lane

[1] http://groups.google.com/group/json-rpc/web/json-rpc-1-2-proposal
[2] http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/REST-API.html#rfc.section.3.1
[3] http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RPC-Protocol.html#rfc.section.6
[4] http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RPC-Protocol.html#rfc.section.2

[goosemanjack]

It's actually a simple change since 75% of the data formats are in
alignment between REST and RPC. It comes down to the use of
"entry" (REST) version the use of "result" (RPC-single) and
"result.list" (RPC-list). If we can fold the formats where content is
always under "entry" or always under "result"/"result.list" then lots
of problems with special-case handling within the EL of data coming
from REST vs. data coming from RPC go away. My assumption is that
this misalignment is simple a matter of drift between the specs and
not an intentional incompatibility.

Proposal is to modify result format in RPC spec to place both singular
and list data under the "entry" label in the response object.
--
clc

> [2]http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RES...
> [3]http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RPC...
> [4]http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RPC...

>
> On Wed, Oct 7, 2009 at 4:48 PM, goosemanjack <cc...@myspace.com> wrote:
>
> > There is a mismatch between the REST and RPC specs with regards to
> > response formats.
>
> > The REST spec treats all responses identically, wrapping all in a
> > Response JSON object as defined under section 3.1 of the spec.
>

> >http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RES...

>
> > The actual data, be it a single object or a list of objects always is
> > placed under the "entry" label.
>
> > (REST data format)
> > {
> >   "startIndex" : 1,
> >   "itemsPerPage" : 10,
> >   "totalResults" : 100,
> >   "entry" : [
> >      {...first item...},
> >      {...second item...}
> >      ...
> >   ]
> > }
>
> > The RPC spec, in contrast, treats single items and lists differently.
> > Single elements have their data placed under the "result" label, while
> > multiple-item responses have their results placed under the label
> > "list" as a child of "result".
>

> >http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RPC...
>
> >http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RPC...

[Louis Ryan]

The actual data, be it a single object or a list of objects always is 

placed under the "entry" label. 

Not exactly. When fetching values known a priori to be singular "entry" is not used, e.g. /people/@me/@self. Only results that could be plural use "entry". As Lane mentions the "result" object in JSON-RPC is part of its standard enveloping and can be ignored for the purposes of this discussion

This is a great example of how XML/Atom-first thinking in API design makes our JSON and by implication osapi ugly. Use of a singular name like "entry" makes sense in XML because you represent collections by repeating that element. In a syntax like JSON and in most programming languages you use a plural name like "items" for collections because you have real plural types. Considering that the OSAPI syntax is 1:1 with the JSON-RPC un-enveloped value were now talking about converting from

osapiresult.items[1].emails[0]

to

osapiresult.entry[1].emails[0]

Note that for other collections in our API like emails & ims we use plural names which are more natural. For the sake of osapi I'd rather see POCO adopt a more sensible name in their JSON schema, "entries"?, than make this change. I do see the need for making the names align between REST and RPC so I agree we should fix this I just wish we could use a better name.

[goosemanjack]

I agree that it's ugly for singular items like the @people/@me/@self
example, but the MySpace 0.9 REST implementation chose to use the
wrapper object for every response, regardless of singular or multiple
results. We also are using the "Entry" object to bridge the divide
between strongly typed languages and loosely typed languages in the
implementation.

We have ended up with quite a bit of ugliness syntactically, but I'm
not sure the best approach to fixing it. The label "result" is
actually better than "entry" from my standpoint, but we're now at a
place where we have the current 0.9 REST spec out in production and
being used by some third parties. We would have to follow a
deprecation plan to change the REST side of things, which is the
reason I proposed using "entry" as the label of choice. We're open to
an alternative syntax, but it will require a deprecation schedule at
this point.

[Evan Gilbert]

On Mon, Oct 12, 2009 at 11:15 AM, goosemanjack <cc...@myspace.com> wrote:

I agree that it's ugly for singular items like the @people/@me/@self
example, but the MySpace 0.9 REST implementation chose to use the
wrapper object for every response, regardless of singular or multiple
results.  We also are using the "Entry" object to bridge the divide
between strongly typed languages and loosely typed languages in the
implementation.

We have ended up with quite a bit of ugliness syntactically, but I'm
not sure the best approach to fixing it.  The label "result" is
actually better than "entry" from my standpoint, but we're now at a
place where we have the current 0.9 REST spec out in production and
being used by some third parties.  We would have to follow a
deprecation plan to change the REST side of things, which is the
reason I proposed using "entry" as the label of choice.  We're open to
an alternative syntax, but it will require a deprecation schedule at
this point.

Should add version to the format param string? I.e. http://opensocialserver/api/people?format=json09 or format=json-s (for shindig)?

In general, we're going to need these for any backwards incompatible changes. Adding would also make it possible to address the existing slight incompatibilities in implementations - for example, it would be trivial for Shindig to add support for what MySpace has in production.

 

On Oct 12, 10:20 am, Louis Ryan <lr...@google.com> wrote:
> > The actual data, be it a single object or a list of objects always is
>
> placed under the "entry" label.
>
> Not exactly. When fetching values known a priori to be singular "entry" is
> not used, e.g. /people/@me/@self. Only results that could be plural use
> "entry". As Lane mentions the "result" object in JSON-RPC is part of its
> standard enveloping and can be ignored for the purposes of this discussion
>
> This is a great example of how XML/Atom-first thinking in API design makes
> our JSON and by implication osapi ugly. Use of a singular name like "entry"
> makes sense in XML because you represent collections by repeating that
> element. In a syntax like JSON and in most programming languages you use a
> plural name like "items" for collections because you have real plural types.
> Considering that the OSAPI syntax is 1:1 with the JSON-RPC un-enveloped
> value were now talking about converting from
>
> osapiresult.items[1].emails[0]
>
> to
>
> osapiresult.entry[1].emails[0]
>
> Note that for other collections in our API like emails & ims we use plural
> names which are more natural. For the sake of osapi I'd rather see POCO
> adopt a more sensible name in their JSON schema, "entries"?, than make this
> change. I do see the need for making the names align between REST and RPC so
> I agree we should fix this I just wish we could use a better name.

--

[goosemanjack]

Version is currently explicit in our REST path, so we would not need
it as a parameter. We will have a different endpoint for each version
instead of serving multiple versions from the same endpoint.

ex:

http://opensocial.msappspace.com/roa/09/people/@me/@friends

On Oct 12, 12:41 pm, Evan Gilbert <uid...@google.com> wrote:
> On Mon, Oct 12, 2009 at 11:15 AM, goosemanjack <cc...@myspace.com> wrote:
>
> > I agree that it's ugly for singular items like the @people/@me/@self
> > example, but the MySpace 0.9 REST implementation chose to use the
> > wrapper object for every response, regardless of singular or multiple
> > results.  We also are using the "Entry" object to bridge the divide
> > between strongly typed languages and loosely typed languages in the
> > implementation.
>
> > We have ended up with quite a bit of ugliness syntactically, but I'm
> > not sure the best approach to fixing it.  The label "result" is
> > actually better than "entry" from my standpoint, but we're now at a
> > place where we have the current 0.9 REST spec out in production and
> > being used by some third parties.  We would have to follow a
> > deprecation plan to change the REST side of things, which is the
> > reason I proposed using "entry" as the label of choice.  We're open to
> > an alternative syntax, but it will require a deprecation schedule at
> > this point.
>

> Should add version to the format param string? I.e.http://opensocialserver/api/people?format=json09or format=json-s (for

[Louis Ryan]

working through the different flavors...

So today for MySpace REST we have

Singular -> { entry : {...} }
Plural -> { entry : [...] }

For Shindig REST we have

Singular -> {...}
Plural -> { entry : [...] }

and for JSON-RPC we have

Singular -> { result : {...} }
Plural -> { result : { items : [] }}

Options are :
1a - MySpace to unwrap "entry" for singular case to align with Shindig REST
1b - Shindig to wrap with "entry" for the singular case to align with MySpace REST

For JSON-RPC the "result" label is just part of standard enveloping so its really just

Singular -> { ... }
Plural -> { items : [...] }


Options are
2a Wrap the singular case in "entry".i.e   {result : { entry : {...} } }
2b Rename "items" to "entry"
2c Both of the above

Unsurprisingly I would go for 1a & 2b. [ I really hate the "entry" wrapper for the singular case ]

On Mon, Oct 12, 2009 at 11:15 AM, goosemanjack <cc...@myspace.com> wrote:

I agree that it's ugly for singular items like the @people/@me/@self
example, but the MySpace 0.9 REST implementation chose to use the
wrapper object for every response, regardless of singular or multiple
results.  

 

We also are using the "Entry" object to bridge the divide
between strongly typed languages and loosely typed languages in the
implementation.

Not sure what you mean by this at least for the singular case. Currently "entry" provides a polymorphic reference to either an array or an object.
 

[goosemanjack]

It appears that in this case Shindig has branched from the spec on
REST format:
(from http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/REST-API.html#rfc.section.3.1
)

application/json representation:

{
"startIndex" : 1,
"itemsPerPage" : 10,
"totalResults" : 100,
"entry" : [
{...first item...},
{...second item...}
...
]
}

or, for only one item:

{
"startIndex" : 1,
"itemsPerPage" : 10,
"totalResults" : 100,

"entry" : {...only item...}
}


We're still conferring with the group doing our REST and other back-
end APIs to get their feedback on the path forward. I'll update the
thread as we have more information.
--
clc

[Chris Chabot]

Not quite, the key difference is that the entry is unwrapped when the result is singular by nature (ie retrieving the owner), however anything that could contain 0 to many entries is wrapped under the entry key, so in this case there is only one item, in a result set that could contain more then one items.

[goosemanjack]

Unfortunately, we've run out of time to come up with a proposal that
addresses all the issues in aligning REST and RPC response structure.
We're going to abandon this proposal and maybe pick it up again next
cycle.
--
clc

On Oct 14, 2:29 pm, Chris Chabot <chab...@google.com> wrote:
> Not quite, the key difference is that the entry is unwrapped when the result
> is singular by nature (ie retrieving the owner), however anything that could
> contain 0 to many entries is wrapped under the entry key, so in this case
> there is only one item, in a result set that could contain more then one
> items.
>
> On Wed, Oct 14, 2009 at 7:25 PM, goosemanjack <cc...@myspace.com> wrote:
>
> > It appears that in this case Shindig has branched from the spec on
> > REST format:
> > (from

> >http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RES...