A modest proposal for an OpenSocial RESTful API

50 views
Skip to first unread message

John Panzer

unread,
Feb 27, 2008, 3:31:32 AM2/27/08
to opensocial-an...@googlegroups.com
All,

I'd like to propose a starting point for an OpenSocial RESTful API specification.  The draft below is the result of a lot of individual discussions, a bunch of research and experiments, and, over the past couple of days, a whole lot of coffee.  Having said that, I regard this as a way to spark discussion rather than something being put forward for approval.  Please reply to the list with your opinions, reactions, and alternatives. 

One suggestion:  There are a lot of pieces to this (still under 10 pages though).  If you start a response thread, it could be useful to edit the subject line to advertise what you're discussing -- the section name, for example.  Otherwise discussions could easily be missed.

Have fun,
John

---

Proposal for an  OpenSocial RESTful API (DRAFT)

The primary goal of this proposal is to serve as a common protocol understood by all OpenSocial clients and servers.  If this draft is adopted, it would replace the previously proposed GData-based OpenSocial data API.

Overview

This API defines a language- and platform- neutral protocol for OpenSocial clients and container servers to interact outside of gadgets on a web page.  As a protocol, it is intended to be reasonably easy to implement in any language and on any platform.  It should also be usable across a range of clients, from gadgets operating within a web page to servers communicating to synchronize data about a user.

The protocol operates primarily in terms of resources and operations on them.  It is defined on top of the HTTP protocol, and uses the standard HTTP methods (GET, POST, PUT, DELETE, etc.) to retrieve and change server state.

No single data representation is ideal for every client.  This protocol defines dual representations for each resource in two widely supported representations, JSON [RFC4627] and Atom [RFC4287][RFC5023], using a set of generic mapping rules.  The mapping rules allow a server to write to a single interface rather than implementing the protocol twice.

OpenSocial container servers are free to define additional representations but must support at least the JSON and AtomPub profiles defined in this document.

The protocol defines Activity, People, and AppData resources and collections.  Most operations consist of retrieving (GET), updating (PUT), creating (POST or PUT), or destroying (DELETE) these resources.  The protocol provides a optional feature to batch multiple requests together in a pipeline to avoid multiple HTTP round trips.  Finally, it specifies an optional partial update feature which avoids sending large resources over the wire to update just one field.

Data Representations

Each resource has a dual representation as JSON and Atom (XML).  All data must be representable in both formats, but we do not attempt to map from generic XML or Atom to JSON.  Instead, we define an internal data model using English and JSON syntax, and then define the mappings between this and Atom/JSON.

Each activity, person, and appdata is represented as a hierarchical tree of elements.  Ordering of elements within a parent element may or may not be significant, depending on the context.  Mapping consists of converting from the internal hierarchy to a JSON data structure or an Atom entry.  Examples of the three primary types of data follow:

A Person Example

application/json representation:

{

  'id' : 'orkut.com:34KJDCSKJN2HHF0DW20394',
  'name' : {'unstructured' : 'Jane Doe'}

}

application/atom+xml representation:
<entry xmlns="http://www.w3.org/2005/Atom>
  <content type="text/xml">

    <name>
      <unstructured>Jane Doe</unstructured>
    <name>
  </content>

  <title/>
  <updated>2003-12-13T18:30:02Z</updated>
  <author/>
 
<id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
</entry>
Person mapping rules:
atom:entry/content == *                         Flatten entry/content into top level of JSON representation.  In the inverse mapping, all otherwise unknown JSON fields are collected into atom:entry/content.
atom:entry/id == id  (type:guid)               Map ids, using a urn:guid: prefix for the Atom representation
/* atom:entry/title == name.unstructured */ This would map atom:title field to the unstructured name; optional but useful -John Panzer 2/26/08 1:56 PM 
atom:entry/updated == updated               Map updated field; date/time representation is the same in both.

In this example, we have chosen to flatten the "content" element by moving all its child elements to the top level. If there were a conflict, the rules could include a renaming for conflicting fields. This renaming scheme, as well as the choice to flatten "content", is driven by metadata defined above for the "Person" service.

(In general clients can ask for a full representation of a resource, with all fields and sub-fields, or a partial representation, using query parameters.)

An Activity Example

Activity application/json:

{

  'id' : 'http://example.org/activities/example.org:87ead8dead6beef/self/af3778',

  'title' : { 'type' : 'html',

              'value' : '&lt;a href="foo"&gt;some activity&lt;/a&gt;'

           }

  'updated' : '2008-02-20T23:35:37.266Z',

  'content' : 'Some details for some activity',
  'received' : '
2008-02-20T23:35:37.266Z'

}


Activity application/atom+xml:
<entry xmlns="http://www.w3.org/2005/Atom>
 
<category term="status"/>
  <id>http://example.org/activities/example.org:87ead8dead6beef/self/af3778</id>
  <title type="html">&lt;a href="foo"&gt;some activity&lt;/a&gt;</title>
  <content>Some details for some activity</content>
  <updated>2008-02-20T23:35:37.266Z</updated>
 
<link rel="self" type="application/atom+xml" href="http://www.google.com/activity/feeds/.../af3778"/>

  <osa:received>2008-02-20T23:35:37.266Z</osa:received>
</entry>
Activity mapping rules:
atom:entry/content == content (type:text)       Map entry/content to content field in JSON representation, and vice versa.
atom:entry/summary == summary (type:text)  Same for summary.
atom:entry/id == id                                         Map ids (representation is the same)
atom:entry/title == title (type:text)                  Map titles over; if not plain text, use a structured type/value object to represent data.
atom:entry/updated == updated                      S.O.P.
osa:received == received                                Map received field, stripping the namespace.  In the JSON to Atom conversion, received always means osa:received.

An AppData Example

The first example is of a collection of key/value pairs for a particular application/user pair:

application/json representation:

{

  'pokes' : 3,

  'last_poke' : '2008-02-13T18:30:02Z'
}

application/atom+xml representation:
<entry xmlns="http://www.w3.org/2005/Atom>
  <content type="text/xml">

    <pokes>3</poke>
    <last_poke>
2008-02-13T18:30:02Z</last_poke>
  </content>
  <title/>
  <updated>2003-12-13T18:30:02Z</updated>
  <author><url>
urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</url></author>
  <id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
</entry>
AppData mapping rules:
atom:entry/content == *                         Flatten entry/content into top level of JSON representation.  In the inverse mapping, all JSON fields are collected into atom:entry/content.

An AppData Collection Example

In this example, a client has requested a collection of data that spans multiple users.  The result is a collection which is given a special default JSON representation as a mapping from users to their data.

application/json representation:

{

  'orkut.com:34KJDCSKJN2HHF0DW20394' : {'pokes' : 3, 'last_poke' : '2008-02-13T18:30:02Z' },

  'orkut.com:58UIDCSIOP233FDKK3HD44' : {'pokes' : 2, 'last_poke' : '2007-12-16T18:30:02Z' }
}

application/atom+xml representation:
<feed xmlns="http://www.w3.org/2005/Atom>
  <id>...</id>
  <title>...</title>
  <entry>
    <content type="text/xml">

      <pokes>3</poke>
      <last_poke>
2008-02-13T18:30:02Z</last_poke>
    </content>
    <title/>
    <updated>
2008-02-13T18:30:02Z</updated>
    <author><url>
urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</url></author>
    <id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
  </entry>
  <entry>
    <content type="text/xml">
      <pokes>2</poke>
      <last_poke>
2007-12-16T18:30:02Z</last_poke>
    </content>
    <title/>
    <updated>
2007-12-16T18:30:02Z</updated>
    <author><url>
uurn:guid:orkut.com:58UIDCSIOP233FDKK3HD44</url></author>
    <id>urn:guid:orkut.com:58UIDCSIOP233FDKK3HD44</id>  
  </entry>
</entry>
AppData mapping rules:
        atom:feed/entry/author/url (type:index guid)                 Use the author/url field as the unique key for the JSON collection; on inverse mapping, take keys and store in author/url, using a guid mapping that adds urn:guid: prefix
atom:entry/content == *                                           Flatten entry/content into each record in resulting set.

The default representation of a collection would simply be a JSON list of records ([ {...}, {....}, ...]).  The special type "index" means that the collection is instead to be represented as a mapping, using the index field.

Collections

Collections are a useful abstraction for dealing generically with multiple things.  They again have both Atom and JSON representations; the Atom representation is simply a feed whose entries are as specified above.  The default JSON representation is a JSON list of JSON objects, with the individual objects as specified above.

Activity stream collections are ordered chronologically by default, most recent activities first.  Other orderings can be requested for collections using query parameters.

Operations

OpenSocial uses standard HTTP methods: GET to retrieve, PUT to update in place, POST to create new, and DELETE to remove.  POST is special; it operates on collections and creates new activities, persons, or app data within those collections, and returns the base URL for the created resource in the Location: header.

Discovery

This proposal specifies URL template rules for discovering collections; the expectation is that all containers can support the same URL paths and that the rules for composing URLs are simple enough to avoid the need for generic discovery mechanisms in the basic protocol.  Should this prove impossible, more flexible discovery mechanisms can be added.

Activities

URL patterns:
/activities/{uid}/self         -- Collection of activities for given user ({uid}=me is a placeholder for currently logged in user)
/activities/{uid}/friends     -- Collection of activities for friends of the given user {uid}
/activities/{uid}/{groupid} -- Collection of activities for people in group {groupid} belonging to given user {uid}
/activities/{uid}/self/{aid} -- Individual activity resource; usually discovered from collection

Modifiers:
?app={aid}                   -- Restrict the collection to a given app's activities ("me" is a placeholder for the app making the current request)

People

URL patterns:
/people/{uid}/all                        -- Collection of all people connected to user {uid}
/people/{uid}/friends                  -- Collection of all friends of user {uid}; subset of all
/people/{uid}/{groupid}              -- Collection of all people connected to user {uid} in group {groupid}
/people/{uid}/all/{pid}                -- Individual person record.
/people/{uid}/self                       -- Self Profile record for user {uid}


/groups/{uid}/self                       -- Collection of groups owned by the user, which always contains 'all' and 'friends' and may contain more.  (Details TBD)

App Data

URL patterns:
/appdata/{uid}/self/{aid}                    -- Individual App Data record for a given user+app, consisting primarily of a bag of key/value pairs.
/appdata/{uid}/friends/{aid}                -- Collection of App Data records for friends of {uid}.

Additional Standard Query Parameters

These query parameters may be used with any URL above. 

format={format}                                      -- Format desired; one of (atom, json-c); default is json-c.
fields={field+}                                         -- List of fields to include in representation or in the members of a collection. 
startPage={startPage}                               -- Index into a paged collection
count={count}                                          -- Set page size for paged collection

Authentication and Authorization

Requests use OAuth [OAuth.net] for authentication and authorization.  The container is the OAuth Service Provider and the client is the OAuth Consumer.  Clients can use and pass in a full user-level authorization token, using the full OAuth protocol.  Alternatively, if a client has a trust relationship with a container, it can use "two legged" OAuth authorization in which there is no user-based token.  (A "two legged" use profile is being defined for within the OAuth working group.)

In each of these cases, the URL and auth[nz] data is signed by the OAuth protocol.  Additional security can be obtained with SSL connections.

Clients may supply an OpenSocial-specific OAuth extension parameter, x-oauth-os-uid, to identify the current user (the user who the client is acting on behalf of).  The format of this parameter is the urn: GUID of the current user, as supplied by the container to the client in an out of band manner.

Concurrency Control (Optional)

All resources use the HTTP/AtomPub ETag-based optimistic concurrency mechanism.  Servers provide ETags for resources when they are retrieved, and clients supply If-Match: headers with the etag when they perform modifying operations.  If the resource has been modified between the initial retrieval and the update, the update fails with a conflict HTTP code.


A server which chooses not to support optimistic concurrency should omit ETags on its responses. 

Batching (Optional)

There are use cases where clients want to send several operations, in a specified sequence, to a server to execute in turn, but do not want to incur the overhead of multiple HTTP round trips.  A specific use case is a gadget which first adds a new friend, then retrieves the 5th page of friends (what it's currently viewing).  These are two ad hoc requests from the point of view of the server involved, but for optimization purposes the gadget wants to batch up the requests and responses to minimize latency.  It is also possible that the requests and responses would be to different types of data (activities, people, or persistence) and/or to different servers.  In the most general case it would be useful to be able to batch any type of request, including requests to third party servers (e.g., "phone home" requests) that need to be executed in sequence with container-local requests.

The semantics of a set of batched requests in this proposal are exactly the same as sending the same requests individually.  In particular, the server does not provide any atomicity guarantees.  The server is not allowed to reorder or parallelize the requests unless it can do so without client-visible side effects.  The simplest server implementation is to loop over the requests, executing each in turn, and appending responses into a batch response to send to the client.

The batch interface is an optional part of OpenSocial.  If a server does not support it, a client falls back to using individual requests and responses with no changes in semantics.  Depending on the exact situation, it may be more efficient to use individual requests which can be parallelized and cached, or to use batch serial requests; a goal with this API is to make it simple for a client to switch between the two mechanisms and to keep the semantics consistent and straightforward.

To this end, a batch request is represented by an envelope HTTP request sent to a standard proxy resource (/batch_proxy, in this proposal).  The client POSTs the document described below to the proxy URL, which returns 200 OK when the contents have been processed (successfully or not) and returns a response sequence document.

Both request and response sequence documents are represented by standard multipart/mixed documents.  The request:

POST /batch_proxy HTTP/1.1
Host: container.org
Content-Type: multipart/mixed; boundary=batch-a73hdj3dy3mm347ddjjdf

-batch-a73hdj3dy3mm347ddjjdf
X-Batch-Operation: POST /people/me/all HTTP/1.1
Host: container.org
Content-Type: application/json

...representation of new Person elided...

-batch-a73hdj3dy3mm347ddjjdf
X-Batch-Operation: GET /people/me/friends?startPage=5&count=10&format=json-c
Host: container.org
If-None-Match: "837dyfkdi39df"

The response:

HTTP/1.1 200 OK
Content-Type: multipart/mixed; boundary=batch-a73hdj3dy3mm347ddjjdf

-batch-a73hdj3dy3mm347ddjjdf
X-Batch-Status: 201 Created
Location: http://container.org/people/me/all/container.org:5ea3gh838kjk34834
ETag: "993hhhgjgkkd"

-batch-a73hdj3dy3mm347ddjjdf
X-Batch-Status: 304 Not Modified
ETag: "837dyfkdi39df"
In this case, the friend add succeeded and the response to the retrieval of page 5 of the friends list indicates that page 5 didn't change (the friend was added below that page's last entry).  If the resource had changed, of course, you'd see a list of the friends in the response.

X-Batch-Operation: Contains the HTTP request line verbatim with verb, URL path, and HTTP version.
X-Batch-Status: Contains the HTTP status line verbatim with response code and any following text.

Only the top level request can specify overall request related things such as keepalives and content transfer encoding. The top level request response code is:

  • 200 if the batch was processed, even partially;
  • A 4xx error if the batch document itself was bad and could not be processed (note that a bad document might not be detected before partial processing occurs, so clients cannot rely on the server to do syntax checking).  An example of this would be a document that is not of type multipart/mixed.
  • A 5xx error if the proxy batch service itself is having severe problems.

Individual responses use the HTTP response codes they would have used had they been executed individually.  Requests and responses are matched by position, and the server guarantees that if it sends back a response, the ith response is for the ith request.  It does not guarantee that the number of responses matches the number of requests; in particular, network problems might truncate a response document or drop a connection, and in this case clients are left not knowing the status of the remaining responses.  If a server needs to indicate that a particular sub-request timed out without any indication of whether it succeeded or failed, it should use "X-Batch-Status: 504 Gateway Timeout" to indicate that it has a known unknown.  Clients seeing a 504 sub-response should be advised that the request may have completed before timing out.

Clients may discover whether a server supports batching by performing a GET or HEAD on the well known /batch_proxy URL endpoint.  The server response should include an Allow: header which contains POST if the server supports batching.  The response should be marked as cacheable so that clients can avoid round trips at startup.

Note that a container may support proxying to foreign servers (via the Host: header); however, servers are not required to support this and clients should not assume the capability.

In general, HTTP fields may be added to either the envelope request or sub-requests.  For example, a client may add an Authorization: header to the envelope request (to authorize itself to the proxy service) and also add Authorization: header(s) to different sub-requests (to authorize itself to other services). 

References: http://www.snellspace.com/wp/?p=788http://www.intertwingly.net/wiki/pie/PaceBatch

Partial Updates (Optional)

Partial updates avoid the need to send full representations of data on updates, especially for People and App Data.  The primary use case is when a client has retrieved a full representation and subsequently wishes to send a small update to a server.  In the most general case, the client my have a series of small updates (e.g., changing tags on a large number of contacts) scattered across many resources.  It should also allow for concurrency control.  We wish to have a generalized partial update / patching mechanism to handle such cases.  As with the batching mechanism, it should be optional on the server side; clients can always fall back to sending full representations back to the server if it does not support the mechanism.

The exact syntax is TBD at this point.

Kevin Brown

unread,
Feb 27, 2008, 4:12:28 AM2/27/08
to opensocial-an...@googlegroups.com
Awesome starting point -- is there perhaps a wiki or something we could use to make collaborating on this large document easier? Google docs perhaps? Or should we just make inline comments in email messages?
--
~Kevin

If you received this email by mistake, please delete it, cancel your mail account, destroy your hard drive, silence any witnesses, and burn down the building that you're in.

Kevin Marks

unread,
Feb 27, 2008, 10:02:37 AM2/27/08
to opensocial-an...@googlegroups.com
On Wed, Feb 27, 2008 at 12:31 AM, John Panzer <jpa...@google.com> wrote:
All,

I'd like to propose a starting point for an OpenSocial RESTful API specification.  The draft below is the result of a lot of individual discussions, a bunch of research and experiments, and, over the past couple of days, a whole lot of coffee.  Having said that, I regard this as a way to spark discussion rather than something being put forward for approval.  Please reply to the list with your opinions, reactions, and alternatives. 

One suggestion:  There are a lot of pieces to this (still under 10 pages though).  If you start a response thread, it could be useful to edit the subject line to advertise what you're discussing -- the section name, for example.  Otherwise discussions could easily be missed. ---

Excellent draft, John. Some inline comments on pieces of this (If you decide to move this to another form, let me know and I'll re-enter them).
 

Proposal for an  OpenSocial RESTful API (DRAFT)

The primary goal of this proposal is to serve as a common protocol understood by all OpenSocial clients and servers.  If this draft is adopted, it would replace the previously proposed GData-based OpenSocial data API.

Overview

This API defines a language- and platform- neutral protocol for OpenSocial clients and container servers to interact outside of gadgets on a web page.  As a protocol, it is intended to be reasonably easy to implement in any language and on any platform.  It should also be usable across a range of clients, from gadgets operating within a web page to servers communicating to synchronize data about a user.

The protocol operates primarily in terms of resources and operations on them.  It is defined on top of the HTTP protocol, and uses the standard HTTP methods (GET, POST, PUT, DELETE, etc.) to retrieve and change server state.

No single data representation is ideal for every client.  This protocol defines dual representations for each resource in two widely supported representations, JSON [RFC4627] and Atom [RFC4287][RFC5023], using a set of generic mapping rules.  The mapping rules allow a server to write to a single interface rather than implementing the protocol twice.

This seems a good compromise between the two worldviews.


It would be good for the person example to include one of the enum-type fields, so that the representation of those is clear in both forms.

A comment from Joe Gregorio on earlier drafts was that these Atom feeds will look oddly thin in a Feed-reader. He suggested adding an (optional) <summary> that was an HTML representation of the resources, so that viewing in a feed-reader makes sense.
 

An Activity Example

Activity application/json:

{

  'id' : 'http://example.org/activities/example.org:87ead8dead6beef/self/af3778',

  'title' : { 'type' : 'html',

              'value' : '&lt;a href="foo"&gt;some activity&lt;/a&gt;'

           }

  'updated' : '2008-02-20T23:35:37.266Z',

  'content' : 'Some details for some activity',
  'received' : '
2008-02-20T23:35:37.266Z'

}

Some questions on the 'html' - ness:
- the 'title' has a type, but the content doesn't - is the implied type 'text' as in Atom?
- You show HTML(double) escaped in both JSON and Atom; the Atom need is clear to follow spec, but escaping in that way in JSON looks unnatural and trouble-prone. Having the JSON form in raw HTML would make more sense. Make the escaping part of the Atom transform.

 


Activity application/atom+xml:
<entry xmlns="http://www.w3.org/2005/Atom>
 
<category term="status"/>
  <id>http://example.org/activities/example.org:87ead8dead6beef/self/af3778</id>
  <title type="html">&lt;a href="foo"&gt;some activity&lt;/a&gt;</title>
  <content>Some details for some activity</content>
  <updated>2008-02-20T23:35:37.266Z</updated>
 
<link rel="self" type="application/atom+xml" href="http://www.google.com/activity/feeds/.../af3778"/>

  <osa:received>2008-02-20T23:35:37.266Z</osa:received>
</entry>
Activity mapping rules:
atom:entry/content == content (type:text)       Map entry/content to content field in JSON representation, and vice versa.
atom:entry/summary == summary (type:text)  Same for summary.
atom:entry/id == id                                         Map ids (representation is the same)
atom:entry/title == title (type:text)                  Map titles over; if not plain text, use a structured type/value object to represent data.
atom:entry/updated == updated                      S.O.P.
osa:received == received                                Map received field, stripping the namespace.  In the JSON to Atom conversion, received always means osa:received.

Why doesn't osa:received map to atom:published ?
 

An AppData Example

The first example is of a collection of key/value pairs for a particular application/user pair:

application/json representation:

{

  'pokes' : 3,

  'last_poke' : '2008-02-13T18:30:02Z'
}

application/atom+xml representation:
<entry xmlns="http://www.w3.org/2005/Atom>
  <content type="text/xml">

    <pokes>3</poke>
    <last_poke>
2008-02-13T18:30:02Z</last_poke>
  </content>
  <title/>
  <updated>2003-12-13T18:30:02Z</updated>
  <author><url>
urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</url></author>
  <id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
</entry>
AppData mapping rules:
atom:entry/content == *                         Flatten entry/content into top level of JSON representation.  In the inverse mapping, all JSON fields are collected into atom:entry/content.

If the AppData JSON has substucture (a list or object), how is this translated through?
Also, the XML in content has lost the JSON type info for the number of pokes  - should the implied numeric type be mapped through:
<pokes type="number">3</pokes>

We may need similar rules for JSON true, false, null

The JSON definition here is different from the AppData collection above, which is an Object, not an Array
 

Kevin Brown

unread,
Feb 27, 2008, 1:31:22 PM2/27/08
to opensocial-an...@googlegroups.com

I agree -- additionally, the JSON spec actually requires double quotes for all strings (see http://json.org). We should adhere to that.
 

Louis Ryan

unread,
Feb 27, 2008, 2:05:15 PM2/27/08
to opensocial-an...@googlegroups.com
Looks good. Some notes based on earlier work that Kevin and I did related to this.

When referring to a collection of entities that is not owned by the referring path we will need to introduce a linking structure so that we don't muddy the REST semantics for updates
e.g.
/appdata/{uid}/friends  -- The data of the friends of a user
This collection is immutable at this address, either each entity provides its explicit 'edit' link or is wrapped in an <data-link> or somesuch. The alternative is to have client libraries that can calculate the appropriate paths to address entities from properties of the entity itself.

Need to work on the url scheme to make resource identification a little more deterministic. It will be hard/expensive to parse
/people/{uid}/self vs. /people/{uid}/{groupid}
or it will impose functional limitations on the backed by disallowing the use of "self" as a group name.


On Wed, Feb 27, 2008 at 7:02 AM, Kevin Marks <kevin...@google.com> wrote:

Joseph Smarr

unread,
Mar 4, 2008, 11:45:07 AM3/4/08
to OpenSocial and Gadgets Specification Discussion
This generally sounds pretty good to me. It looks like you're using
clever mapping rules to make the JSON representation look simple and
json-y, and not using it as a tacked-on second-class citizen, so nice
job having your cake and eating it too. :)

The things that worry me are the pieces that require special HTTP
mechanics to use--specifically the use of PUT/DELETE and the batch
syntax. IMO, these are a non-trivial barrier to adoption for many
developers, and in particular they remove one of the most useful and
productive features of simple REST APIs, namely that you can try URLs
in the browser and see the results immediately. Amazon and others have
talked about how this alone has lead to their REST APIs being used way
more than SOAP or any other formats that you can't just "play with" as
easily. I understand the elegance of this design, but I think
engineering for adoptability is even more important than engineering
for parsimony and elegance. For instance, would you consider at least
providing a way to specify the HTTP verb as a GET query var, e.g.
&action=DELETE&... so you could simulate the same effect in a web
browser? And for the batching, would you consider a JSON
representation that could be sent up in normal post data, e.g.
"commands:" [ { /* command1 json */ }, { ... }, ... ] again so you can
just get URL+JSON and you're done, rather than also having to know how
to read and write custom HTTP headers, which you can't do when living
in a web browser or even a JavaScript gadget. Finally, as for the URL-
patterns for the commands, I think it looks good but I hope the
container can at least specify a URL-prefix, as we used to be able to
do. IMO, if you can let containers pick their "base url", they can
easily emulate sub-patterns, but it's harder to ask every site to be
able to do domain.com/opensocial/... for instance, and I don't see any
reason why there couldn't be some very simple/standard way for the
container to specify an alternative root in the JavaScript or as a
meta-tag in the head of / (for server-side curling ala yadis) or
something.

Thanks for the update and I'm excited to see the progress here!
Hopefully my main feedback is clear: please don't underestimate the
loss of adoption you're risking with each bit of complexity you're
adding, and make sure that whatever remains really had to earn its
place.

Thanks! js
> representations, JSON [RFC4627 <http://www.ietf.org/rfc/rfc4627.txt>]
> and Atom [RFC4287 <http://www.rfc-editor.org/rfc/rfc4287.txt>][RFC5023
> <http://www.rfc-editor.org/rfc/rfc5023.txt>], using a set of generic
> mapping rules.  The mapping rules allow a server to write to a single
> interface rather than implementing the protocol twice.
>
> OpenSocial container servers are free to define additional
> representations but must support at least the JSON and AtomPub profiles
> defined in this document.
>
> The protocol defines /Activity, People, /and /AppData /resources and
> Person *mapping rules:*
> atom:entry/content == *                         /Flatten entry/content
> into top level of JSON representation.  In the inverse mapping, all
> otherwise unknown JSON fields are collected into atom:entry/content./
> atom:entry/id == id  (type:guid)               /Map ids, using a
> urn:guid: prefix for the Atom representation/
> /* atom:entry/title == name.unstructured */ This would map atom:title
> field to the unstructured name; optional but useful -John Panzer 2/26/08
> 1:56 PM
> atom:entry/updated == updated               /Map updated field;
> date/time representation is the same in both./
> *Activity mapping rules:*
> atom:entry/content == content (type:text)       /Map entry/content to
> content field in JSON representation, and vice versa./
> atom:entry/summary == summary (type:text)  /Same for summary./
> atom:entry/id == id                                         /Map ids
> (representation is the same)/
> atom:entry/title == title (type:text)                  /Map titles over;
> if not plain text, use a structured type/value object to represent data./
> atom:entry/updated == updated                      /S.O.P./
> osa:received == received                                /Map received
> field, stripping the namespace.  In the JSON to Atom conversion,
> received always means osa:received./
>
>       An AppData Example
>
> The first example is of a collection of key/value pairs for a particular
> application/user pair:
>
> application/json representation:
>
>     {
>
>       'pokes' : 3,
>
>       'last_poke' : '2008-02-13T18:30:02Z'
>     }
>
> application/atom+xml representation:
>
>     <entry xmlns="http://www.w3.org/2005/Atom>
>       <content type="text/xml">
>         <pokes>3</poke>
>         <last_poke>2008-02-13T18:30:02Z</last_poke>
>       </content>
>       <title/>
>       <updated>2003-12-13T18:30:02Z</updated>
>       <author><url>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</url></author>
>       <id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
>     </entry>
>
> AppData *mapping rules:*
> atom:entry/content == *                         /Flatten entry/content
> into top level of JSON representation.  In the inverse mapping, all JSON
> fields are collected into atom:entry/content./
> AppData *mapping rules:*
>         atom:feed/entry/author/url (type:index guid)                
> /Use the author/url field as the unique key for the JSON collection; on
> inverse mapping, take keys and store in author/url, using a guid mapping
> that adds urn:guid: prefix/
> atom:entry/content == *                                          
> /Flatten entry/content into each record in resulting set./
>
> The default representation of a collection would simply be a JSON list
> of records ([ {...}, {....}, ...]).  The special type "index" means that
> the collection is instead to be represented as a mapping, using the
> index field.
>
>     Collections
>
> Collections are a useful abstraction for dealing generically with
> multiple things.  They again have both Atom and JSON representations;
> the Atom representation is simply a feed whose entries are as specified
> above.  The default JSON representation is a JSON list of
> ...
>
> read more »

John Panzer

unread,
Mar 4, 2008, 2:04:14 PM3/4/08
to opensocial-an...@googlegroups.com
Thanks for the feedback -- the intention is definitely to make the API speak 'native' JSON rather than a translation.

On Tue, Mar 4, 2008 at 8:45 AM, Joseph Smarr <jsm...@gmail.com> wrote:

This generally sounds pretty good to me. It looks like you're using
clever mapping rules to make the JSON representation look simple and
json-y, and not using it as a tacked-on second-class citizen, so nice
job having your cake and eating it too. :)

The things that worry me are the pieces that require special HTTP
mechanics to use--specifically the use of PUT/DELETE and the batch
syntax. IMO, these are a non-trivial barrier to adoption for many
developers, and in particular they remove one of the most useful and
productive features of simple REST APIs, namely that you can try URLs
in the browser and see the results immediately. Amazon and others have
talked about how this alone has lead to their REST APIs being used way
more than SOAP or any other formats that you can't just "play with" as
easily. I understand the elegance of this design, but I think
engineering for adoptability is even more important than engineering
for parsimony and elegance. For instance, would you consider at least
providing a way to specify the HTTP verb as a GET query var, e.g.
&action=DELETE&... so you could simulate the same effect in a web
browser?

I think that providing a way to "play around" is really important, thanks for highlighting it.

For a couple of reasons, I don't think that embedding actions like DELETE in GET URLs is a good idea.  First off, this is what leads to crawlers/accelerators deleting content by accident (or by malicious intention).  To prevent this, you typically need to add additional hard-to-duplicate parameters that authorize the request in some way -- but then you've just lost your simple in-browser test case.  Finally, most of these requests require robust auth[nz] in any case, especially for modifying operations like PUT/POST/DELETE.  Cookies are insecure for production use here, so you end up needing to add those auth[nz] parameters again.

Also, I don't think that just doing this for DELETE is too compelling.  If you consider creating/adding/editing data, you also need to pass a payload of some kind, so unless you encode that in the URL (difficult to get right by hand...) you need to drop back to something else anyway, at least POST.

If containers want to provide an API-only playground (and they should!) I'd suggest some forms and/or free text fields that let people create the appropriate Javascript... and of course some running samples using things like curl :)
 
And for the batching, would you consider a JSON
representation that could be sent up in normal post data, e.g.
"commands:" [ { /* command1 json */ }, { ... }, ... ] again so you can
just get URL+JSON and you're done, rather than also having to know how
to read and write custom HTTP headers, which you can't do when living
in a web browser or even a JavaScript gadget.

You can write these headers just fine in a browser (using XHR+Javascript) or gadget.  Actually we've done a bit of experimentation in both writing and parsing the whole thing from raw Javascript + XHR, and it works well with near-zero overhead.

 
Finally, as for the URL-
patterns for the commands, I think it looks good but I hope the
container can at least specify a URL-prefix, as we used to be able to
do. IMO, if you can let containers pick their "base url", they can
easily emulate sub-patterns, but it's harder to ask every site to be
able to do domain.com/opensocial/... for instance, and I don't see any
reason why there couldn't be some very simple/standard way for the
container to specify an alternative root in the JavaScript or as a
meta-tag in the head of / (for server-side curling ala yadis) or
something.

This is a great point, I should probably mention that Google also can't use the proposed URL patterns as-is either due to deployment issues :).  Is single, discoverable URL prefix per container sufficient?  That would be great if true.
 

Kevin Brown

unread,
Mar 5, 2008, 1:15:57 AM3/5/08
to opensocial-an...@googlegroups.com

gadgets.io.makeRequest can be used to handle all of this as well if experimentation is desired. It would be trivial to write up a restful api test gadget under that model. This would, of course, need to accompany examples in PHP / Ruby / Python / Java / whatever to be complete.




--
~Kevin

Joe Gregorio

unread,
Mar 10, 2008, 10:45:34 AM3/10/08
to OpenSocial and Gadgets Specification Discussion
On Feb 27, 4:31 am, John Panzer <jpan...@google.com> wrote:
> A Person Example
>
> application/json representation:
>
> {
>
> 'id' : 'orkut.com:34KJDCSKJN2HHF0DW20394',
> 'name' : {'unstructured' : 'Jane Doe'}
>
> }
>
> application/atom+xml representation:
>
> <entry xmlns="http://www.w3.org/2005/Atom>
> <content type="text/xml">

Please use "application/xml" instead of "text/xml". See
http://www.xml.com/pub/a/2004/07/21/dive.html for all the gory details
on why.

-joe

Joe Gregorio

unread,
Mar 10, 2008, 10:55:55 AM3/10/08
to OpenSocial and Gadgets Specification Discussion
On Feb 27, 4:31 am, John Panzer <jpan...@google.com> wrote:
> application/json representation:
>
> {
>
> 'orkut.com:34KJDCSKJN2HHF0DW20394' : {'pokes' : 3, 'last_poke' :
> '2008-02-13T18:30:02Z' },
>
> 'orkut.com:58UIDCSIOP233FDKK3HD44' : {'pokes' : 2, 'last_poke' :
> '2007-12-16T18:30:02Z' }
> }
>

I put forward a similar collection document a while ago,
<http://bitworking.org/news/restful_json_client>
which in this case would look like:


{
"members": [
{"href": "1",
"etag":"0hf0239hf2hf9fds09sadfo90ua093j",
"entity", {
'orkut.com:34KJDCSKJN2HHF0DW20394' : {'pokes' : 3,
'last_poke' :
'2008-02-13T18:30:02Z' },

}
},
{"href": "1",
"etag":"0hf0239hf2hf9fds09sadfo90ua093j",
"entity", {
'orkut.com:58UIDCSIOP233FDKK3HD44' : {'pokes' : 2,
'last_poke' :
'2007-12-16T18:30:02Z' }

}
},


...
],
"next": null
}

This adds a 'next' link so that the server can enable paging for large
collections.
It also has an 'href' (in this case a relative URI) for each member of
the collection
so that you can GET/PUT/DELETE to update that individual entry. It
also
optionally contains the etag for that URI, so that you can send along
an If-Match: header on your PUTs and avoid the lost-update
problem <http://www.w3.org/1999/04/Editing/> without having to do an
extra GET on the individual URI to find the etag to begin with.

-joe

Joe Gregorio

unread,
Mar 10, 2008, 11:09:00 AM3/10/08
to OpenSocial and Gadgets Specification Discussion
On Feb 27, 4:31 am, John Panzer <jpan...@google.com> wrote:
> Discovery
>
> This proposal specifies URL template rules for discovering collections;
> the expectation is that all containers can support the same URL paths
> and that the rules for composing URLs are simple enough to avoid the
> need for generic discovery mechanisms in the basic protocol.

I would advise against this for many reasons and would suggest
link following over implicit URI construction. I have a much longer
article on XML.com for all the gory details:

http://www.xml.com/pub/a/2005/04/06/restful.html

The root of the problem is that putting the URL template
rules in the specification creates a single namespace
with no disambiguation scheme.
For example, this makes experimentation with
extensions problematic, i.e. if company A and B
both experiment with format=foo but mean entirely
different things by "foo".

It also forces all the requests to go through a single domain
name and doesn't allow resources to be distributed among
different domain names.

This actually arises quite a bit when doing development with AtomPub
and is easy to handle since AtomPub only allows link following
and doesn't have URL templates embedded in the spec, i.e.
AtomPub doesn't restrict the URI space of the server providing the
service.

For example Google currently doesn't supply valid Service Documents
for any of the GData services. I have actually gotten around that
by hosting a valid Service Document on my own server that points
to the GData services. In addition I can 'paste together' services
such as creating a service document that points at the Blogger
service and the Picasa Web service and now have a unified
service that supports blogging and adding pictures all w/o
the involvement of Google. These are just two of the benefits
of link following over apriori URL construction.

-joe

denka

unread,
Mar 13, 2008, 5:23:18 PM3/13/08
to OpenSocial and Gadgets Specification Discussion
Hi!

I really appreciate that someone would do this kind of a job: getting
down to which parts of a URL mean what, what the payload is and how
Collections are represented. But as a developer I would appreciate
even more if I could program to interfaces, rather than to a spec
(even if it is a modestly short, 10-page spec). In this regard, will
there be tools available that would allow to generate client stubs and
server interfaces given some easy-to follow description either in form
of IDL, or Java, or C#, or in some other popular language? Certainly,
those client stubs and server interfaces will need to be generated in
multiple languages, which could be left out for interested parties to
do. I realize that an industry-wide efforts to come up with such
tooling are underway, in this case, wouldn't invention of Open Social-
specific RESTFul APIs become an incompatible legacy as soon as dust
settles? Ideally, maybe tool set I'm asking about already exists,
please kindly point me in the right direction, and let us know how it
could be applied to abstract the protocol knowledge in this case.

I'm sorry if my note appears not supportive enough, but I'd argue that
minority of people care deeply about protocol if there are client and
server libraries allowing to just communicate using it. Lack of those
libraries represents a great entry barrier. I think it would be a very
productive exercise to first use APIs and language bindings that come
up (or are planned to) with GData, and then make this protocol
enhancement just plug into existing infrastructure.

Thank you.
Denis.

Vasu Nori

unread,
Apr 7, 2008, 9:33:38 PM4/7/08
to OpenSocial and Gadgets Specification Discussion
> Data Representations
>
>
> Each activity, person, and appdata is represented as a hierarchical tree
> of elements. Ordering of elements within a parent element may or may
> not be significant, depending on the context. Mapping consists of

is the ordering based on the way they are retrieved from the backend
where they are persisted?
Or, does the OpenSocial API server have to sort the data retrieved
from the backend based on certain query parameters? If so, what are
those query params?

> converting from the internal hierarchy to a JSON data structure or an

> application/atom+xml representation:
>
> <entry xmlns="http://www.w3.org/2005/Atom>
> <content type="text/xml">
> <name>
> <unstructured>Jane Doe</unstructured>
> <name>
> </content>
> <title/>
> <updated>2003-12-13T18:30:02Z</updated>

The above 2 fields are NOT in JSON.
JSON output has less info than Atom/XML does.
This lossy conversion to JSON is not a cause for concern?

> <author/>
> <id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
> </entry>

Atom allows id in URI format? how is it translated into JSON?
I only see conversion for "id" starting with "urn:guid:"


> In this example, we have chosen to flatten the "content" element by
> moving all its child elements to the top level. If there were a
> conflict, the rules could include a renaming for conflicting fields.
> This renaming scheme, as well as the choice to flatten "content", is
> driven by metadata defined above for the "Person" service.
>
> Note: The atom:summary element is the appropriate place to put a text or
> HTML representation of the structured data present in the content element;
> there should be a standard wayfor a client to request such a human readable
> representation, and servers may provide it if they decide to support it.

It says there "should" be a standard way. but what is it? do we need
to specify?


General question: Atom/XML defines pagination links Next/Prev.
In JSON, are those supposrted? if so, how do they look?

John Panzer

unread,
Apr 7, 2008, 9:42:49 PM4/7/08
to opensocial-an...@googlegroups.com
On Mon, Apr 7, 2008 at 6:33 PM, Vasu Nori <vasu...@gmail.com> wrote:

>     Data Representations
>
>
> Each activity, person, and appdata is represented as a hierarchical tree
> of elements.  Ordering of elements within a parent element may or may
> not be significant, depending on the context.  Mapping consists of

is the ordering based on the way they are retrieved from the backend
where they are persisted?
Or, does the OpenSocial API server have to sort the data retrieved
from the backend based on certain query parameters? If so, what are
those query params?

In some cases there's a default ordering (activity streams: reverse timestamp), in other cases there a query parameters (which ones MUST be supported by every container is an open issue I think).
 


> converting from the internal hierarchy to a JSON data structure or an

> application/atom+xml representation:
>
>     <entry xmlns="http://www.w3.org/2005/Atom>
>       <content type="text/xml">
>         <name>
>           <unstructured>Jane Doe</unstructured>
>         <name>
>       </content>
>       <title/>
>       <updated>2003-12-13T18:30:02Z</updated>

The above 2 fields are NOT in JSON.
JSON output has less info than Atom/XML does.
This lossy conversion to JSON is not a cause for concern?

It's an XML representation of the JSON data tree.  There's an open issue about annotating with the JSON types, probably with a type="bool" type of attribute.

 


>       <author/>
>       <id>urn:guid:orkut.com:34KJDCSKJN2HHF0DW20394</id>
>     </entry>

Atom allows id in URI format? how is it translated into JSON?
I only see conversion for "id" starting with "urn:guid:"

Yes, Atom allows generalized URIs as IDs.  The JSON mapping is explained elsewhere in the doc.  The JSON id is the non-URI string orkut.com:34KJDCSKJN2HHF0DW20394.  The URN thing was the quickest, simplest thing I could think of to map this non-universally-unique string into something universally unique :).
 



> In this example, we have chosen to flatten the "content" element by
> moving all its child elements to the top level. If there were a
> conflict, the rules could include a renaming for conflicting fields.
> This renaming scheme, as well as the choice to flatten "content", is
> driven by metadata defined above for the "Person" service.
>
> Note: The atom:summary element is the appropriate place to put a text or
> HTML representation of the structured data present in the content element;
> there should be a standard wayfor a client to request such a human readable
> representation, and servers may provide it if they decide to support it.

It says there "should" be a standard way. but what is it? do we need
to specify?

If we want it, we'll need to specify it :).  Do we want it?  Puts a burden on servers (though they can always punt with an empty string I guess).  Nice for some potential lightweight use cases.
 



General question: Atom/XML defines pagination links Next/Prev.
In JSON, are those supposrted? if so, how do they look?

This is a great question.  Should they be?  Do they need to be? 
 




Joe Gregorio

unread,
Apr 7, 2008, 9:57:35 PM4/7/08
to opensocial-an...@googlegroups.com

Kevin Brown

unread,
Apr 7, 2008, 10:09:09 PM4/7/08
to opensocial-an...@googlegroups.com

Maybe I missed something along the way here, but isn't parity between the two wire formats still an objective? It seems to me that  we'd need all fields in all places to make that happen. In order to use the JSON binding for implementing the opensocial JS APIs, pagination would be very useful.


 









--
~Kevin

John Panzer

unread,
Apr 7, 2008, 11:08:06 PM4/7/08
to opensocial-an...@googlegroups.com
It's a goal but it's not reachable in edge cases; JSON-as-she-is-spoke doesn't allow for namespaces, for example, so the conversion will necessarily be potentially "lossy".  So we're deliberately trying for "useful parity" of the non-edge cases that people care about.

Subbu Allamaraju

unread,
Apr 8, 2008, 1:58:40 PM4/8/08
to opensocial-an...@googlegroups.com
> > not be significant, depending on the context. Mapping consists of
>
> is the ordering based on the way they are retrieved from the backend
> where they are persisted?
> Or, does the OpenSocial API server have to sort the data retrieved
> from the backend based on certain query parameters? If so, what are
> those query params?
>
> In some cases there's a default ordering (activity streams: reverse
> timestamp), in other cases there a query parameters (which ones MUST
> be supported by every container is an open issue I think).
>
Sorry - but how does this lead to the statement that "Ordering of
elements within a parent element may or may not be significant,
depending on the context"?

Subbu

John Panzer

unread,
Apr 8, 2008, 2:33:26 PM4/8/08
to opensocial-an...@googlegroups.com

There are three possibilities depending on the particular collection resource type:

1. Ordering is unspecified in the spec, so servers are free to deliver the data in any order that makes their lives easier or their servers cooler.
2. Ordering is specified and hard-coded in the spec, so servers have to deliver the data in the exact order.
3. Ordering is requested by the client and may vary based on URL parameters, and servers have to deliver the data in that order.

#3 becomes particularly interesting when you have to page through large result sets, since it's not generally feasible to re-sort the entire collection on every call before delivering page 1 of 50.  #2 is not quite as bad but might still result in secondary indices.  #1 is obviously easiest for servers so if clients don't care about ordering for a particular type of collection we can go with #1. 
 


Subbu



Subbu Allamaraju

unread,
Apr 8, 2008, 2:57:52 PM4/8/08
to opensocial-an...@googlegroups.com
Ah - I see.

I think it is better to distinguish between elements representing collections (e.g. an element called "contacts" containing "contact" elements) and elements representing an entity (such as a "contact"). In the former case, the client may like to fetch the collection in a particular order. But in the later case, for the sake of simplicity, ordering should be avoided.

Subbu

Vasudeva Nori

unread,
Apr 8, 2008, 4:59:33 PM4/8/08
to opensocial-an...@googlegroups.com

Just to be sure, are we talking about
#1. ordering of data within "atom:content" element (in the case of Atom/XML and corresponding entities in JSON)
Or
#2. Ordering of atom:entry elements within atom:feed. 

If it is #1, then John has already enumerated the choices above.
If it is #2, Atom prescribes the ordering to be in reverse chronological order of the element atom:updated, if it exists. Are we saying that this reverse chronological ordering can be changed by the server, depending on possible user criteria?
 


Subbu









John Panzer

unread,
Apr 8, 2008, 5:11:35 PM4/8/08
to opensocial-an...@googlegroups.com

For #2, AtomPub prescribes the ordering of "Collection" feeds to be reverse app:updated order.  However, non-Collection feeds need not be ordered in this way.  The usefulness of mandating app:updated order for the (editable) collection type feeds is that it makes it possible for a client to synchronize its view of a collection without loss of data and without having to go back to the end of time on every sync.  Some resources exposed by OpenSocial are likely to be read-only feeds where lossless sync is a non-issue, and in which a client-provided sort order might be useful (example: alphabetical sort order by username for People).
 

 


Subbu












Subbu Allamaraju

unread,
Apr 8, 2008, 5:23:09 PM4/8/08
to opensocial-an...@googlegroups.com

My question is related to the contents of the atom:content element,
i.e #1 above, for which it is better to leave things unordered (in the
lines of interleave in RNG). I agree with John's comments about
collections. It is better to clarify these two cases in the draft
proposal.

Subbu

John Panzer

unread,
Apr 8, 2008, 5:28:06 PM4/8/08
to opensocial-an...@googlegroups.com
For things inside atom:content specifically, I think we should try to default to unordered (server can order however it wants) unless there's some good design reason to mandate an ordering.

Subbu Allamaraju

unread,
Apr 8, 2008, 5:35:28 PM4/8/08
to opensocial-an...@googlegroups.com
I agree.

Subbu

Reply all
Reply to author
Forward
0 new messages