Proposal: Radically simplify the Persistence/AppData API

[Arne Roomann-Kurrik]

AppData has a few hard-to-fix issues which make it unpalatable for developers. 

For example, paging through the data store isn't possible, which is a problem on large data sets.  (orkut allows 1000 friends with 10k AppData each - this would all be returned from a single request!).  A proposal to allow paging revealed some additional complications that kept this feature from going into the spec:  http://groups.google.com/group/opensocial-and-gadgets-spec/browse_thread/thread/62c760430bd15e79/

Additionally, the API is needlessly complicated.  If the spec says that I can only write to VIEWER data, why do I need to specify the VIEWER on write requests?  If I just want one person's data, why does it come back indexed by their ID number?- that makes me do things like request an additional OWNER object each time I just want the OWNER's data, so that I can find out the correct ID number to index the AppData response by.

This may be too dramatic of a change for 0.9, in which case I'll propose it for the next version, but I think AppData should live closer to the Person object in the JS API.  Here are the changes I'd like to see:

1.) Change newUpdatePersonAppDataRequest to only take (key, value) as its arguments.  Writing to VIEWER will be implied.

2.) Change newRemovePersonAppDataRequest to only take (keys) as its argument.  Deleting from VIEWER data will be implied.

3.) Remove newFetchPersonAppDataRequest entirely.

4.) Add opensocial.DataRequest.PeopleRequestFields.APP_DATA, defined as:
"An array of strings, specifying the app data keys to fetch for each of the person objects."

5.) Add an opensocial.Person.getAppData(key) method, defined as:
"Gets the app data for this person that is associated with the specified key.
 Parameters:
    String key - The key to get app data for."

~Arne

[Lane LiaBraaten]

I think I like it.  Let me walk through the code to make sure:

So to store a key/value I would do:

var req = opensocial.newDataRequest();

req.add(req.newUpdatePersonAppDataRequest('gift', 'a cashew'));

// NOT req.add(req.newUpdatePersonAppDataRequest("VIEWER", 'gift', 'a cashew'));

req.send();

and to retrieve it I would:

var params = {};

params[opensocial.DataRequest.PeopleRequestFields.APP_DATA] = ['gift'];

var req = opensocial.newDataRequest();

req.add(req.newFetchPersonRequest(opensocial.DataRequest.PersonId.VIEWER, params), 'viewer');

req.send(function(data) {

  var viewer = data.get('viewer').getData();

  var gift = viewer.getAppData('gift');

});

instead of:

var req = opensocial.newDataRequest();

req.add(req.newFetchPersonRequest("VIEWER"), 'viewer'); // unnecessary request for viewer

var viewer = opensocial.newIdSpec({ "userId" : "VIEWER" });

req.add(req.newFetchPersonAppDataRequest(viewer, 'gifts'), 'data');

req.send(function(data) {

  var viewer = data.get('viewer').getData();

  var appdata = data.get('data').getData();

  var gift = appdata[viewer.getId()] // but you need to have viewer's ID to get the data

});

I like how this aligns the appData with the person object (and gets rid of the "unnecessary" request for viewer).

Can we tweak the proposal so it's backwards compatible?

1) & 2) could be implemented in a backwards compatible way thanks to loose typing in JS (i.e. treat the params differently based on how many you get).  We could deprecate the v0.8 signatures for now.

3) "Deprecate", not "Remove"

4) & 5) These are additions so there's no conflict.

-Lane

[Zhen Wang]

I'd like to see a more radical simplification of the JS API -- a simple, SQL-like query language (OSQL?) that unifies the data access interface of appdata, people, and activities.

Think of appdata, people, and activities as three big tables of key-value pairs indexed by owner ids. All data requests will then boil down to a clean OSQL call.

For example, a "newPersonAppDataRequest" will translate to:

var osql = new opensocial.query('SELECT owner, key FROM appdata WHERE owner IN group_id');
osql.execute(function(result) {
  // result is an array of [owner id, appdata value]
});

Pagination can be supported by the "LIMIT ... OFFSET" syntax (e.g. "SELECT owner, key FROM appdata WHERE owner IN group_id ORDER BY owner LIMIT 20 OFFSET 40").

Similarly, we can update or remove appdata using queries like "UPDATE appdata SET key = value", "DELETE key1, key2 FROM appdata" with an implicit "WHERE owner = owner_id" clause.

Thoughts?

On Fri, Oct 31, 2008 at 4:52 PM, Arne Roomann-Kurrik <kur...@google.com> wrote:

[Kevin Brown]

On Fri, Oct 31, 2008 at 6:07 PM, Zhen Wang <wa...@google.com> wrote:

I'd like to see a more radical simplification of the JS API -- a simple, SQL-like query language (OSQL?) that unifies the data access interface of appdata, people, and activities.

Think of appdata, people, and activities as three big tables of key-value pairs indexed by owner ids. All data requests will then boil down to a clean OSQL call.

For example, a "newPersonAppDataRequest" will translate to:

var osql = new opensocial.query('SELECT owner, key FROM appdata WHERE owner IN group_id');
osql.execute(function(result) {
  // result is an array of [owner id, appdata value]
});

Pagination can be supported by the "LIMIT ... OFFSET" syntax (e.g. "SELECT owner, key FROM appdata WHERE owner IN group_id ORDER BY owner LIMIT 20 OFFSET 40").

Similarly, we can update or remove appdata using queries like "UPDATE appdata SET key = value", "DELETE key1, key2 FROM appdata" with an implicit "WHERE owner = owner_id" clause.

Thoughts?

While interesting, this is orthogonal to Arne's proposal. I think coupling app data to People objects makes sense (especially since global app data never really worked out anyway and in practice they're keyed that way), and would make it much more useful.

A query language is more general purpose and a far bigger change than this. It's not a bad idea by any means, but I think it's probably not the sort of thing you want to drop in a point release :).

[Evan Gilbert]

+1 on this, but would love to hear feedback from people about:
- Whether it's working making a backwards-incompatible change, and
- If there are any reasons we shouldn't join app data to people.

Evan

[Arne Roomann-Kurrik]

To Zhen:
I agree that a query-style syntax would be very powerful, but really merits its own proposal. 

Responding to Lane's feedback:
Do we need backwards compatibility from the method signatures?  If users want to write using the old syntax, they would <Require feature="opensocial-0.8">, right?
I'd imagine that a JS translation layer could be used to map 0.9 back into 0.8 (like was done with 0.8->0.7).  Containers would just have to continue support for newFetchPersonAppData, so I'm fine with deprecating instead of removing that method.

~Arne

--
OpenSocial IRC - irc://irc.freenode.net/opensocial

[Scott Seely]

On this mod—can we piggy back some language indicating the minimum size of the AppData? I know that there has been concern over the paltry amount of space MySpace provides vs. the 10K that ShinDig provides. Language like ‘Containers SHOULD provide at least 10KB of space for AppData.’ would be enough to take care of another set of concerns.

[Louis Ryan]

I think this is a reasonable change to make

+alot to supporting joins in some form. Im going to write something up on this to get a separate thread going

 

I don't see any major issue with this being backward incompatible.

On Wed, Nov 5, 2008 at 7:31 AM, Evan Gilbert <uid...@google.com> wrote:

[Arne Roomann-Kurrik]

I pulled the feedback from this thread together into a patch and uploaded to codereview.appspot.com:

  Changes to datarequest.js:
  http://codereview.appspot.com/7752/diff/5/203

  Changes to person.js:
  http://codereview.appspot.com/7752/diff/5/202

This patch incorporates the following:

1.) Changed newUpdatePersonAppDataRequest to only take (key, value) as its arguments. 
2.) Changed newRemovePersonAppDataRequest to only take (keys) as its argument. 
3.) Deprecated newFetchPersonAppDataRequest.
4.) Deprecated opensocial.DataRequest.DataRequestFields.ESCAPE_TYPE
5.) Added opensocial.DataRequest.PeopleRequestFields.ESCAPE_TYPE
6.) Added opensocial.DataRequest.PeopleRequestFields.APP_DATA
7.) Added opensocial.Person.getAppData(key)
8.) Added language to newUpdatePersonAppDataRequest indicating that containers supporting this method SHOULD provide 10KB of space per user per app.

~Arne

[Lane LiaBraaten]

+1 - and +1 on codereview.appspot.com!

[Scott Seely]

It seems like we should deprecate opensocial.DataRequest.DataRequestFields instead of opensocial.DataRequest.DataRequestFields.ESCAPE_TYPE. Reason: the proposal deprecates the only field in the enum.

 

Is there a reason to keep the type alive?

 

From: opensocial-an...@googlegroups.com [mailto:opensocial-an...@googlegroups.com] On Behalf Of Lane LiaBraaten
Sent: Friday, November 07, 2008 10:21 AM
To: opensocial-an...@googlegroups.com
Subject: [opensocial-and-gadgets-spec] Re: Proposal: Radically simplify the Persistence/AppData API

 

+1 - and +1 on codereview.appspot.com!

[Arne Roomann-Kurrik]

No, I'll update the patch to deprecate the entire structure.

~Arne

[Arne Roomann-Kurrik]

http://codereview.appspot.com/7752 is now updated with this modification.

~Arne

[Scott Seely]

+1

 

Total of 4 yes votes: (Arne, Evan, Lane, Me)

[Chris Chabot]

+1 from me too, nice work arne!