Proposal: Data Store paging

[Oleg Pylnev]

Hello All,

 

The newFetchPersonAppData() call does not support paging when parameters VIEWER_FRIENDS and OWNER_FRIENDS used. For users with large number of friends it would result in potentially very expensive calls.

 

Current method:

newFetchPersonAppDataRequest(idSpec, keys)

 

Proposed:

newFetchPersonAppDataRequest(idSpec, keys, opt_params)

 

where opt_params are:

Map<opensocial.DataRequest.PeopleRequestFields.FIRST, opensocial.DataRequest.PeopleRequestFields.MAX>

 

Regards,

Oleg Pylnev

Myspace.com

 

[Arne Roomann-Kurrik (Google)]

+1 in certain situations, not having this functionality could
potentially cripple an application.

[Cassie]

Couple of things here...

How is the data going to be sorted? If it isn't sorted consistently than of course you can't page correctly. What do first and max mean? Does max = maximum number of key value pairs, or users? The former seems strange so I would go with the latter. Also, if it does mean max # of users than should user sorting be allowed? So that you can say "give me the data for my top friends"? And, if so, then shouldn't we add all of the sorting parameters to these calls too? But if we do that.. then your container will have to do a bunch of joins whenever fetching data, etc etc.

As an alternative..

What if you just give app developers quotas? Say, if they fetch more than XXX data in XXX hours then they get a "limit exceeded" error (orkut and hi5 are doing this). This will also help prevent evil (or just naive) gadget developers from hurting you. Then, gadget developers should just be encouraged to do any paging they want by themselves. In other words, they can use a fetchPeople request to fetch a certain number of people (top friends, sorted by name, excluding sally or whatever) get the ids for these people that they want and pass them to you. Then, the api stays clean (people joins and whatnot separate from data) the servers/wire are protected (because we have lots of quotas and developers are "charged" correctly for stuff) and everybody is happy.

Thoughts?

Thanks.

- Cassie

[Oleg Pylnev]

Sorting should be done by user Id.

Max = number of users per each request

 

Top friends can be an option, but as an explicit parameter. Same for passing in list of user Ids.

 

As for ‘fetch quotas’: They do make sense even if not enforced by some containers. The quotas may also prevent JS code errors like infinite loops

 

-Oleg

[Paul Lindner]

+1 if we can integrate the fetch logic for FetchPeople here. They are
similar use cases and common backend code could implement both features.

Alternatively one could use FetchPeople to grab a list of IDs and pass
that into a fetch for data, however that has latency downsides.

> You received this message because you are subscribed to the Google Groups "OpenSocial and Gadgets Specification Discussion" group.
> To post to this group, send email to opensocial-an...@googlegroups.com
> To unsubscribe from this group, send email to opensocial-and-gadg...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/opensocial-and-gadgets-spec?hl=en
> -~----------~----~----~----~------~----~------~--~---
>

--
Paul Lindner ||||| | | | | | | | | |
lin...@inuus.com

[Paul]

+1 for general paging support in anything in which a list of users is
involved.

This is useful whether or not the app has a list of specific ids and
whether or not the OS provider has implemented quotas. The app often
will not know the list of friends that have appdata for a particular
user, but may want to simply display the first 10 that have data. In
this case, they would get the app data first and then make a
fetchPerson request with the ids of the users.

The FIRST and MAX parameters would act the same as the
PeopleRequestFields:
http://code.google.com/apis/opensocial/docs/0.7/reference/opensocial.DataRequest.PeopleRequestFields.html#MAX

~Paul

[Louis Ryan]

If the most common intention is to fetch a page of users and fetch the app data associated with them then I dont think this quite covers it. The paging rules are specified in terms of number of entries. A 'page' of 20 users may only have app-data for 10 of them, trying to correlate these pages in JS on the client when their indices get offset is going to be a big headache for developers. In this situation a join may be the more natural semantic.

[Cassie]

What exactly do you mean by this latest comment, Louis? How would we modify the code change for this proposal to instead use a concept of joins?
Thanks!

- Cassie

[Louis Ryan]

Define the fetch of people in the usual fashion and include the optional bind-id. The opt_params can specify paging of the People results which is already allowed for

newFetchPeopleRequest(idSpec, fields, opt_params, <person-bind-id>)

In the same batch we allow the IdSpec to be a join specifier which refers to the bind-id of the person fetch and constrains itself to app data only for those people

newFetchPersonAppDataRequest({..., JOIN: <person-bind-id>, ...}, keys, opt_params , bindId)

I dont think we need to provide any join type other than a full join (no outers or anti-joins). We should also limit the join to one and only one per fetch app data request.

The real question is which is more useful, correlated joins on pageable users, or arbitrary paging of app data. I believe its the former.

-Louis

[Arne Roomann-Kurrik (Google)]

I think that I was expecting the paging of the data store to act like
the data was already joined with the people table... i.e. a data
request with the same parameters as a people request would return only
data for the people returned by the people request. If the people in
the page returned by the people request didn't have app data, then the
data request would return null (even if the next page returned some
data).

Of course, now I see that that's a little awkward and that Louis'
proposal makes this expectation clearer. +1 to correlated joins on
pageable users.

~Arne

On Apr 28, 10:03 am, "Louis Ryan" <lr...@google.com> wrote:
> Define the fetch of people in the usual fashion and include the optional
> bind-id. The opt_params can specify paging of the People results which is
> already allowed for
> newFetchPeopleRequest(idSpec, fields, opt_params, <person-bind-id>)
> In the same batch we allow the IdSpec to be a join specifier which refers to
> the bind-id of the person fetch and constrains itself to app data only for
> those people
> newFetchPersonAppDataRequest({..., JOIN: <person-bind-id>, ...}, keys,
> opt_params , bindId)
>
> I dont think we need to provide any join type other than a full join (no
> outers or anti-joins). We should also limit the join to one and only one per
> fetch app data request.
>
> The real question is which is more useful, correlated joins on pageable
> users, or arbitrary paging of app data. I believe its the former.
>
> -Louis
>

> On Mon, Apr 28, 2008 at 4:30 AM, Cassie <d...@google.com> wrote:
> > What exactly do you mean by this latest comment, Louis? How would we
> > modify the code change for this proposal to instead use a concept of joins?
> > Thanks!
>
> > - Cassie
>
> > On Sat, Apr 26, 2008 at 4:27 AM, Louis Ryan <lr...@google.com> wrote:
>
> > > If the most common intention is to fetch a page of users and fetch the
> > > app data associated with them then I dont think this quite covers it. The
> > > paging rules are specified in terms of number of entries. A 'page' of 20
> > > users may only have app-data for 10 of them, trying to correlate these pages
> > > in JS on the client when their indices get offset is going to be a big
> > > headache for developers. In this situation a join may be the more natural
> > > semantic.
>

> > > > lind...@inuus.com

[Cassie]

So it looks like we want this functionality to be added but we haven't quite settled on what the code change would actually look like. For this reason I would like to defer this to 0.9. This will give us more time to get the paging done correctly.
Thanks.

- Cassie

[Louis Ryan]

Im fine deferring. We can experiment with joins etc through the opt_params....