REST: Always fetch all fields?

[Chris Chabot]

From the RESTful spec:
fields={-join|,|field} -- <snip> Default is to include all available
fields.

It would seem to me that that is the opposite from how the JavaScript
API works, where you have to query for supported fields, and then
specify which fields you want to retrieve.

While having always all fields return is somewhat convenient to the
end user, are we not worried about the extra bandwidth consumption and
IO strain on the social data back-end? There are a -lot- of possible
fields which might be spread out over quite a few sources, and always
collecting all of them when applied to a massive scale can have an
impact on performance.

When it always returns everything, i don't see why a developer would
specify which fields to retrieve , why not always fetch (the default)
all fields, even if you just need a name?

Just a thought :)

-- Chris

[Ian Boston]

I suspect any developer concerned about UI responsiveness will always
ask for the full set, as their gadget will cache the in javascript
scope and they wont want to do more than one network roundtrip if
possible.

Perhaps an approach might be for the spec to allow the server to
decide what will be returned in the default set, and also provide a
list of available fields. If the gadget developer wants more than the
default set, they will have to ask for them ?

Thinking further, as a gadget developer, on the minimization of
roundtrip, and reducing the number of chained async calls, one
approach could be for the gadget to request the fields they want, and
the restful api to respond with the default set, and any additional
requested field, indicating (by some json structure) which fields
were not available.

Putting all this on the URL would encourage developers not to request
everything, and increase the possibilities for http caching.

I guess if this looks like an approach, its something for the future.

Ian

[Chris Chabot]

Actually i was thinking from the perspective from the 'external rest atom/json consumer', 3rd party servers that consume social information through the rest interface.

However you actually bring up a very good point, the gadgets talk to this same interface, so the rest spec actually -contradicts- how the js open social spec currently works... 

Currently a javascript newPersonRequest() call does returns a default set of fields (id, name, thumbnail & profile url), however the RESTful API spec says this very same call returns all the fields. Since the javascript/open social gadgets will be using the -same- rest interface for their social data communications as 3rd parties connecting directly to the rest interface, this -could- automatically mean that if you don't specify the fields, you get all of them? Thats quite a change :)

A change i wouldn't encourage .. if someone has a 200 friends (and gadgets don't actually do much of paging, which makes sense since that's more round trips so they expect complete lists back), and a full profile is ~ 4kb  (thats Joseph Smarr's example) to 10k (fully filled in test profile on i created in partuza), we're already talking about somewhere between 800kb to 1.6mb of information ... per gadget that loads a friends list!

Say that an average profile has 2 to 4 gadgets on their profile that use 'friends' (that's a very social thing to do), then your looking at a worst case scenario of 6.4mb of json data.. that takes a VERY considerable time to download even on the fastest commonly available broadband connections.. and then your not even yet thinking of the time it would take for a browser to parse the json into internal data structures ... that would take a very considerable amount of time too ... and *poof* the worst case load time for a social page goes up to minutes instead of seconds? 

That sounds like the best way to kill a social site experience i've ever heard ...  c64 tape drive type loading times of pages :)

So i guess the obvious solution to the above problem is to leave things as they are for gadgets, and enforce the basic subset of fields from javascript (ie no fields specified != all but is id, name,profile & thumbnail url), which leaves us with an inconsistent state between spec's ... and inconstancy not a great way to make a spec accessible to the world en large.

Plus it still leaves us with the original problem statement.. 800k to 1.6mb of information isn't fun when your doing server to server actions either, it puts a large load on the SNS's infrastructure to be able to produce that hundreds of times per second, it puts a large load on the receiving end to digest that much information quickly enough to get a page out in a few ms, and it would consume many times more bandwidth for both parties as well.. I think that would leave us with a somewhat destructive situation if that became the 'normal way' to use the restful interface.

The other way around it also leaves us with a problem, if we only return a default set of fields and expect the service consuming client to specify which fields it requires, it also needs to have a way to find out which fields are supported by the container, and currently there is no such mechanism in place ... and it would make that bit incompatible with Joseph's Opencontacts alignment proposal, which also assumes getting -all- fields back when doing a /people/{id}/@all request.

Comments? Thoughts?

-- Chris

[Ian Boston]

Chris,
Ahh I wasn't thinking client servers, but either way, if the spec
doesn't minimize the impact on the server then its going to hit
scalability regardless of client.

4 -- 10k per request, feels acceptable, but what is not acceptable is
200x10K for the friends list. Does the spec say that when a friends
list is returned the full profile of each friend must be sent ? I
should go and read some more.

It would be a little worrying to expect that the server might be
required to generate this much json. On current tests in java thats
~50ms per request just to serialize the json, leaving out getting the
object tree or raw data. On the same platform the whole request
should really be < 10ms.

Perhaps there should some form of depth specification in the request
to give control over how populated the response tree is ? (ie
populate down to 3 levels, pointers beyond that) That would allow a
full Friends list, with lazy loading of more detailed metadata.

Ian

[Cassie]

Chris -

The newFetchPerson call will still work as intended, regardless of what the js api does. The contract of that method is that if you don't specify any fields you will always get the default fields requested (id, name etc) In the js we explictly add these default fields to the list of fields we will request from the server. So because we do it that way (as opposed to leaving the fields param out) then the js api won't be affected.

That said, I still agree with your inital comment. The rest api outputting all of its fields all the time seems like a real burden on the container... which is exactly why the js api made the choices it did...

- Cassie

[Kevin Marks]

I can see how this might make sense in the portable contacts case,
where 'give me everything in on shot and I'll merge it' is attractive
(especially when combined with the updatedSince option), but even
there the chances that both parties support the union of all fields is
unlikely.

It also raises permission issue - the default everything model is
problematic, and users need to be ware that this is the permission
they are giving, as opposed to the 'just the fields that are
required' model that suits most apps - a personal contact manager
merging app is unusual in the depth of trust needed in it, compared to
most applications that won't need phone numbers and home addresses for
all your friends.

The other reason a full representation is needed is that if an
application wants to modify a field in a Person resource, the AtomPub
model is that you should GET the whole resource, change the fields
client-side, and PUT the modified version back. Again, this has an
implicit trust model where the client has full access. In practice,
particularly for Person, containers are unlikely to allow this to any
client app. This is done to avoid having a separate filed-level delete
option, so fields removed are assumed to be deleted by design.
Clearly, if the client has a narrow remote representation that doesn't
round-trip correctly, this can lead to data loss.

So I agree the default should be a minimal set, but we should add a
simple option to get all fields (@all in the fields param perhaps).

This raises the issue of PUTting or POSTing with a fields parameter to
indicate which fields to write back, which may be a good answer, but
is at variance with AtomPub practice, as it doesn't have an idea of
server-controlled merge.

[Cassie]

This is already part of the restful spec.
For example, when updating appData if you say fields=count, only the count field will be updated all other fields will be left alone.
 

[Chris Chabot]

On Jul 18, 2008, at 5:30 PM, Kevin Marks wrote:

> I can see how this might make sense in the portable contacts case,
> where 'give me everything in on shot and I'll merge it' is attractive
> (especially when combined with the updatedSince option), but even
> there the chances that both parties support the union of all fields is
> unlikely.

Thats exactly why i proposed to leave the portable contacts out of the
scope of the RESTful spec discussion if we can; I would much rather
get to a well working, well documented RESTful spec first that makes
sense to everyone, and then add support for portable contacts, then
take away our ability to still change it due to the portable contacts
requirements. (Doesn't mean we can't keep it in mind, mind you, but
lets not restrict our selves either to thinking there can only be one
road that leads to rome).

> It also raises permission issue - the default everything model is
> problematic, and users need to be ware that this is the permission
> they are giving, as opposed to the 'just the fields that are
> required' model that suits most apps - a personal contact manager
> merging app is unusual in the depth of trust needed in it, compared to
> most applications that won't need phone numbers and home addresses for
> all your friends.

That is a good point and probably something we should add to the
RESTful spec, right now it says it returns 'all the fields', while i
think the intention (remembering Smarr's Google IO presentation on
this topic) that a user can choose to give permission to certain parts
of his profile information (probably you don't want to allow every
site access to all your private email addresses and your home fax
number for instance?). The current wording of the spec doesn't touch
on this, so people (consumers and SNS's alive) might assume they don't
have to deal with the permissions, and that could lead to some nasty
headlines in the media (open, unrestricted REST API's made the media
once before for myspace i believe)

> The other reason a full representation is needed is that if an
> application wants to modify a field in a Person resource, the AtomPub
> model is that you should GET the whole resource, change the fields
> client-side, and PUT the modified version back. Again, this has an
> implicit trust model where the client has full access. In practice,
> particularly for Person, containers are unlikely to allow this to any
> client app. This is done to avoid having a separate filed-level delete
> option, so fields removed are assumed to be deleted by design.
> Clearly, if the client has a narrow remote representation that doesn't
> round-trip correctly, this can lead to data loss.

Kevin had a good remark about this ... probably we would want to allow
a books application to modify the favorite books field for instance,
while we probably don't want it to modify, say, the password field :)
So it's good that the spec 'allows for the concept of modifying the
persons record' in it's self.

The spec does mention partial updates, but currently lists it as 'to
be determined', i guess that's hopefully where we'll find the answer
to the above situation.

[Louis Ryan]

I believe there has occasionally been mention of using field=* to denote that the client wants to retrieve all fields. Making the default 'include all' is dangerous and Im with Chris on his analysis of the traffic and performance costs.

[Chris Chabot]

Actually the app data works that way idd.

Extending that methodology to people data too would work for me (I'm a sucker for consistency)

-- Chris

[John Panzer]

The topic of default field sets has been batted around a bit but no one has been brave enough to propose a spec change :).  I personally like the idea of:

o Default is a useful subset of fields
o You can explicitly ask for a specific subset
o You can explicitly ask for 'everything' but this is never the default

I also like field=* if that character doesn't cause problems in the various contexts under discussion.  (If it is problematic for some reason, perhaps @all?)



John Panzer (http://abstractioneer.org)

[Chris Chabot]

That leaves us with one questions still unanswered: How does the client know what fields the container support?

Without that bit of information easily available, we'll soon be walking down the road of RESTful clients containing container specific code.

-- Chris

[Paul Lindner]

It seems that we need to surface the introspection functionality that's available in the JavaScript API.

An atom/js feed of supported fields perhaps?

Paul Lindner

plin...@hi5.com

[Cassie]

the default set for the opensocial js apis is: id, name, thumbnailUrl
we should use the same set for rest

- Cassie

[Chris Chabot]

/people/{guid}/@supportedFields perhaps? :)

[Chris Chabot]

agreed, the question is more about "how does the client find out what
fields the container supports" (shindig/config/container.js :
supportedFields in shindig)

[John Panzer]

Though /people/@supportedFields would make more sense from a container perspective (and be probably extremely cacheable).

John Panzer (http://abstractioneer.org)

[Ropu's iPhone]

U need the guid? I dont think so.

Can that be described somewhere else? Like how we have it in the js to avoid an extra call? 

Sent from Ropu's iPhone

[Ian Boston]

I like that, as it can be stored as a JS object inside either the OS
js api, or a gadget developers js binding to the rest API.

One step further might be to make the container spec a generated
feature (as in gadget feature) containing all the container setup,
avoiding most of the round trips, and pumping the result directly
into the javascript scope. There would need to be an alternative for
server-server interactions.

Ian

[Chris Chabot]

> I like that, as it can be stored as a JS object inside either the OS
> js api, or a gadget developers js binding to the rest API.

On the gadget / javascript side of things the supportedFields is
already injected into scope through the initial configuration, when
you look at a gadget source you'll see something like:

gadgets.config.init({
<snip lots of stuff>
"supportedFields":{
"person":
["id
","name
","thumbnailUrl","profileUrl","id","name","thumbnailUrl","profileUrl"],
"activity":["id","title","id","title"]
},
<snip>
});

And there is a API available in open social to get the supported
fields (and their key name) through javascript for the gadgets, which
already used by a lot of gadgets, and seems to be working well. No
need for any new features or anything, that's already part of how it
works :)

The question is "How do we expose the same information to the external
RESTful service consumer" And for that a /people/@supportedFields and /
activities/@supportedFields would be a good solution.

On Jul 19, 2008, at 12:05 PM, Ian Boston wrote:

> One step further might be to make the container spec a generated
> feature (as in gadget feature) containing all the container setup,
> avoiding most of the round trips, and pumping the result directly
> into the javascript scope. There would need to be an alternative for
> server-server interactions.

That's what shindig/config/container.js is for, and it's injected into
the JS scope through the gadgets.config.init({}) call.

[Ian Boston]

Would it make sense to use a container wide url, and embed the
information inside the json ?

eg
/containerSupport

returns as selected subset of JSON structure passed to
gadgets.config.init(...);

Although we have a small number of types at the moment, I suspect
that there will be more, and it leaves and extension point open for
real SNS implementations.

Ian

Ian

[Paul Walker]

+1

Done @MySpace