Proposal for 1.0: Define EL access to meta-data about data-pipeline tag response

[goosemanjack]

Synopsis:

The EL allows access to the underlying data that results from a data
pipeline tag, but does not clearly expose any of the meta-data also
included in the response. We need a way to expose and access this
information from the EL.


We would propose two new reserved values be added to the base data for
any data-pipeline generated data registered in the Data Context.

.requestParams
.responseInfo

The "requestParams" key would contain subkeys corresponding to all
attributes that were part of the originating request.

Ex: A viewer has a friends list of 212 people. The second page of
data is requested with a people request tag defined below

<os:PeopleRequest key="foo" userId="@viewer" groupId="@friends"
startIndex="6" count="5" />

would generate data registered with the data context under key "foo"
and would expose the startIndex value of "6" in the EL as follows:

${foo.requestParams.startIndex}

and the totalResults of "212" would be accessed with EL as follows:

${foo.responseInfo.totalResults}

This will both allow meta information being returned by the underlying
REST endpoint in the response defined under the OpenSocial REST spec
here:
http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/REST-API.html#rfc.section.3.1

It will also facilitate future functionality in v.Next for
reprocessing original requests with differing parameters - the most
obvious use-case being paged data lists updated by modifying the value
under foo.requestParams.startIndex.

[Jorge Reyes]

+1 I think the metadata information specially if the user is trying to page is critical to the developer. 

[Adam Winer]

+1 to the general idea - reprocessing requests with small changes is a
big win - but I'd want to think through the format. In particular,
I'm not thrilled with putting the request params directly inside the
dataContext for "foo"; I imagine this as a separate bucket of
variables - ${requestParams.foo} instead of ${foo.requestParams} -
though that's not a strong conviction yet. And, though I'm not
looking at the spec right now, I thought there was a way to get
"totalResults" already.

[goosemanjack]

In the EL, there is no explicit way to get totalResults. In
Javascript one should be able to retrieve the data stored under a
given key and interrogate the meta-information in the wrapper object
(officially after we sort out changes to the text of what the DP tags
return), but the EL does not expose it. Currently we carry meta
information through at the root, but this is a sloppy solution that
attaches four of five nodes of information at the top level. When
pulling in data from a third-party site the possibility of a collision
of keys increases to the point where it is unacceptable and nesting
this information (for the purposes of the EL) starts to look like a
more appropriate solution.

> >>http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RES...

[Lev Epshteyn]

I don't think a collision is currently possible - the JSON format I believe is something like

{
   totalResults: x,
   startIndex: y,
   data : { /* actual data received, this is the object surfaced to ${} */ }
}

So the information *is* present (not the request info, but the response info), but not accessible through templating.

As Adam, I'm +1 on the idea of surfacing it to EL, but not sure I like the specific version being proposed.

Putting an API on DP tags to make them mutable/reprocessable through JS has been something I've been suggesting for a while and am a big proponent of, but this seems a little different.

[Evan Gilbert]

+1 on the idea, should discuss naming as part of writing up the spec change. 

We should clarify how this data is surfaced for all three cases where data pipelining is used: templates, JavaScript API, and proxied content.

Some specific feedback:

- Prefer single words (request/response/error? request/data/error)?

- totalResults, etc should be part of the response object. The actual list should be a named field (list, items, entries?)

Evan

--

[Lane LiaBraaten]

okay - let's call this "in scope" for now and work out the details later, so we can focus on whether the other new proposals are in scope or not.

-Lane

[goosemanjack]

Patch created for evaluation

http://codereview.appspot.com/150052

I also updated the wiki with the patch contents

http://wiki.opensocial.org/index.php?title=Define_EL_access_to_meta-data_about_data-pipeline_tag_response


The short synopsis is root-level response meta information is accessed
via the "_response" key.
Ex: access totalResults via statement $
{friends._response.totalResults}

Originating request attributes may be accessed via "_request" key.
Ex: access requested count via statement ${friends._request.count}

The patch is specifically not specific on how to store the request
attribute data and leaves that to the container. I've left the
reprocessing Javascript API syntax out of this version so we can
experiment and bring something to the table in v.Next.
--
clc

[Evan Gilbert]

On Fri, Nov 6, 2009 at 2:38 PM, goosemanjack <cc...@myspace.com> wrote:

Patch created for evaluation

http://codereview.appspot.com/150052

I also updated the wiki with the patch contents

http://wiki.opensocial.org/index.php?title=Define_EL_access_to_meta-data_about_data-pipeline_tag_response


The short synopsis is root-level response meta information is accessed
via the "_response" key.
Ex: access totalResults via statement  $
{friends._response.totalResults}

Originating request attributes may be accessed via "_request" key.
Ex: access requested count via statement  ${friends._request.count}

Like this in general, couple of specific comments:

- We've been using UpperCase for reserved fields in EL. Possibly we can use Request instead of _request

- For the response, think it might make sense to have both the array and the response metadata at the root, instead of encapsulating.

So for 

people {
  startIndex: 0,

  count: 20,

  totalResults: 143,

  entry: [...]
}

The expressions would evaluate with the following:

${people.count} -> 20

<div repeat="${people}">${displayName}</div> -> Shows every person's name

${people[0].displayName} -> the first persons name

${people['count']} -> 20

This means that ${foo[number]} is evaluated differently than ${foo[string]}, which doesn't seem to cause any practical problems, although it might require a special case in the EL evaluation code.

--

You received this message because you are subscribed to the Google Groups "OpenSocial and Gadgets Specification Discussion" group.
To post to this group, send email to opensocial-an...@googlegroups.com.
To unsubscribe from this group, send email to opensocial-and-gadg...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/opensocial-and-gadgets-spec?hl=en.

[goosemanjack]

Encapsulation of response meta data is to mitigate the possibility of
a key collision when we load the response, particularly from
<os:HttpRequest> calls where arbitrary JSON data is the result.

In the 0.9 spec we base EL statements on the "entry" key of the
response. This proposal isn't to re-base EL statements at the JSON
response root as that would lead to a cumbersome EL syntax. It's to
provide a reserved key that can expose the upper-level data (and
original request data) to EL statements without making the EL syntax
less natural.

The response in your example would actually have two "count"
equivalent values - the natural one coming off the array (aka
"length"), and the meta-data response value returned by the REST
endpoint. A clearer example would be to use "totalResults" since this
has no natural equivilent.

If we modify the proposal to use the upper case convention for the
reserved properties, the people request would work like this:

ex:
<os:PeopleRequest key="people" ... />

returning response

{
startIndex: 0,
count: 20,
totalResults: 143,
entry: [...]

}

(in 0.9)
${people[0].displayName} -> "John Doe"
${people.Response.totalResults} -> null/undefined at "Response" key

(in 1.0)
${people[0].displayName} -> "John Doe"
${people.Response.totalResults} -> 143

On Nov 10, 12:42 pm, Evan Gilbert <uid...@google.com> wrote:
> On Fri, Nov 6, 2009 at 2:38 PM, goosemanjack <cc...@myspace.com> wrote:
> > Patch created for evaluation
>
> >http://codereview.appspot.com/150052
>
> > I also updated the wiki with the patch contents
>

> >http://wiki.opensocial.org/index.php?title=Define_EL_access_to_meta-d...

> > opensocial-and-gadg...@googlegroups.com<opensocial-and-gadgets-spec%2Bunsu...@googlegroups.com>

[Tosh Meston]

+1

[Evan Gilbert]

This takes the raw JSON response and inverts it - hoisting the entry array to the top level, and pushing all of the other field in the response down under Response. We've found it much easier when the protocol message & the response are 1:1. 

I was OK hoisting entry[] because it's such a common operation, but think that doing this and pushing down the other fields will be confusing to developers.

To unsubscribe from this group, send email to opensocial-and-gadg...@googlegroups.com.

[Henry Saputra]

+1 for keeping the "entry" level of the response.

I would prefer to keep the "entry" key. This will help developers to separate between meta-data of the response object and the actual data.

- Henry

[goosemanjack]

I agree that the inversion of the REST structure (for meta data) when
being referenced within EL is less than ideal, but the alternative of
forcing the developer to constantly dig down into the entry key for
their 80% use cases would be worse. Remember that the developer can
still access the REST response in it's raw format by pulling the
object out of the client-side data context and accessing it through
JavaScript.
--
clc

> ...
>
> read more »

[Evan Gilbert]

On Fri, Dec 18, 2009 at 10:23 AM, goosemanjack <cc...@myspace.com> wrote:

I agree that the inversion of the REST structure (for meta data) when
being referenced within EL is less than ideal, but the alternative of
forcing the developer to constantly dig down into the entry key for
their 80% use cases would be worse.  Remember that the developer can
still access the REST response in it's raw format by pulling the
object out of the client-side data context and accessing it through
JavaScript.

I was suggesting another alternative - hosting the list but leaving the other parameters at the top level as well. 

Difference from your example in bold below:

With response

 {
 startIndex: 0,
 count: 20,
 totalResults: 143,
 entry: [...]
}

(in 0.9)
${people[0].displayName} -> "John Doe"
${people.Response.totalResults} -> null/undefined at "Response" key

(in 1.0)
${people[0].displayName} -> "John Doe"

${people.totalResults} -> 143

[goosemanjack]

The reason I don't like leaving them at the top level is because of
the probability of a key collision, especially when dealing with
arbitrary data coming from an os:HttpRequest that is not OpenSocial
data.

ex: <os:HttpRequest format="json" href="http://example.com/
mystuffpagedlist.svc?foo=x&bar=y&t=furry" key="external" />

yields data:

{
format: "furry",
items: [{object: "doll", type: "teddy bear"}, {object: "doll", type:
"kitty cat"}]
}

And EL statement:

You asked for ${external.format} items

Proposal as it stands evalutates:
${external.format} -> "furry"
${external.Request.format} -> "json"

Your proposed alternative:
${external.format} -> ?? "furry" | "json" (probably resolved with a
"last value set wins" situation)

The proposal as it stands would make it clear what the value of $
{external.format} should be by isolating the meta data away from the
data root. Your proposed alternative would be fine if we were only
dealing with one or two values, but we're dealing with multiple values
on both the Request and the Response, and these values could grow over
time.

Additionally, there is the concern about the synthetic Count variable
identified in the spec. On a <os:PeopleRequest> this results in two
"count" properties - one on the request and one on the response.
These might differ if, for example, 50 people are requested, but only
15 are available.

EX: <os:PeopleRequest key="pagedFriends" userId="@owner"
groupId="@friends" startIndex="51" count="50"/>

given the above only returns 15 results on the second page,

Proposal as it stands evalutates:
${pagedFriends.Count} -> 15
${external.Request.count} -> 50

Your proposed alternative:
${pagedFriends.Count} -> 15
${external.count} -> 50

Having two such similar variables at the root level will be a source
of confusion and bugs for developers.
--
clc

On Dec 18 2009, 5:41 pm, Evan Gilbert <uid...@google.com> wrote:
> On Fri, Dec 18, 2009 at 10:23 AM, goosemanjack <cc...@myspace.com> wrote:
> > I agree that the inversion of the REST structure (for meta data) when
> > being referenced within EL is less than ideal, but the alternative of
> > forcing the developer to constantly dig down into the entry key for
> > their 80% use cases would be worse.  Remember that the developer can
> > still access the REST response in it's raw format by pulling the
> > object out of the client-side data context and accessing it through
> > JavaScript.
>
> I was suggesting another alternative - hosting the list but leaving the
> other parameters at the top level as well.
>
> Difference from your example in bold below:
>
> With response
>  {
>  startIndex: 0,
>  count: 20,
>  totalResults: 143,
>  entry: [...]
>
> }
>
> (in 0.9)
> ${people[0].displayName} -> "John Doe"
> ${people.Response.totalResults} -> null/undefined at "Response" key
>
> (in 1.0)
> ${people[0].displayName} -> "John Doe"

> ${*people.totalResults*} -> 143

> ...
>
> read more »

[Evan Gilbert]

I think we may be agreeing. The line in bold below (from your example) is the one I thought you didn't want to support:

<quote>

EX: <os:PeopleRequest key="pagedFriends" userId="@owner"
groupId="@friends" startIndex="51" count="50"/>

given the above only returns 15 results on the second page,

Proposal as it stands evalutates:
 ${pagedFriends.Count} -> 15

 ${pagedFriends.Request.count} -> 50

</quote>

What I didn't want is to require wrapping the response fields in a Response object:

 ${pagedFriends.Response.Count} -> 15 

If the proposal supports the line in bold above as well as hoisting the entries to the top level of the response, then I'm +1.

[goosemanjack]

What you've said is correct. The only thing under the "Response" key
is the meta-data about the response. They actual data ("entry" label)
and synthetics (currently, I think just Count, but there might be more
I'm missing) are at the top level.
--
clc

On Jan 5, 6:10 pm, Evan Gilbert <uid...@google.com> wrote:
> I think we may be agreeing. The line in bold below (from your example) is
> the one I thought you didn't want to support:
>
> <quote>
>
> EX: <os:PeopleRequest key="pagedFriends" userId="@owner"
> groupId="@friends" startIndex="51" count="50"/>
>
> given the above only returns 15 results on the second page,
>
> Proposal as it stands evalutates:

>  *${pagedFriends.Count} -> 15*

> ...
>
> read more »

[Evan Gilbert]

On Wed, Jan 6, 2010 at 3:50 PM, goosemanjack <cc...@myspace.com> wrote:

What you've said is correct.  The only thing under the "Response" key
is the meta-data about the response.  They actual data ("entry" label)
and synthetics (currently, I think just Count, but there might be more
I'm missing) are at the top level.

I didn't realize Count was a synthetic in your example - thought that was just additional response data that was being left at the top level.

I'm currently at:

+1 for hoisting entry

+1 for putting request parameters in a separate object in the EL

-1 for pushing down existing response data into a Response object

So with the following:

request -> people.get({groupId: '@friends'})

people {
  startIndex: 0,

  count: 20,

  totalResults: 143,

  entry: [...]
}

+1 <div repeat="${people}">${displayName}</div> -> Shows every person's name

+1 on ${people.Request.groupId} -> '@friends'

-1 on ${people.Response.count} -> 20 - think this should remain ${people.count}

[goosemanjack]

OK. I think we're mostly in agreement. I tried to find the reference
to a synthetic "Count" variable being attached in the EL, but it must
have been in a very early version of the spec, because the text I'm
recalling is not to be found.

From your example, I would say we get "length" at the root as a
natural value, and the "count" value will hold the same value, but be
attached to the Response.

<os:PeopleRequest key="people" groupId="@friends" />

${people.Request.count} -> undefined, in this case
${people.length} -> 20 (as a natural property of the array)
${people.Response.count} -> 20 (as part of the response meta data)

Is that acceptable, Evan? I only split hairs on the count item
because the value is already available in "length" and I'd rather not
be cherry-picking meta-data items to hoist without a compelling case.
--
clc

> ...
>
> read more »

[goosemanjack]

I've updated the patch to use reserved keys "Request" and "Response"
per convention instead of _request and _response.

http://codereview.appspot.com/150052

I've not added an explicit statement RE: hoisting count/length pending
your comments, Evan.

We could change the sample line to include an explicit example:

(now)
You requested ${friends.Request.count} records.

(could be)
You requested ${friends.Request.count} records and got $
{friends.length} back.
or
You requested ${friends.Request.count} records and got $
{friends.Count} back.

My preference would be for ${friends.length} since this is a natural
value (at least on the JavaScript side of the EL implementation).
--
clc

> ...
>
> read more »

[Evan Gilbert]

On Thu, Jan 7, 2010 at 10:54 PM, goosemanjack <cc...@myspace.com> wrote:

OK.  I think we're mostly in agreement.  I tried to find the reference
to a synthetic "Count" variable being attached in the EL, but it must
have been in a very early version of the spec, because the text I'm
recalling is not to be found.

From your example, I would say we get "length" at the root as a
natural value, and the "count" value will hold the same value, but be
attached to the Response.

<os:PeopleRequest key="people" groupId="@friends" />

${people.Request.count} -> undefined, in this case
${people.length} -> 20 (as a natural property of the array)
${people.Response.count} -> 20 (as part of the response meta data)

Is that acceptable, Evan?  I only split hairs on the count item
because the value is already available in "length" and I'd rather not
be cherry-picking meta-data items to hoist without a compelling case.

I was suggesting that we don't need the "Response" object, only the "Request" object. It seems natural in this case for the collection to support the additional metadata fields (${people.startIndex} and ${people[0]} wold both work}, and it's a relatively minor implementation issue on our side to make it work 

Any other opinions out there on the spec list?

 

[Lane LiaBraaten]

I agree with Evan...I see the benefit of 'result' having a "Request" property, but I think the metadata about the response should remain as properties of 'result' and not be put under 'result.Response'

This is currently in the spec draft (e.g. http://www.opensocial.org/Technical-Resources/draft/Core-Data.xml#Collection), but we don't have the 'Request' property documented.  I guess we'd need to add it here: http://www.opensocial.org/Technical-Resources/draft/Core-Data.xml#rfc.section.3.8

-Lane

[Lev Epshteyn]

+1 on keeping response data at the top level. Accessing this is the 95% use case, we shouldn't make it harder by forcing people to type 9 extra characters every time.

The seeming symmetry between "Request" and "Response" doesn't hold up here, given the relative usage frequency of each. Response is by far the more important piece of that package, and should be easily accessible.

[goosemanjack]

I pulled the trigger a bit early when applying the patch for RC1. I'm
OK with nixing Response and placing the three values as reserved root
keys:

StartIndex
TotalResults
Count

As there are reserved values I'm assuming we should use the initial
Uppercase letter convention. I'll update the RC1 committed code with
this change.
--
clc

On Jan 12, 10:17 am, Lev Epshteyn <le...@google.com> wrote:
> +1 on keeping response data at the top level. Accessing this is the 95% use
> case, we shouldn't make it harder by forcing people to type 9 extra
> characters every time.
>
> The seeming symmetry between "Request" and "Response" doesn't hold up here,
> given the relative usage frequency of each. Response is by far the more
> important piece of that package, and should be easily accessible.
>

> On Tue, Jan 12, 2010 at 12:40 PM, Lane LiaBraaten <lliab...@google.com>wrote:
>
> > I agree with Evan...I see the benefit of 'result' having a "Request"
> > property, but I think the metadata about the response should remain as
> > properties of 'result' and not be put under 'result.Response'
>
> > This is currently in the spec draft (e.g.

> >http://www.opensocial.org/Technical-Resources/draft/Core-Data.xml#Col...),

> > but we don't have the 'Request' property documented.  I guess we'd need to
> > add it here:

> >http://www.opensocial.org/Technical-Resources/draft/Core-Data.xml#rfc...

> ...
>
> read more »

[Lane LiaBraaten]

I'd prefer to keep the existing lower-case values:

startIndex

totalResults

count

itemsPerPage

sorted

filtered

Not only does this preserver backward compatibility, it maintains alignment with PoCo.

-Lane

[goosemanjack]

OK. Updated for RC1 as outlined above

> ...
>
> read more »

[goosemanjack]

I'm not thrilled with having so many keys added at the root. Looking
over the poco and OpenSearch specs, tho, I see what you mean. If
we're going to do this I'd like the spec to explicitly call out these
values as reserved keys so that devs don't create external JSON
objects that use any of the same keys.

I'm going to completely nix the "Response" key as you propose, then
add a draft section under EL (7.4) that clearly identifies these as
reserved keys under the EL.
--
clc

> ...
>
> read more »

[goosemanjack]

I've committed a change to template spec:

Modifying casing for EL meta-data access, addition of draft "reserved
properties" section for EL

FYI, the x:draft element causes the spec to no longer display in a
browser re: XML Parsing Error: prefix not bound to a namespace
--
clc

> ...
>
> read more »

[Lane LiaBraaten]

On Tue, Jan 12, 2010 at 4:40 PM, goosemanjack <cc...@myspace.com> wrote:

I've committed a change to template spec:

Modifying casing for EL meta-data access, addition of draft "reserved
properties" section for EL

FYI, the x:draft element causes the spec to no longer display in a
browser re: XML Parsing Error: prefix not bound to a namespace

Thanks for the heads up.  The OpenSocial-Templating file was missing a namespace declaration.  I've checked in a fix, so if you update you should see the <x:draft> tag render properly.

Cheers,

Lane

[Lane LiaBraaten]

Hi Chris,

Sorry I didn't catch this earlier, but patch 150052 and 143063 have changes to the OpenSocial-Data-Pipelining.xml file.  The content of this file has been moved to Core-Gadget and Social-Gadget, though I think your changes apply only to Core-Gadget.xml (See the Data Pipelining section).

Can you reapply the changes to Core-Gadget.xml?  You can also go ahead and delete OpenSocial-Data-Pipelining.xml now that its content has been relocated.

Sorry for the mixup,

Lane

http://codereview.appspot.com/150052

http://codereview.appspot.com/143063

[goosemanjack]

I've applied 143063 (case sensitivity) to Core-Gadget.xml, but 150052
(EL meta info access) is a problem because a good deal of the Data
Pipeline spec does not seem to have made it across to the new spec
structure. Specifically:

* Dynamic Parameters
* tag <os:ViewerRequest>
* tag <os:PeopleRequest>
* tag <os:OwnerRequest>
* tag <os:ActivitiesRequest>

I'll look over the other threads, but I don't recall any discussion on
the spec list of dropping these features.
--
clc

On Jan 13, 9:26 am, Lane LiaBraaten <lliab...@google.com> wrote:
> Hi Chris,
>
> Sorry I didn't catch this earlier, but patch 150052 and 143063 have changes
> to the OpenSocial-Data-Pipelining.xml file.  The content of this file has
> been moved to Core-Gadget and Social-Gadget, though I think your changes
> apply only to Core-Gadget.xml (See the Data Pipelining section).
>
> Can you reapply the changes to Core-Gadget.xml?  You can also go ahead and
> delete OpenSocial-Data-Pipelining.xml now that its content has been
> relocated.
>
> Sorry for the mixup,
> Lane
>
> http://codereview.appspot.com/

> <http://codereview.appspot.com/150052>150052<http://codereview.appspot.com/150052>

> ...
>
> read more »

[Lane LiaBraaten]

I've got a patch for moving the Dynamic Parameters section:

http://codereview.appspot.com/189069/show

Please let me know if you find other content that needs to be added to Core-Gadget.xml (e.g. EL Statements?).

The tags you mention are already in the Social-Gadget.xml spec:

http://www.opensocial.org/Technical-Resources/draft/Social-Gadget.xml#rfc.section.4

Thanks,

Lane

[goosemanjack]

I found a casing error in core-gadget. This patch changes from
Initial Upper to initial lower case as agreed to:

from: reserved keys "TotalResults", "StartIndex", and "Count".
to: ...reserved keys "totalResults", "startIndex", and "count".

Patch updated:
http://codereview.appspot.com/150052

I can commit this, if that's appropriate. Also, the dynamic
parameters patch looks good.

On Jan 14, 7:46 am, Lane LiaBraaten <lliab...@google.com> wrote:
> I've got a patch for moving the Dynamic Parameters section:http://codereview.appspot.com/189069/show
>
> Please let me know if you find other content that needs to be added to
> Core-Gadget.xml (e.g. EL Statements?).
>

> The tags you mention are already in the Social-Gadget.xml spec:http://www.opensocial.org/Technical-Resources/draft/Social-Gadget.xml...

> ...
>
> read more »

[Lane LiaBraaten]

Looks good.

On Jan 20, 2010 2:00 PM, "goosemanjack" <cc...@myspace.com> wrote:

I found a casing error in core-gadget.  This patch changes from
Initial Upper to initial lower case as agreed to:

from:   reserved keys "TotalResults", "StartIndex", and "Count".
to:  ...reserved keys "totalResults", "startIndex", and "count".

Patch updated:

http://codereview.appspot.com/150052

I can commit this, if that's appropriate.  Also, the dynamic
parameters patch looks good.

On Jan 14, 7:46 am, Lane LiaBraaten <lliab...@google.com> wrote:

> I've got a patch for moving the...

> The tags you mention are already in the Social-Gadget.xml spec:http://www.opensocial.org/Technical-Resources/draft/Social-Gadget.xml...
>
> Thanks,
> Lane

>
> On Wed, Jan 13, 2010 at 10:37 PM, goosemanjack <cc...@myspace.com> wrote:

> > I've applied 14306...

> ...
>
> read more »

--
You received this message because you are subscribed to the Google Groups...