[Proposal] Data Pipelining

[Evan Gilbert]

Here's an updated proposal for data pipelining in OpenSocial: http://docs.google.com/View?docid=dc5qrdst_19fbb8fpd3

Brief overview:
Data Pipelining is a declarative syntax for defining the data you want the container to provide to you for processing. Examples would be <os:PersonRequest id="@viewer"/> or <os:DataRequest href="http://www.developer.com/api">.

The data that is retreived will be available in three contexts:
1. Proxied content requests. This data will be POSTed to the developer party server with requests for proxied content.
2. OpenSocial Templates. The data will be available as named variables in OpenSocial templates
3. JavaScript. This data will be available in the JavaScript API

There are five initial data request tags...

Four tags equivalent to RESTful get calls:
<os:PersonRequest> - request to get a single person's profile
<os:PeopleRequest> - request to get profile information for a group or list of people
<os:ActivitiesRequest> - request to get activities
<os:PersonAppDataRequest>. A request for app data

And one tag that is for requesting arbitrary content:
<os:DataRequest>, equivalent to gadgets.io.makeRequest

Integration with JavaScript and OpenSocial templates is discussed in detail in this document. Kevin Brown is going to update the proxied content proposal (http://groups.google.com/group/opensocial-and-gadgets-spec/browse_thread/thread/c67095188a3614f0/) to discuss how Data Pipelining works with proxied content and I'm going to also send a follow-up (& separable proposal) for type="osml" content.

Here's the document link again: http://docs.google.com/View?docid=dc5qrdst_19fbb8fpd3.

Thanks,
Evan

[Chris Chabot]

About the proposed single vs plural calls in the tag list that's mentioned:

• RESTful call equivalents:

1. <os:PersonRequest> - request to get a single person's profile
2. <os:PeopleRequest> - request to get profile information for a group or list of people
3. <os:ActivitiesRequest> - request to get activities
   
4. <os:PersonAppDataRequest>. A request for app data
   

Why do we chose to give PersonAppData (which is a single form, so for a single user?) a user + group id (@owner + @friends = plural, no?), while the person request has 2 calls, and the plural form with @viewer, @self would give a singular response? That's how it's working for people data in the REST spec anyhow.

With activities it's the same kind of 0deal, if you parse it a single ID in the @activitiesIds you get a straight activity record, if you specifiy a collection (either by giving multiple activity id's or specifying a @owner/@self type request, you get a collection back).


Something I think might confuse developers down the road

This works:
<Content type="osml">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>

This doesn't:
<Content type="html">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>

But this does (but behaves differently):
<Content href="http://foo/bar">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>

I would almost be inclined to suggest that type="osml" becomes type="html", and on gadget parsing we do a quick gadgetContent.indexOf('<os'): to see if we need to do any dynamic parsing? I think that would make things a good bit easy to use for dev's. Or if people see some huge problems with this, how about adding a new 'osml="true"' to the type="html" statement (I would prefer not having to add any tags though to prevent any possible confusion)

Likewise i'm also not 100% clear yet on what something like this should do:

<Content href="http://foo/bar">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>
   <b>Hello {$vf.Name}</b>

Would the parsed html content be posted to the url, the unparsed html be posted, or would it raise an error or be ignored?

I am quite happy to see consistency between the REST, osml, data pipelining and js templating btw, if that wasn't the case it would've been a nightmare in the making :)

   -- Chris

[Chris Chabot]

Ah sorry for the double post, but my app-dev-pov itch tells me this will happen too:

<Content href="http://foo/bar">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>

http://foo/bar returns:
..
<div repeat="${vf}">${Name} is a friend.</div>   
..

and when that fails people might try something like:

<Content href="http://foo/bar" type="osml">

  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>

And in a way I could really understand the use-case of this, say that a gadget dev is experienced with OSML and he loves it to show things like the owner name and info (and he has lots of osml code he can cut and paste), but he wants to create a custom html blob on the server that lists the owner's favorite songs, so he would want to output something like:

<b>${vwr.Name}'s favorite songs:</b>
<table>... lots of server side generated html </table>

In the current proposal I don't see a way for this to be possible, while from an app dev point of view I wouldn't understand why not :)


Something that's also not covered by this proposal is how this will work in contexts other then html. We've spend some time educating app devs that this works:

<i>This is app version __MSG_versionString_, checking to see if its new enough..</i>
<script>
if ( __MSG_versionString_ < 2) {
alert('this is an old app!');
}
</script>

However how would this work with OSML'd tags?
<b>Hello ${vwr.Name}</b>
<script>
if ("${vwr.Name}" == "") {
alert('I don't like anonymous users, shoo!');
}
</script>

Since the tags are no longer inside of <script type="text/os-template"> blocks, wouldn't people expect them to be available anywhere in the content block, just like messages?

The proposal doesn't give me insight currently on what the intentions are behind this.


The more I think about it, the more I feel it might be a good idea to make osml completely pervasive, ie parse osml tags on type="html", on returned href="http://foo/bar" content, and even allow the tags to be used outside of html content (as in the example: in <script> blocks inside of the content)

Am I nuts to think so? :)

   -- Chris

[Kevin Brown]

I don't believe so. I've preferred that OSML be a pure text pre-processor (in the vein of PHP, JSP, or other templating systems). Where this breaks down is the inclusion of programmatic attributes in OSML itself. Eliminating attributes like if and repeat on arbitrary HTML elements makes this work.

[Kevin Brown]

On Thu, Nov 6, 2008 at 4:27 AM, Chris Chabot <cha...@google.com> wrote:

About the proposed single vs plural calls in the tag list that's mentioned:

• RESTful call equivalents:

1. <os:PersonRequest> - request to get a single person's profile
2. <os:PeopleRequest> - request to get profile information for a group or list of people
3. <os:ActivitiesRequest> - request to get activities
   
4. <os:PersonAppDataRequest>. A request for app data
   

Why do we chose to give PersonAppData (which is a single form, so for a single user?) a user + group id (@owner + @friends = plural, no?), while the person request has 2 calls, and the plural form with @viewer, @self would give a singular response? That's how it's working for people data in the REST spec anyhow.

With activities it's the same kind of 0deal, if you parse it a single ID in the @activitiesIds you get a straight activity record, if you specifiy a collection (either by giving multiple activity id's or specifying a @owner/@self type request, you get a collection back).

I agree with this. I think it's confusing to have singular and plural forms of some things and not others. Why not all one way or the other?

 

Something I think might confuse developers down the road

This works:
<Content type="osml">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>

This doesn't:
<Content type="html">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>

But this does (but behaves differently):
<Content href="http://foo/bar">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>

I would almost be inclined to suggest that type="osml" becomes type="html", and on gadget parsing we do a quick gadgetContent.indexOf('<os'): to see if we need to do any dynamic parsing? I think that would make things a good bit easy to use for dev's. Or if people see some huge problems with this, how about adding a new 'osml="true"' to the type="html" statement (I would prefer not having to add any tags though to prevent any possible confusion)

The argument for type="osml" (and yes, this could just be a new attribute, but I don't see the point) is that it's the only safe way to seamlessly move from client side to server side processing without requiring additional script tags.

In discussions about this, Evan has voiced a strong opinion that OSML should be able to work in a standalone HTML document. I disagree with this requirement (you need a container anyway for lots of other things), but that's the main argument. Without that requirement, the difference is irrelevant. This distinction should probably be explained better in the proposal.
 

Likewise i'm also not 100% clear yet on what something like this should do:

<Content href="http://foo/bar">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>
   <b>Hello {$vf.Name}</b>

Would the parsed html content be posted to the url, the unparsed html be posted, or would it raise an error or be ignored?

Given our lax parsing requirements for "unknown" tags in other XML contexts, it should be ignored, since that HTML tag isn't a part of the data pipelining subset of OSML.
 

[Scott Seely]

Can we eliminate the need for OSML to function correctly outside of a container?

[Chris Chabot]

Just to verify, I assume the inspiration for including programmatic tags is for constructions like (forgive the syntax im making up on the fly which is just to clarify and does not resemble real templating code)

<os:hasApp>
Woot you have my app already
<os:else>
Please install this way cool app
</os:hasApp>

which is the type of constructions which i think are quite popular in fbml.. but i'm no expert in either of these 2 assumptions, so if someone could expand on the reasoning behind this that would be swell :)

[Chris Chabot]

Could you please clarify your question Scott?

I mean, if it's not in a container, why would OSML be useful at all? So my best guess at the meaning of your question, I don't see why that need would exist at all :)

  -- Chris

[Chris Chabot]

<Content href="http://foo/bar">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>
   <b>Hello {$vf.Name}</b>

Would the parsed html content be posted to the url, the unparsed html be posted, or would it raise an error or be ignored?

Given our lax parsing requirements for "unknown" tags in other XML contexts, it should be ignored, since that HTML tag isn't a part of the data pipelining subset of OSML.

It's still a bit funny though to have a 'content' block that has no content, and that just kind of disappears into the great /dev/null.

Are there any use-cases imaginable where posting this (parsed) content to the app dev's server would be useful?

If not we could just leave it as a 'funny anecdote' of design (to have content blocks that can't contain content), but otherwise it might be wise to spec it

  -- Chris

[Lev Epshteyn]

I think this is a great starting point, Evan. One thing that I'd like to see clarified is the

opensocial.data.DataContext.putDataSet(key, value) - the proposal says @value is a JSON object. Does this specifically exclude other types of objects (for example Opensocial JS API objects, which have methods in addition to JSON properties)?

[Chris Chabot]

On Thu, Nov 6, 2008 at 5:32 PM, Kevin Brown <et...@google.com> wrote:

chabotc wrote: I would almost be inclined to suggest that type="osml" becomes type="html", and on gadget parsing we do a quick gadgetContent.indexOf('<os'): to see if we need to do any dynamic parsing? I think that would make things a good bit easy to use for dev's. Or if people see some huge problems with this, how about adding a new 'osml="true"' to the type="html" statement (I would prefer not having to add any tags though to prevent any possible confusion)

The argument for type="osml" (and yes, this could just be a new attribute, but I don't see the point) is that it's the only safe way to seamlessly move from client side to server side processing without requiring additional script tags.
 

In discussions about this, Evan has voiced a strong opinion that OSML should be able to work in a standalone HTML document. I disagree with this requirement (you need a container anyway for lots of other things), but that's the main argument. Without that requirement, the difference is irrelevant. This distinction should probably be explained better in the proposal

I presume we expect the opensocial javascript API to function inside of a social gadget at any time right, even if a good part of it's content is parsed on the server? Also things like preloading require modifications of the html document by the container, so the container modifications will remain a hard requirement i would imagine.

Or is type="osml" going to be lacking in other functionalities, current and future? I would think that's a horrible direction to go in!

Personally I think consistency and simplicity are the 2 most powerful features a API that is supposed to be consumed by many can have, I would take consistency and simplicity over more features any day.

Having OSML tags being available anywhere, in the same old type="html" block as people are using now is a very powerful thing; Having them parsed in the response of a proxied content block is also a very powerful and probably will be a much used thing..

Evan can you please outline why this requirement that is blocking this is so important? I want to give you all the benifit of the doubt, but without this information it's hard to see why we have to give up so much good stuff :)

   -- Chris

[Kevin Brown]

Your example is exactly what I think we should be supporting. What I disagree with is:

<div if="{$...}">hello</div>

Which makes treating OSML as a text processing language more or less impossible.

[Kevin Brown]

On Thu, Nov 6, 2008 at 9:46 AM, Chris Chabot <cha...@google.com> wrote:

On Thu, Nov 6, 2008 at 5:32 PM, Kevin Brown <et...@google.com> wrote:

chabotc wrote: I would almost be inclined to suggest that type="osml" becomes type="html", and on gadget parsing we do a quick gadgetContent.indexOf('<os'): to see if we need to do any dynamic parsing? I think that would make things a good bit easy to use for dev's. Or if people see some huge problems with this, how about adding a new 'osml="true"' to the type="html" statement (I would prefer not having to add any tags though to prevent any possible confusion)

The argument for type="osml" (and yes, this could just be a new attribute, but I don't see the point) is that it's the only safe way to seamlessly move from client side to server side processing without requiring additional script tags.
 

In discussions about this, Evan has voiced a strong opinion that OSML should be able to work in a standalone HTML document. I disagree with this requirement (you need a container anyway for lots of other things), but that's the main argument. Without that requirement, the difference is irrelevant. This distinction should probably be explained better in the proposal

I presume we expect the opensocial javascript API to function inside of a social gadget at any time right, even if a good part of it's content is parsed on the server? Also things like preloading require modifications of the html document by the container, so the container modifications will remain a hard requirement i would imagine.

Or is type="osml" going to be lacking in other functionalities, current and future? I would think that's a horrible direction to go in!

Just to be clear, the only difference between type="osml" and type="html" is that in type="html" OSML isn't natively supported -- you have to add <script type="text/os-template"> around the OSML parts.
 

[Scott Seely]

Nevermind—shouldn’t have hit send…

[Chris Chabot]

On Thu, Nov 6, 2008 at 6:59 PM, Kevin Brown <et...@google.com> wrote:

Just to be clear, the only difference between type="osml" and type="html" is that in type="html" OSML isn't natively supported -- you have to add <script type="text/os-template"> around the OSML parts.

 

The use case i have in mind that we have Joe the Webdesigner (not related to the plumber i hope), he doesn't dig javascript as much as we do but he knows how to use a cms like system, and knows <web dev language FOO> and html.

So he would create a gadget with something like:

<Content url="http://joewebdesigner.com/myapp.php">
<os:PersonRequest key="own" userId="@owner" groupId="@self">

<os:PersonRequest key="vwr" userId="@viewer" groupId="@self">
</Content>

and a myapp.php file with something like:

<b>Welcome ${vwr.Name}, these are {$own.Name}'s favorite pictures:</b>
<?php
.. do a  "select * from pictures where userId = {$_POST['own']['id']}" type thing..
foreach ($result as $picture) {
  echo "<img src="$picture"><br>";
}
?>
Want to have this too? Click here:
<os:addAppLink>

Now i realize the example is terribly lame and simplified, but I presume we'll see many of thousands of gadgets that will look very much like this, it's quick to make, simple, understandable, doesn't require a lot of knowledge and allow anyone with some web dev background to be productive and also use any of the super-duper tags we'll come up with in the future like <os:hasApp>, <os:InviteFriends> etc, and just mix and match whatever he's most comfertable with.

Now with this use-case in mind it's hard to imagine why requiring <script type="text/os-template"> blocks would be required.. nor why we would have to educate people on the difference between type="osml" and type="html", and where osml might or might not be parsed.

[Kevin Brown]

I agree that they shouldn't -- I'd prefer to not have type="osml" and also not require the script tags. Evan is the only person I know of so far who has voiced support for the need for the script tags (in order to allow OSML to work outside of a gadget context), but others may feel the same way.

My take is that if you're writing a gadget, it's assumed that you need a container to process it, period.
 

[Chris Chabot]

On Thu, Nov 6, 2008 at 8:47 PM, Kevin Brown <et...@google.com> wrote:

I agree that they shouldn't -- I'd prefer to not have type="osml" and also not require the script tags. Evan is the only person I know of so far who has voiced support for the need for the script tags (in order to allow OSML to work outside of a gadget context), but others may feel the same way.

If so, speak up now guys, if there are strong arguments to be made for requiring type="osml" and script tags, please share them!
 

My take is that if you're writing a gadget, it's assumed that you need a container to process it, period.

Big +1 on this too

[Lev Epshteyn]

A big +1 on preserving a distinction between type="html" with <script> tags and type="osml".

There are a couple of reasons for it:

• OSML needs to be well-formed XML, which regular HTML does not. Making all HTML be treated as OSML would require even developers who do not need these features to turn their <br>s into <br/>s, etc. This will break a number of apps.
• Content of type="osml" will be treated as one big template. That is not always what you want. In fact, if you are creating an AJAX-type application, you will likely want to define a number of templates/tags and invoke them via JS or other mechanisms. With type="osml", you'd need to define such templates inside of a bigger template, which is awkward.

The second point is much more important - you really can't do complex UI interactions without being able to use markup in a modular manner, and not all applications will work with having these modules only be view-granular.

type="osml" is for a fairly simplistic use case where you want your application to render once with a certain set of data and where UI interactions all result in View navigation. If you need more granularity than that, you will need to define many OSML templates and create some code to render them as needed. We cannot be cutting off the ability to do complex UIs for the sake of simplifying the simple case. By providing both osml and html types we enable both.

[Chris Chabot]

Why would OSML required well formed xml?

When i look at one of the possible implementation paths to use in php, namely libxml2, it fully supports not-well-formed html documents.. libxml has at times even been used as a back-end for actual web browsers, so i'm moderately confident that that should hold up well enough in practise.

I presume the same is possible in other envirioments too?

Also when your working on the basis of 'proxied content', it already implies rewriting of links, so complex navigation could be as simple as <a href="myapp.php?foo=bar">something complex</a>

And sure you could put <script type="text/os-template"> blocks in your content if you wanted too, and use them dynamically from javascript even if you just treat type="html" as type="osml", I don't understand the assumption that this would be cut off?

But lets also be aware that most app devs who will benifit the most from this model is the guys who want to just 'make a web app' and don't 'do javascript'

Also let us not forget that we plan to overload the view="foo" param with sub-sections (view="canvas.Settings", ="canvas", ="canvas.ComplexAction") and that combines really well with either plain osml based content, and proxied content.. so that would also allow for more complex interactions without requiring a single line of javascript.

In that perspective I strongly disagree that type="osml" is only for simple apps without a complex UI .. either through multiple content sections for simpler apps, or link rewriting for more complex apps, either way no javascript and template blocks need to be involved.

[Chris Chabot]

ps, caja and other such features (such as content rewriting) work perfectly well on plain-old-mallformed-html for both versions of shindig currently.

I would think that if you can parse the dom structure, you can parse template tags in it right?

[John Hayes]

Lev,

 

I think the use case of rendering a page once with a certain set of data is actually the one that's most like an ordinary web page, and dividing up templates into reusable components is both common practice today for ajax applications and neccessary to do fine-grained updates as part of a type="html" document.

 

I think an important process for HTML developers is to:

 

1. Make an ordinary HTML document

2. Add a tag to begin merging in some social data

3. Make a more complex ajax application

 

For the current Gadget developer, they start with a complex ajax application to integrate any social data:

 

1. Look at your current gadget

2. Delete a dataRequest and replace text concatenation code with template

3. Breath a sigh of relief and build a more complex ajax application

 

I want to throw out another suggestion, if you include a <Require feature="osml"/> push both the Content bodies or the proxied response through the Data Pipeline interpreter. Then eliminate the @type attribute.

 

John

[Chris Chabot]

On Fri, Nov 7, 2008 at 2:51 AM, John Hayes <john.mar...@gmail.com> wrote:

I want to throw out another suggestion, if you include a <Require feature="osml"/> push both the Content bodies or the proxied response through the Data Pipeline interpreter. Then eliminate the @type attribute.

that would work for me, simplicity and consistency, winning attributes in my book :)

[David Byttow]

I am also a strong advocate of allowing client-side script tag templates in type=html. I have relied on this heavily in my own apps. This was important for developing applications that were both standalone and gadget-based (http://ostemplates-devapp.appspot.com/), yet managed to utilize templates (both server-side django and client-side osml).

It has been a breath of fresh air to me knowing that my templates will work across all containers today. And using custom, modular templates to show/hide/re-render based on AJAX operations is a pattern that I leverage often.

I think moving yet another feature onto the server and requiring a container to process them is a mistake. However, I do like the idea of allowing some server implementations to pre-process certain templates or markup for performance.

In the case of one-off static tags such as <os:hasApp>, can we distinguish between the concept of templating/data-binding as we have today and a future brand of server-side OSML (like fbml) which is always processed on the server.

[Chris Chabot]

Hey David,

It might be the fact that I have not yet had enough coffee today (and hence violating the internation don't-post-before-2-coffee's rule), but i am not a 100% yet on how to place all of your remarks in the context of the proposals.

What's on http://ostemplates-devapp.appspot.com/ was a proposal that hasn't made it to spec yet, and as such it's not yet a part of open social, it's a very well made, usable proposal mind you, but not set in stone. Likewise data-pilelining and proxied content are not set in stone yet either.

Also what's currently in ostemplates is intended for gadget/client side use only, and that means javascript.. however don't forget that creating something for 'the rest of the developers' who don't dig javascript is the main thing that got this discussion started, there's countless of devs out there who want to create an OpenSocial app (or port it from an existing fb app), their used to either a standard CMS system and/or programming in FBML, and they get stuck on the JavaScript based API that OpenSocial uses..

Making  the JavaScript API more powerful is *not* a solution for this problem space, and loosing track of that would mean loosing sight of the very goal that we set out to reach, so while I'm not against making the end proposal work as well as possible from that use-case point of view as well, it's not my personal main motivation in these discussions.

With that in mind, isn't this the perfect opportunity to see if we can combine these concepts into one , consistent, well working , thought out solution? As such I find a blanket statement saying that we should punt server sided rendering to the future to 'another type of OSML', a less then optimal situation, I would never want to advocate having 2 or 3 types of solutions for the same problem space, that only confuses developers and increases the pressure on containers who have to implement it.

[David Byttow]

I agree with what you are saying. But, from reading this thread (and perhaps I'm interpreting things wrong), it seemed like some of the arguments were entering the client-side space and wanting to cut out certain features such as content html <script> template tags, which I am very much supportive of.

It was also mentioned on this thread that Evan was the only voice for supporting OSML outside of gadgets, so I wanted to back up his opinions publicly.

As a general virtue, you're right that we should not create "another type of OSML", but what I was saying is that it is becoming increasingly difficult to maintain feature parity with OSML that is to be processed on either the client or the server, so in my opinion, it might be okay to allow for certain constructs to be supported and processed by the server on returned content (as per development model that you're describing) outside of <script> tags.

David

[Evan Gilbert]

Hi all - thanks for the great feedback. I'm first going to answer all of the easy issues - going to follow up shortly on the type="osml" vs "html" issue.

I've tried to capture all of the points raised in the thread above - please add your issue if I missed it.

<os:PersonAppDataRequest> - why "Person" and not "People"?
This was to match the existing JS API call - also the "Person" here is really used as an adjective to describe the type of app data. I don't feel strongly about naming here.

Why <os:PersonRequest> and <os:PeopleRequest> but not singular & plural for other requests
Having both <os:PersonRequest> and <os:PeopleRequest> seems useful, as these are very common calls and <os:PeopleRequest id="@viewer"> just doesn't feel right.
The singular of activity and app datum are much less frequently used - I'm happy to add if people think they would add value.

<Content href="foo"> with HTML content could be confusing
I believe the idea here is not to POST the data to the 3rd party server, but instead to use this as default content for the view while the page is loading. The full discussion for this should be on the proxied content thread..

<Content type="html"><os:PersonRequest> wouldn''t work with type="html"
I'll bundle this in with the type="osml|html" discussions.

Returning OSML content from proxied requests - what happens?
Discussion was that that it might be confusing if the following wouldn't work:

<Content href="http://foo/bar">
  <os:PersonRequest key="vf" userid="@viewer" groupid="@friends"/>

http://foo/bar returns:
..
<div repeat="${vf}">${Name} is a friend.</div>   

Think that the content returned from proxying should be processed with the same semantics as type"html|osml" depending on the content type of the response. The data request for the outbound side should be kept and used for reprocessing the response if it contains content. This still leaves open the <script type="text/os-template"> discussion, but I think covers most of the otther caess

Templates in <script> tags - supported?
This is a potentially useful case but there were concerns both about implementation as well as it leading to bad developer practices (by making dynamic functions instead of passing dynamic values into functions). Practically I'd like to see good use cases that can't be solved in other ways before adding to the spec.

opensocial.data.DataContext.putDataSet(key, value) - value should be not just JSON
Yep, should support any object. Added and also noted that getDataSet() also may return an object.

Attributes vs. elements for OSML tags
I'm going to defer this to a separate thread.

Content type="osml|html"
The elephant in the room - I'll have a separate post on this shortly

[Evan Gilbert]

On type="html" vs type="osml"...

The biggest benefit of allowing OSML to work in standalone HTML content is that it works on all containers, including those that only support 0.7 or 0.8, and those who might not want to implement or support content rewriting.

We have a JS feature in Shindig that works today on almost all containers by using <script src="">. It took me about 20 minutes to get the Super Lame Game (my simple template app) working on Ning, which is an 0.7 container.

Here's my detailed list of pros & cons...

Pros (in order of importance)

1. Can be used on all containers, today.
• The only exception is that we have to make it work with Cajoling for containers that require Caja.
2. Does not require containers to parse HTML body content.
• Although there are parts of the OpenSocial spec (hangman substitution is one) that need containers to go through the entire content of the page, I believe that many non-Shindig implementations do not support this.
• Also, OSML would require a more complex parser than just a string replacer. This is more work to implement and more CPU cycles on the server.
  
• Because of these, I'm not sure if OSML would ever be universally supported across OpenSocial containers if it required server side processing.
  
3. Can be tested without a gadget container
• It's very useful to be able to test without running a container. Especially with <Content href="foo.html"> coming out, developers will be able to run their HTML testing without needing a gadget container.
• The only difficult part to replicate is data access, and we've designed the OSML spec to allow for passing JSON in instead of the OpenSocial data. This means that you can easily provide fake data in a dev environment.
• I'll send out a versoin of the Super Lame Game later today or this weekend showing how this works.
• The alternative is requiring developers to publicly expose their "in-progress" features on the internet or downloading a Shindig binary.
• Both of these are feasible, but are not free for developers and eliminating these steps provides value.
4. Compatible with standalone HTML pages
• One of the big goals of the REST API is to allow for an OAuth flow where the developer writes pages on their own web site. This is not fully supported yet but should be coming soon.
  
• Many of these pages share UI functionality with the gadget versions of the applications.
• By providing a JS library, developers can use templates in content that is visible both on their gadget and on their own web site.

Cons

1. Requires adding two <script> tags
   

So... I like the idea of adding type="osml" for pages that require server side processing. I wouldn't support this for <script> tags alone, but this will allow us to experiment with a more optimized syntax in a number of areas.

I do want to mention that using embedded <script> tags was already agreed to as part of the original templates design @ http://groups.google.com/group/opensocial-and-gadgets-spec/browse_thread/thread/21e0e6848d4134b7/8d48c6a731866795.

It's completely valid to re-raise issues before we release but I'd encourage people to use the agreed upon design as a starting point for discussions. We're very close to being able to release an implementation based on this proposal, and I had though we were at the point of discussing syntax details, not major design issues.

I'd strongly encourage people who have interest in the template spec to:
1. Get involved early and often. <script type="text/os-template"> has been in place for a whle and I'm surprised it comes as a surprise.
2. Write some apps using the existing demos @ http://ostemplates-demo.appspot.com/. It's really important to build apps to better understand the developer experience. It took about a day for me to get used to writing <script type="text/os-template">.

Hope this hopes clarify the goals of supporting OSML inside HTML content.

Cheers,
Evan

[Chris Chabot]

On Fri, Nov 7, 2008 at 5:18 PM, Evan Gilbert <uid...@google.com> wrote:

On type="html" vs type="osml"...

The biggest benefit of allowing OSML to work in standalone HTML content is that it works on all containers, including those that only support 0.7 or 0.8, and those who might not want to implement or support content rewriting.

We have a JS feature in Shindig that works today on almost all containers by using <script src="">. It took me about 20 minutes to get the Super Lame Game (my simple template app) working on Ning, which is an 0.7 container.

Ok please let interject here... Your saying we should cater to containers who do not want to upgrade to the latest spec revision, or choose not to support a MUST part of the OpenSocial specification?

I had a longer response all typed up, but really there is nothing to add to this question ...

To the first point, we always presume containers will upgrade, we presume gadget devs will write gadges that <Require feature="opensocial-0.8"/> and use all the new goodies we put into each new spec.. and we also assume people will use REST right? All these things won't work on 0.5 (yes those are around still too) and 0.7 based containers, however that never stopped us from moving forward and trying to improve with each spec revision.

Yes there will always be a period of time in which some containers support an older version, and some apps won't work everywhere if they use the latest spec revision, but the goal here is not to develop a solution that will work on all older specs, but to develop a new spec revision, right?

To the second point, we have protection build in that you can't call your self 'OpenSocial' if you do not support MUST part of the spec, so to assume and expect that containers 'chose not to support server sided rewriting' sounds absurd to me; I've had plenty of conversations with containers here in the EMEA region who would love to support ONLY server side templating and not do javascript (it scares them from a security POV), but I always explained best I could to them that to call your self an OpenSocial container you have to support all parts of it ... otherwise the spec looses all meaning if people get to pick only the parts their interested in, and we'd end up with opensocial apps that would never run on more then just one container since each supports a different part of the 'spec'

If we chose to halt development and block solutions from now on 'because it won't work on containers with older opensocial specifications' we might as well close this list and call it a day

[Kevin Brown]

It does not come as a surprise. Many of us have voiced concerns about this approach months ago, when the very first OSML proposals came out. Back then you told us to "wait and see" before making judgments. We waited and saw, and now we're making judgments. Please don't claim that this is a new problem, because it was discussed right here on this list. Just look at the discussions from June.
 

[Lev Epshteyn]

I'm not quite sure I understand the argument here, or the counter proposal. Here's my understanding of the matter:

• Both server-side and client-side templating are useful and needed
• It doesn't make sense to have different syntax for OSML and templating
• This by the way is why I mentioned OSML being well-formed XML
  
• As a first pass, much of OSML functionality can be implemented via client-side libraries, to be seamlessly picked up by the server as containers gain these capabilities
  
• To implement modular client-side templating (AJAX pattern), we need <script> tags (or some other kind of tags) - to define where the different templates reside.
  
• For the simple render-against-social-data use cases, it would be optimal to not need these tags, since they confuse HTML-only developers who are afraid of JS

It seems like a developer is presented with two fairly distinct paths for building their app - use AJAX or HTML with data bindings. Since these are different kinds of apps, I think it is not problematic to require them to have different Content @types - this way neither development pattern intrudes on the other.

Why is there so much opposition to this, and what's the counter proposal?

[Chris Chabot]

"By providing a JS library, developers can use templates in content that is visible both on their gadget and on their own web site."

Isn't one of the goals of OSML to allow for things like:
<os:hasApp> foo <os:else> bar </os:hasApp>

or constructions like
<os:profilePic uid="12345" linked="true" />

<os:if-is-friends-with-viewer uid="12345">
Hey your already friends, cool!
</os:if-is-friends-with-viewer>

These kind of tools would make the life of a web-1.0 type developer a lot easier, require no client side parsing or extra requests, and even have the potential of solving some consistency in UI problems too.

Now that seems pretty impossible if your argumentation is that the template should work on a stand alone website (without any social context from the container) too, unless you expect them to implement something like shindig, in which case server side rendering is possible too :)

"Can be tested without a gadget container"

I have a hard time imagining how you can test a social app, without a social context ... what exactly would you show in the friends selector? What profile pictures? What media album entries? What owner and viewer information would you build on?

You could build 'an web application' without a container, sure... but not a social app, and that's kind of the point of Open*Social* isn't it, and given the name of this list, i thought that was everyone's priority here :)

"Can be used on all containers, today."

in my opinion this should be a non-argument out of principle, as i outlined in my earlier response

"Does not require containers to parse HTML body content.

• Although there are parts of the OpenSocial spec (hangman substitution is one) that need containers to go through the entire content of the page, I believe that many non-Shindig implementations do not support this.
• Also, OSML would require a more complex parser than just a string replacer. This is more work to implement and more CPU cycles on the server.
  
• Because of these, I'm not sure if OSML would ever be universally supported across OpenSocial containers if it required server side processing."

I find these assumptions very hard to support without any data or references to back this up, so far from my talks with containers here in europe, their all very excited about server sided rendering and templating, and there are also a lot of posts indicating excitement about lower latency on initial render that this could bring.

Also the question if it will be universally supported, if we make it a 'MUST' in the spec, I would assume so; Besides from the containers on this list I've gotten the impression they seem ok with the proposal of server sided processing (anyone from myspace, yahoo, hi5, etc want to weigh in on this discussion?)

Lastly the pressure is quite high to support smart tags like <os:addAppLink> <os:hasApp> <os:profilePicture link="true"> and all such type of constructions, people are used to these kind of tags "on that other social platform" and they add a lot of value to app developers, plus they offer a method for better UI consistency with the container.

" I had though we were at the point of discussing syntax details, not major design issues"

See kevin's remark on which i'm completely +1 since I was there on those threads too and remember them quite well.

[Chris Chabot]

ps, just in case the un-asked question wasn't apparent:

What can we do to get the best of both of these development models?

1. Ajax / Javascript web 2.0 type development (as we already have, but then with templating added)
2. Web 1.0 type development using proxied content + OSML
3. Web 1.0 type development using proxied content + REST

Ideally you would want to be able to mix and match these too, where you could create a proxied content type gadget (<Content href="foo"), populate or even pre-populate data via REST, and output html that includes OSML tags *and* complex javascript logic too.

And preferably in a way that is optimized for the joe-blow-developer who wants nifty smart tags (<os:addAppButton>), and depending on anything that even looks or smells like javascript, unless he choses to use that development model :)

  -- Chris

[Evan Gilbert]

On Fri, Nov 7, 2008 at 8:55 AM, Chris Chabot <cha...@google.com> wrote:

On Fri, Nov 7, 2008 at 5:18 PM, Evan Gilbert <uid...@google.com> wrote:

On type="html" vs type="osml"...

The biggest benefit of allowing OSML to work in standalone HTML content is that it works on all containers, including those that only support 0.7 or 0.8, and those who might not want to implement or support content rewriting.

We have a JS feature in Shindig that works today on almost all containers by using <script src="">. It took me about 20 minutes to get the Super Lame Game (my simple template app) working on Ning, which is an 0.7 container.

Ok please let interject here... Your saying we should cater to containers who do not want to upgrade to the latest spec revision, or choose not to support a MUST part of the OpenSocial specification?

"Cater" is a bit of a strong word. I wouldn't make large concessions, but we're talking about just two script tags and no loss of functionality. We should be pragmatic and that realize specifications aren't useful until they are generally adopted, and the server-side processing will limit adoption for a significant period.

If the spec wouldn't work both ways, I'd have a different response. But we have a spec that can for both cases, and I'm not sure why we would sacrifice this for minor syntactic sugar.

 

I had a longer response all typed up, but really there is nothing to add to this question ...

To the first point, we always presume containers will upgrade, we presume gadget devs will write gadges that <Require feature="opensocial-0.8"/> and use all the new goodies we put into each new spec.. and we also assume people will use REST right? All these things won't work on 0.5 (yes those are around still too) and 0.7 based containers, however that never stopped us from moving forward and trying to improve with each spec revision.

Yes there will always be a period of time in which some containers support an older version, and some apps won't work everywhere if they use the latest spec revision, but the goal here is not to develop a solution that will work on all older specs, but to develop a new spec revision, right?

Hope I covered this above - agreed that we shouldn't cater to this, but supporting it at no cost seemes useful.
 

To the second point, we have protection build in that you can't call your self 'OpenSocial' if you do not support MUST part of the spec, so to assume and expect that containers 'chose not to support server sided rewriting' sounds absurd to me; I've had plenty of conversations with containers here in the EMEA region who would love to support ONLY server side templating and not do javascript (it scares them from a security POV), but I always explained best I could to them that to call your self an OpenSocial container you have to support all parts of it ... otherwise the spec looses all meaning if people get to pick only the parts their interested in, and we'd end up with opensocial apps that would never run on more then just one container since each supports a different part of the 'spec'

It's great that there are containers that only want to support server side templating, but I don't believe it follows that we should create a spec that requires server side templating.

 

[Evan Gilbert]

It's not a new problem, but the spec list did come to agreement on a high level design using <script> tags. Concerns were voiced, but the goal was to inform discussions with implementations, not to reopen basic design questions without additional data.

Client side interoperabilty was listed as one of the original requirements, and we came up with a solution that works for both server and client side. Are you now claiming that we shouldn't support client side? Or that the solution doesn't work for server side?

[Evan Gilbert]

On Fri, Nov 7, 2008 at 10:13 AM, Chris Chabot <cha...@google.com> wrote:

"By providing a JS library, developers can use templates in content that is visible both on their gadget and on their own web site."

Isn't one of the goals of OSML to allow for things like:
<os:hasApp> foo <os:else> bar </os:hasApp>

or constructions like
<os:profilePic uid="12345" linked="true" />

<os:if-is-friends-with-viewer uid="12345">
Hey your already friends, cool!
</os:if-is-friends-with-viewer>

These kind of tools would make the life of a web-1.0 type developer a lot easier, require no client side parsing or extra requests, and even have the potential of solving some consistency in UI problems too.

Now that seems pretty impossible if your argumentation is that the template should work on a stand alone website (without any social context from the container) too, unless you expect them to implement something like shindig, in which case server side rendering is possible too :)

With OAuth, the developer can populate the same JSON content that is returned from the OpenSocial REST calls. This is definitely possible from the server side and probably possible from the client side (<iframe src="www.containermodules.com/api/rpc?format=html&st=token">, creates an RPC channel that sends data back to the parent page.

 

"Can be tested without a gadget container"

I have a hard time imagining how you can test a social app, without a social context ... what exactly would you show in the friends selector? What profile pictures? What media album entries? What owner and viewer information would you build on?

Fake JSON data is your friend. You can stub out cotainer calls with fake JSON data.

I think testability is a serious limitation of Shindig currently - I'd be curious to hear how many gadget developers download a local copy of Shindig in order to be able to see how their gadget works with different user imnput. My guess is that the answer is very few.

This seems to be an opportunity to make this process easier.

As I mentioned, I'm going to post an example gadget to clarify how this would work.

 

You could build 'an web application' without a container, sure... but not a social app, and that's kind of the point of Open*Social* isn't it, and given the name of this list, i thought that was everyone's priority here :)

"Can be used on all containers, today."

in my opinion this should be a non-argument out of principle, as i outlined in my earlier response

"Does not require containers to parse HTML body content.

• Although there are parts of the OpenSocial spec (hangman substitution is one) that need containers to go through the entire content of the page, I believe that many non-Shindig implementations do not support this.
• Also, OSML would require a more complex parser than just a string replacer. This is more work to implement and more CPU cycles on the server.
  
• Because of these, I'm not sure if OSML would ever be universally supported across OpenSocial containers if it required server side processing."

I find these assumptions very hard to support without any data or references to back this up, so far from my talks with containers here in europe, their all very excited about server sided rendering and templating, and there are also a lot of posts indicating excitement about lower latency on initial render that this could bring.

Also the question if it will be universally supported, if we make it a 'MUST' in the spec, I would assume so; Besides from the containers on this list I've gotten the impression they seem ok with the proposal of server sided processing (anyone from myspace, yahoo, hi5, etc want to weigh in on this discussion?)

Lastly the pressure is quite high to support smart tags like <os:addAppLink> <os:hasApp> <os:profilePicture link="true"> and all such type of constructions, people are used to these kind of tags "on that other social platform" and they add a lot of value to app developers, plus they offer a method for better UI consistency with the container.

From an abstract level, it seemed likely that parsing every page view would be a burden that all containers might not want to support,  but agreed that we should get more feedback.

I'm also very worried about issues that we would find on implementation or rollout. I could easily see a container wanting to turn *off* this feature if needed if their CPUs were spiking because of increased traffic, or because a developer wrote an infinite template loop. Even if we like this as an abstract direction it seems a useful charateristic to not require this.

Nothing in the proposal would preclude smart tags, such as <os:addAppLink>, etc.

 

[Adam Winer]

I don't see server-side and client-side processing as black-and-white.  A largely client-side solution can require minimal server-side processing  (as is true of all features).

Why not have <Requires feature="osml">, for which one implementation could:

  - add the Javascript library

  - insert <script type="text/os-template"> into the generated HTML as necessary

... which keeps <script type="text/os-template"> out of the content gadget developers ever see, IMO a very good thing.

-- Adam

[Scott Seely]

It seems we have agreement on much of this spec. This is goodness. Items where we have contention may need more time or be separated out and postponed to the next milestone.

 

We have contention on a few points:

1.       Processing returned/proxied content as OSML. If we drop this, devs will have to return fully expanded HTML. Is this OK in a world where we can send the JSON/XML objects on the POST to the dev server?

2.       The script tags: I haven’t seen any discussion for how drop the tag and still allow one to build custom templates. Please see http://wiki.opensocial-templates.org/index.php?title=OpenSocial_Templating#Calling_Templates. In previous discussions, custom/named templates are the one item that let the script tag live. Dropping the script tag would mean dropping named templates from the OSML spec for this rev. Is this a feature we can live without?

3.       Content type=html|osml: It seems like this can be mitigated by using type=html and requiring the OSML feature. For cross container compatibility, the dev would states that the OSML feature is optional. In the optional use case, the inclusion of the <script type="text/osml"> tags enclosing the OSML type plus the OSML client side JS will work. This requires <script> tags to be included and allows for the use of named templates too.

 

If I missed another point of contention, I apologize. The thread is getting hard to follow. Please add the point.

 

Please don’t read my suggestions as ‘we should never do the feature’. My larger concern is getting this out quickly and recognizing that we will iterate and ship again. Our pattern is every 3-4 months.

 

 

From: opensocial-an...@googlegroups.com [mailto:opensocial-an...@googlegroups.com] On Behalf Of Evan Gilbert
Sent: Friday, November 07, 2008 1:32 PM
To: opensocial-an...@googlegroups.com
Subject: [opensocial-and-gadgets-spec] Re: [Proposal] Data Pipelining

 

 

On Fri, Nov 7, 2008 at 10:13 AM, Chris Chabot <cha...@google.com> wrote:

[Chris Chabot]

Thanks for a great summary Scott

On Fri, Nov 7, 2008 at 10:36 PM, Scott Seely <sSe...@myspace.com> wrote:

We have contention on a few points:

1.       Processing returned/proxied content as OSML. If we drop this, devs will have to return fully expanded HTML. Is this OK in a world where we can send the JSON/XML objects on the POST to the dev server?

This for me is the most pivotal point, we all agree I think proxied content and templating are the 2 main features of the 0.9 spec; However I don't think anyone realized it was proxied content OR templating, and that the two couldn't be combined :)
 

2.       The script tags: I haven't seen any discussion for how drop the tag and still allow one to build custom templates. Please see http://wiki.opensocial-templates.org/index.php?title=OpenSocial_Templating#Calling_Templates. In previous discussions, custom/named templates are the one item that let the script tag live. Dropping the script tag would mean dropping named templates from the OSML spec for this rev. Is this a feature we can live without?

I'm not against script tags per say, just that their always required.. if OSML parsing was pervasive and applied everywhere (regardless of script tags), and the script tags were used to define named templates, that would be tip-top-ok-by-me.

However this dives into the larger argument of "Should we make server side processing a MUST feature of the 0.9 spec".

I'm strongly in favor of this since this allows pervasive use of templating everywhere and in a simple and transparent fashion, this does mean scaling happens at the server, but gives the opportunity for latency improvements, the html is ready and visible on load, there's no browser freeze on profiles with lots of complex-template-gadgets, and life is easier for the web 1.0 type developer who probably would love to use proxied content + templating.

Evan is strongly against this on the basis that this requirement will cause delay in uptake because implementation would be harder , and even speculates this might be 'switched off' by containers if cpu usage would spike.

Now as far as the implementation goes, I'd probably do the php-shindig version, kevin brown probably the java-shindig version, and myspace is the only other major OpenSocial implementation of OpenSocial outside of Shindig that I know of, so we do have something of an council of experts here who I feel are qualified to comment on this.

As kevin pointed out, things would be easier if we chose simpler constructs, that means no <div repeat="$foo">$foo.name</div> but stick to the simple <os:ifHasApp>html..</os:ifHasApp> type constructs which are very easy to parse and expand on the server, which might aleviate the implementation worries Evan has Expressed.

Scott, maybe you can give some feedback on what your perspective on this is from a MySpace point of view, does Evan have a strong point according to you? Or would you prefer the latency win of server sided processing?

I don't know if anyone from Yahoo, Hi5 and Orkut are following these threads, but personally I think your opinions would count for a lot here, so please give feedback if you can
 

3.       Content type=html|osml: It seems like this can be mitigated by using type=html and requiring the OSML feature. For cross container compatibility, the dev would states that the OSML feature is optional. In the optional use case, the inclusion of the <script type="text/osml"> tags enclosing the OSML type plus the OSML client side JS will work. This requires <script> tags to be included and allows for the use of named templates too.

You know to me this has all the facets of a good ol' circular reference confusion:

One of the main arguments of javascript js libs, is that it'll work everywhere, however <Require feature="osml" /> will break anywhere but on 0.9 containers, so basically it's not safe to use the require feature statement and depend on 0.9 being present according to Evan.

Also proxied content won't be parsed as OSML, really the *only* remaining case in which you can safely depend on templating being there, is when your writing your gadget according to the ajax/javascript model we already have.. and include the os-templates.js libs via a script tag.

So if we follow down this path of reasoning: you can't do <Content type="osml"> or <Require feature="osml"> because of the slow adoption according to Evan which would make that non usable right now, and it's not usable in proxied content either (which btw will also only work on 0.9 supporting containers), its really only useful for Ajax / web2.0 type developers, who probably already know where to find good templating engines since jQuery, prototype, dojo all have some templating solution build in and there's lots of others available too.. So in essence, to sum this up, we spend all this time thinking about templating to solve pretty much nothing? :)

[Chris Chabot]

On Fri, Nov 7, 2008 at 10:32 PM, Evan Gilbert <uid...@google.com> wrote:

I think testability is a serious limitation of Shindig currently - I'd be curious to hear how many gadget developers download a local copy of Shindig in order to be able to see how their gadget works with different user imnput. My guess is that the answer is very few.

One of the main reasons i started partuza is to alleviate this problem, there are already some efforts under way to package up php-shindig and partuza with all dependencies (apache, mysql, php) into one friendly installer (prototype looks very promising) which should lead to a double-click-and-your-set type solution for local testing and development.

 

As I mentioned, I'm going to post an example gadget to clarify how this would work.

As long as you code it in javascript, but don't use any of the OpenSocial javascript API? Or a proxied content type gadget? Seems rather limiting to me
 

I'm also very worried about issues that we would find on implementation or rollout. I could easily see a container wanting to turn *off* this feature if needed if their CPUs were spiking because of increased traffic, or because a developer wrote an infinite template loop. Even if we like this as an abstract direction it seems a useful charateristic to not require this.

If we simplify the templating language slightly, meaning no <div repeat="$foo"> but <os:hasApp> </os:hasApp> type solutions, which i think combined with either javascript or gadget dev's prefered server sided templating solution, should cover just about all cases. That way you can prevent circular references too.

Besides, it'll still end up being lighter then caja parsing won't it? :)

[Scott Seely]

The use case for repeaters in the OSML is pretty strong—we really want it if at all possible. Table output, selection lists, etc. all benefit in the presence of some form of repeater. My experience with the repeat as attribute vs. control tag argument is that it is a time waster—the two constructs are isomorphic (I have an XSLT laying around that handles the attributeàtag transformation).

 

From the start, we’ve felt pretty strongly that OSML should be agnostic about client or server. MySpace is building a server implementation at this time, so I don’t suspect that we would argue against some of the client side capabilities being dropped if ShinDig thought this was acceptable. However, I know that the ShinDig prototype has done some excellent work along these lines purely on the client.

 

If I understand you correctly, the suggestion here is to require the OSML feature and use the <script type="text/osml" name="myTemplate"> block only for named templates. All other OSML uses would then be in the open.\

 

This leaves the path to back compat as little more than a curiosity from the proof of concept implementation.

 

From: opensocial-an...@googlegroups.com [mailto:opensocial-an...@googlegroups.com] On Behalf Of Chris Chabot
Sent: Friday, November 07, 2008 2:44 PM
To: opensocial-an...@googlegroups.com
Subject: [opensocial-and-gadgets-spec] Re: [Proposal] Data Pipelining

 

Thanks for a great summary Scott

[Chris Chabot]

On Sat, Nov 8, 2008 at 12:08 AM, Scott Seely <sSe...@myspace.com> wrote:

The use case for repeaters in the OSML is pretty strong—we really want it if at all possible. Table output, selection lists, etc. all benefit in the presence of some form of repeater. My experience with the repeat as attribute vs. control tag argument is that it is a time waster—the two constructs are isomorphic (I have an XSLT laying around that handles the attributeàtag transformation).

There are other ways of preventing recursive cycles, either through detection in code ("5000 loops means your killed and the recursive cops will be dispatched!"), or through language definition, not allowing template inclusions in repeat statements is a very simple way of preventing this from ever happening.

Either way I was trying to address Evan's concerns, but I agree it doesn't have to go, there are perfectly valid solutions to this problem.
 

 From the start, we've felt pretty strongly that OSML should be agnostic about client or server. MySpace is building a server implementation at this time, so I don't suspect that we would argue against some of the client side capabilities being dropped if ShinDig thought this was acceptable. However, I know that the ShinDig prototype has done some excellent work along these lines purely on the client.

(ps, it really is 'Shindig', no capital D required before someone would suspect a htdig or digg.com relationship :P)

Neither me nor kevin have any objections to coding a server sided solution, which covers both the php and java versions of Shindig, so that shouldn't have to be a blocker at all..

Evan created a really great prototype but I don't think we should ever let prototype code dictate what solution we feel is the right one, otherwise we never would've made it to 0.8 and 0.8.1, both of which nuked vast amounts of the social api code I had written for php-shindig, and i'm sure you've had similair experiences with your code base too :)
 

If I understand you correctly, the suggestion here is to require the OSML feature and use the <script type="text/osml" name="myTemplate"> block only for named templates. All other OSML uses would then be in the open.\

Well to me that seems like the most transparent and easy-to-use solution
 

 This leaves the path to back compat as little more than a curiosity from the proof of concept implementation.

I'd say that that is correct.

Keep in mind that for Ajax/Web2.0 style development as the current gadgets are, the js libs could easily be modified to work complimentary to the server sided OSML templating, and thus still allowing for those developers to get something extra out of it as well, but i would rather see the js libs as something complimentary then something forcing us to some decision.

[Kevin Brown]

On Fri, Nov 7, 2008 at 3:23 PM, Chris Chabot <cha...@google.com> wrote:

On Sat, Nov 8, 2008 at 12:08 AM, Scott Seely <sSe...@myspace.com> wrote:

The use case for repeaters in the OSML is pretty strong—we really want it if at all possible. Table output, selection lists, etc. all benefit in the presence of some form of repeater. My experience with the repeat as attribute vs. control tag argument is that it is a time waster—the two constructs are isomorphic (I have an XSLT laying around that handles the attributeàtag transformation).

There are other ways of preventing recursive cycles, either through detection in code ("5000 loops means your killed and the recursive cops will be dispatched!"), or through language definition, not allowing template inclusions in repeat statements is a very simple way of preventing this from ever happening.

Either way I was trying to address Evan's concerns, but I agree it doesn't have to go, there are perfectly valid solutions to this problem.
 

 From the start, we've felt pretty strongly that OSML should be agnostic about client or server. MySpace is building a server implementation at this time, so I don't suspect that we would argue against some of the client side capabilities being dropped if ShinDig thought this was acceptable. However, I know that the ShinDig prototype has done some excellent work along these lines purely on the client.

(ps, it really is 'Shindig', no capital D required before someone would suspect a htdig or digg.com relationship :P)

Neither me nor kevin have any objections to coding a server sided solution, which covers both the php and java versions of Shindig, so that shouldn't have to be a blocker at all..

I actually do have an objection, which is that I haven't seen a good way to do this without an interpreter, given the presence of EL. I generally argue that if I need an interpreter, I'd prefer a proper imperative language anyway. That's an orthogonal problem from my perspective though.
 

[John Hayes]

Can we solve the "function declaration" use case more simply with a standard tag like os:template akin to os:repeat or os:if? This could even work with attribute notation:

 

<div template="my:testtag">Some template content ${where}</div>

 

<my:testtag where="over here"/>

 

The only reason to have this tag is local reuse, for more complex reuse there're the template libraries, you can have that functionality whether or not there's a script element.

If a container wants to implement the OSML feature at the client side, they're free to do so by transmitting the template encapsulated in a script element and importing a Javascript file to take it apart. This variation is not disallowed.

 

A developer doesn't have to install shindig to test a gadget with a server, they only need a web browser and an existing container that will honor refreshes. Testing cannot be simplified beyond save file, then refresh browser - the mechanics in between are not interesting as long as they're fast.

 

John

[Lev Epshteyn]

The choice of <script> tags was dictated by browser peculiarities. If <div> tags are used, the browsers interpret the contents as HTML, and transform it to make it valid on the spot. This has unfortunate side effects. For example, if I have <div style="color: ${color}">, Firefox will strip away the style attribute because it's invalid, before the template engine ever gets the chance to process it. (Don't even get me started on browsers' treatment of custom tags like <os:template> dropped into raw HTML space.)

As such, we needed a tag the contents of which the browser wouldn't try to build DOM out of. There are three such tags I am aware of: <xmp> which is deprecated, <textarea> which didn't seem right to use here, and <script> with a non-JS @type.

[Lev Epshteyn]

On Fri, Nov 7, 2008 at 4:36 PM, Scott Seely <sSe...@myspace.com> wrote:

3.       Content type=html|osml: It seems like this can be mitigated by using type=html and requiring the OSML feature. For cross container compatibility, the dev would states that the OSML feature is optional. In the optional use case, the inclusion of the <script type="text/osml"> tags enclosing the OSML type plus the OSML client side JS will work. This requires <script> tags to be included and allows for the use of named templates too.

So Scott, your proposal is that

<Content type="html">

is treated as OSML when the OSML feature is required, but treated as plain HTML when it's not (while still being able to define OSML-using templates in <script> tags)?

This seems odd. You are basically still essentially having two types of content, but the switch happens as a side effect to you requiring the feature. Additionally, you lose granularity of being able to have one View be OSML and one be HTML.

It seems like @type="osml|html" is exactly the granularity we need here. (Unless you are trying to say that there is never a useful case to NOT interpret OSML markup within gadget content).

For Web-1.0 type developers, @type="proxied" can be used, and the mime type of the returned document can be used to determnie whether or not OSML markup is to be processed in the returned content. This would give us consistent behavior for both inline and proxied Content.

<script> tags would be required for delineating OSML areas in @type="html", and be available for creating named templates in @type="osml" Content.

This seems to enable all the use cases on the table in a consistent and straightforward manner without them infecting each other.

[John Hayes]

Putting HTML though an OSML processor should work just fine, and it's important that html, osml and proxied aren't conflated. What is really present is a choice between two transfer modes: inline and external; and a flag for whether OSML processing should proceed.

 

Using the script tag isn't perfect, they can't be nested without escaping.

 

Requiring a developer to set a content-type seems like a "gotcha" that will not save significant CPU time on the server. If OSML isn't used by the majority of gadgets shortly after release, it should be recalled and changed until it is.

 

John

[Lev Epshteyn]

On Fri, Nov 7, 2008 at 11:53 PM, John Hayes <john.mar...@gmail.com> wrote:

Putting HTML though an OSML processor should work just fine,

I think there are cases when you may want to output HTML that won't get interpreted by OSML. (For example if you for whatever reason have ${} markup in your text which you don't want to have variable substitution in) Since all the gadgets that exist today DO NOT use OSML, having it be turned on suddenly with no way to opt out seems like a pretty drastic step.
 

and it's important that html, osml and proxied aren't conflated. What is really present is a choice between two transfer modes: inline and external; and a flag for whether OSML processing should proceed.

 

If we allow that both html and osml are valid, this should hold true for both inline and external content - so we really have two dimensions of rendering content: OSML/HTML and Inline/Proxied. These can be combined however you chose for four different combinations, where there are some use cases for each individual one: Inline HTML, Inline OSML, Proxied HTML and Proxied OSML.

Come to think of it, combinging the @type attribute with the presence/absence of @src attribute seems like the most natural way to express this:

<Content type="html">: Inline HTML
<Content type="html" src="http://example.com/app">: Proxied HTML
<Content type="osml">: Inline OSML
<Content type="osml" src="http://example.com/app">: Proxied OSML

[Kevin Brown]

When proxying content, there's really no good reason to need to specify the type at all. That's what the Content-Type header is for.
 

[Scott Seely]

Just one comment—I’m not too sure what is going on with the comment about recursive cycles and preventing them and recursive cycles. I mean I get what you are saying, I just don’t see what sparked it. I was simply stating that I like the idea of having a mechanism to define that, with a collection of friends, I want each friend’s data in a row in a table. Obviously, an implementation will need to detect recursive cycles or typical errors during development will consume CPU and blow stack on the container dev servers.

 

All that said, we should probably standardize how deep any recursion will be allowed so that devs will know that if they use < n levels of recursion, they are OK.

 

The isomorphic comment means that I can do a lossless transformation from attributeàtag based, making the comment about which one is ‘bad’ moot. I proved this to myself by writing a XSLT that seems to do the job.

 

From: opensocial-an...@googlegroups.com [mailto:opensocial-an...@googlegroups.com] On Behalf Of Chris Chabot
Sent: Friday, November 07, 2008 3:23 PM
To: opensocial-an...@googlegroups.com
Subject: [opensocial-and-gadgets-spec] Re: [Proposal] Data Pipelining

 

On Sat, Nov 8, 2008 at 12:08 AM, Scott Seely <sSe...@myspace.com> wrote:

[Lev Epshteyn]

When proxying content, there's really no good reason to need to specify the type at all. That's what the Content-Type header is for.

Kevin, if you think this is a sufficiently user-friendly mechanism for specifying how proxied content is to be handled, we can simplify the <Content> tag markup to three scenarios:

<Content type="html">
<Content type="osml">
<Content type="proxied" src="http://example.com/app"> - in this case the mime type is interpreted to determine whether the returned output is to be treated as OSML or not.

This seems like it leverages existing standards more (mime type), though is a little less easy to use (I'd imagine some of the developers may not have a good understanding of mime types, though they can be easily educated).

Either way, I think it's valuable to retain Content block level control of output handling, something that keying off of a required OSML feature doesn't give us.

[Kevin Brown]

On Sat, Nov 8, 2008 at 7:36 AM, Lev Epshteyn <le...@google.com> wrote:

When proxying content, there's really no good reason to need to specify the type at all. That's what the Content-Type header is for.

Kevin, if you think this is a sufficiently user-friendly mechanism for specifying how proxied content is to be handled, we can simplify the <Content> tag markup to three scenarios:

<Content type="html">
<Content type="osml">
<Content type="proxied" src="http://example.com/app"> - in this case the mime type is interpreted to determine whether the returned output is to be treated as OSML or not.

I'd take the javascript approach here, and not require the type attribute at all. If the content is sourced externally (because the uri is specified), we get it from the mime type. If it's inline, it should be declared, with an assumption of the default type being "html".

This seems like it leverages existing standards more (mime type), though is a little less easy to use (I'd imagine some of the developers may not have a good understanding of mime types, though they can be easily educated).

Fair enough. I don't care much one way or the other, but if people were actually concerned about needing to specify a type attribute that's one way around it.
 

[Evan Gilbert]

Thanks again to everyone for all of the feedback on the thread - it's definitely been helpful.

I am going to create a Wiki page and try to consolidate feedback and open issues into a document that people can parse through without reading through the 50+ posts on this thread.

Also, I promised before to send out links to a gadget that uses test JS data to work as standalone pages for testing. It's fairly basic, but you can see how this can be used to simulate all of the initial load data:
- Profile: http://lamegame.appspot.com/gadget/profile.html
- Canvas: http://lamegame.appspot.com/gadget/canvas.html
- Gadget: http://lamegame.appspot.com/gadget/superlame.xml - can be installed on Orkut

Evan

[Scott Seely]

Are you going to use the Wiki page that was already created: http://wiki.opensocial.org/index.php?title=Data_Pipelining? Just thinking that Lane’s good work should be used J

 

From: opensocial-an...@googlegroups.com [mailto:opensocial-an...@googlegroups.com] On Behalf Of Evan Gilbert
Sent: Monday, November 10, 2008 10:11 AM
To: opensocial-an...@googlegroups.com
Subject: [opensocial-and-gadgets-spec] Re: [Proposal] Data Pipelining

 

Thanks again to everyone for all of the feedback on the thread - it's definitely been helpful.

[Evan Gilbert]

Yep, I'll add to that wiki page. Thanks,
Evan

[Scott Seely]

Update thread with F2F notes:
Largely reached agreement. Evan Gilbert (uid...@google.com) will be
following up on the Data Pipelining thread with details.

On Nov 7, 6:16 am, David Byttow <davidbyt...@google.com> wrote:
> I agree with what you are saying. But, from reading this thread (and perhaps
> I'm interpreting things wrong), it seemed like some of the arguments were
> entering the client-side space and wanting to cut out certain features such
> as content html <script> template tags, which I am very much supportive of.
>
> It was also mentioned on this thread that Evan was the only voice for
> supporting OSML outside of gadgets, so I wanted to back up his opinions
> publicly.
>
> As a general virtue, you're right that we should not create "another type of
> OSML", but what I was saying is that it is becoming increasingly difficult
> to maintain feature parity with OSML that is to be processed on either the
> client or the server, so in my opinion, it might be okay to allow for
> certain constructs to be supported and processed by the server on returned
> content (as per development model that you're describing) outside of
> <script> tags.
>
> David
>

> On Fri, Nov 7, 2008 at 5:39 AM, Chris Chabot <chab...@google.com> wrote:
> > Hey David,
>
> > It might be the fact that I have not yet had enough coffee today (and hence
> > violating the internation don't-post-before-2-coffee's rule), but i am not a
> > 100% yet on how to place all of your remarks in the context of the
> > proposals.
>

> > What's onhttp://ostemplates-devapp.appspot.com/was a proposal that

> > hasn't made it to spec yet, and as such it's not yet a part of open social,
> > it's a very well made, usable proposal mind you, but not set in stone.
> > Likewise data-pilelining and proxied content are not set in stone yet
> > either.
>
> > Also what's currently in ostemplates is intended for gadget/client side use
> > only, and that means javascript.. however don't forget that creating
> > something for 'the rest of the developers' who don't dig javascript is the
> > main thing that got this discussion started, there's countless of devs out
> > there who want to create an OpenSocial app (or port it from an existing fb
> > app), their used to either a standard CMS system and/or programming in FBML,
> > and they get stuck on the JavaScript based API that OpenSocial uses..
>
> > Making  the JavaScript API more powerful is *not* a solution for this
> > problem space, and loosing track of that would mean loosing sight of the
> > very goal that we set out to reach, so while I'm not against making the end
> > proposal work as well as possible from that use-case point of view as well,
> > it's not my personal main motivation in these discussions.
>
> > With that in mind, isn't this the perfect opportunity to see if we can
> > combine these concepts into one , consistent, well working , thought out
> > solution? As such I find a blanket statement saying that we should punt
> > server sided rendering to the future to 'another type of OSML', a less then
> > optimal situation, I would never want to advocate having 2 or 3 types of
> > solutions for the same problem space, that only confuses developers and
> > increases the pressure on containers who have to implement it.
>

> > On Fri, Nov 7, 2008 at 5:18 AM, David Byttow <davidbyt...@google.com>wrote:
>
> >> I am also a strong advocate of allowing client-side script tag templates
> >> in type=html. I have relied on this heavily in my own apps. This was
> >> important for developing applications that were both standalone and
> >> gadget-based (http://ostemplates-devapp.appspot.com/), yet managed to
> >> utilize templates (both server-side django and client-side osml).
>
> >> It has been a breath of fresh air to me knowing that my templates will
> >> work across all containers today. And using custom, modular templates to
> >> show/hide/re-render based on AJAX operations is a pattern that I leverage
> >> often.
>
> >> I think moving yet another feature onto the server and requiring a
> >> container to process them is a mistake. However, I do like the idea of
> >> allowing some server implementations to pre-process certain templates or
> >> markup for performance.
>
> >> In the case of one-off static tags such as <os:hasApp>, can we distinguish
> >> between the concept of templating/data-binding as we have today and a future
> >> brand of server-side OSML (like fbml) which is always processed on the
> >> server.
>

> >> On Thu, Nov 6, 2008 at 6:36 PM, Chris Chabot <chab...@google.com> wrote:

[Evan Gilbert]

Updated proposal at:
http://wiki.opensocial.org/index.php?title=Data_Pipelining

Change Log: 11/17/08 - Updated with feedback from OpenSocial 0.9 summit and other minor changes:

• Moved to Wiki
• changed @userid and @groupid to @userId and @groupId
• <os:PersonAppDataRequest> removed, obsolete with other 0.9 changes
• @type="osml" removed (and notes why)
• Notes on why <script> tags remain
• <os:DataRequest> defaults to returning JSON, can also return @format="text"
• <os:DataRequest> can use @method to call optional additional RPC methods
• <os:PersonRequest> removed, added <os:ViewerRequest> and <os:OwnerRequest> (plus notes as to why)

Think we're ready to start voting.

Evan

[Louis Ryan]

Some notes:

- Spec does not say for non-proxied non-OSML gadgets how the contents of the preload are made accessible in JS. Something along the lines of opensocial.getPreloaded() which returns DataResponse seems appropriate. A more concise equivalent is also needed in oslite.

- Probably best to explicitly state that the attribute names match 1:1 with REST & JSON-RPC parameter names and makeRequest parameter names where a mapping exists

- Spec does not state what format the content POSTed for a proxied render is in other than as "POSTed JSON data" I believe the JSON-RPC batch response format is the appropriate one and we should state that.

- I think filter attribute should be extended to match the REST spec definition of filterBy, filterOp & filterValue. Most of the predefined filters I  believe map to filterOp=<predefined name> e.g. filterOp=@hasApp. Filter operations apply to activites and other types too.

- For <os:DataRequest> I don't think the permutations of method and href work as described. If href is empty and method is "people.get" what are the parameters to the RPC method? I think it would be fine to use the URL encoded form of JSON-RPC parameters in the href and  have the method be "<service>.<operation>" as described in the JSON-RPC spec. Then detection of the special cased RPC call is simply if the method attribute is anything other than the standard HTTP methods. E.g.

<os:DataRequest href="userId=@me&groupId=@self" method="people.get" key="akey" />

I have to admit that this also seems a bit clunky. It may just be cleaner to support <os:RestRequest ...> and <os:RpcRequest> for arbitrary but container resolveable calls and leave DataRequest to just HTTP

-  I think there's a larger list of relevant makeRequest params that should be documented for DataRequest HTTP use. e.g. OAUTH_SERVICE_NAME. We should state whether the intent is to have full 1:1 functionality with makeRequest and if not list the exceptions

- Needs to be a description of how containers are allowed to vary behavior and still remain spec compliant. E.g. a container suppressing  viewer information in proxied renders of profile views because it wants to enforce a high degree of cacheability for the proxied content.

[Scott Seely]

Evan—great work.

 

I would like to see us remove the text/os-data script tag if possible. I’m not aware of any use cases that make it necessary. It seems to be an artifact of us trying to separate out templates and pipelining. This means that the embedding in @type="html" content looks like this:

 

<Content type="html">

  <script type="text/os-template">

    <os:PeopleRequest key="vf" userid="@viewer" groupid="@friends"/>

    <div repeat="${vf}">${Name} is a friend.</div>

  </script>

</Content>

 

 

Other Comments inline.

 

From: opensocial-an...@googlegroups.com [mailto:opensocial-an...@googlegroups.com] On Behalf Of Louis Ryan

Sent: Monday, November 17, 2008 11:24 PM
To: opensocial-an...@googlegroups.com

Subject: [opensocial-and-gadgets-spec] Re: [Updated Proposal] Data Pipelining

 

Some notes:

 

- Spec does not say for non-proxied non-OSML gadgets how the contents of the preload are made accessible in JS. Something along the lines of opensocial.getPreloaded() which returns DataResponse seems appropriate. A more concise equivalent is also needed in oslite.

[[sseely]] http://docs.google.com/View?docid=dc5qrdst_19fbb8fpd3 has a section, “Updates to OpenSocial JavaScript API”. If nothing else, the getDataSet/putDataSet functions should move along with the proposal OR the proposal should reference the simplified JS API doc if that functionality is migrating elsewhere.

 

- Probably best to explicitly state that the attribute names match 1:1 with REST & JSON-RPC parameter names and makeRequest parameter names where a mapping exists

 

- Spec does not state what format the content POSTed for a proxied render is in other than as "POSTed JSON data" I believe the JSON-RPC batch response format is the appropriate one and we should state that.

[[sseely]] JSON-RPC batch is currently an optional item. Perhaps we want to pull that description to a standard description of ‘batched data’ and then update the JSON-RPC doc and the Data Pipelining docs reference that standard description.

 

- I think filter attribute should be extended to match the REST spec definition of filterBy, filterOp & filterValue. Most of the predefined filters I  believe map to filterOp=<predefined name> e.g. filterOp=@hasApp. Filter operations apply to activites and other types too.

[[sseely]] +1

[Lev Epshteyn]

Evan, this seems to omit a lot of client-side interoperability details that were present in the proposal here: http://wiki.opensocial-templates.org/index.php?title=OpenSocial_Data_Pipelining

Some missing points:

• How does client code access the data in non-proxied scenario (there was a proposal for this in the other spec - via opensocial.data.DataContext[key] )
• At what point is the data available to non-proxied content? Is it guaranteed to be there when the gadget renders (unlikely), when the onLoad callback is made (possible) or do you need to register a listener to be called back (optimal in my view)
• Can proxied responses include more data markup (for example, a PeopleRequest for specific IDs), and how will the pipelined data be available to merge against this content? (this is a new one)
  

[Lev Epshteyn]

Scott, this markup seems ambiguous to me, since I may have more than one template defined. Will the  ${vf} variable be available there automatically, or will I need to define a data context for each template? Also, will this data be fetched if I never render the template?

I like current proposal (with os-data tags) because it implicitly makes all the data shared within the view.

[David Byttow]

On Tue, Nov 18, 2008 at 8:44 AM, Scott Seely <sSe...@myspace.com> wrote:

Evan—great work.

 

I would like to see us remove the text/os-data script tag if possible. I'm not aware of any use cases that make it necessary. It seems to be an artifact of us trying to separate out templates and pipelining. This means that the embedding in @type="html" content looks like this:

I think it's important to separate the two. For one, it seems reasonable that data declarations should be independent of templates, for example, what if the template is named or a custom tag? Otherwise, it seems a bit ambiguous.

Also, not to complicate matters, but in client-side templating I think it's clearer for linking templates to the availability of data (not part of the spec at the moment).

<Content type="html">
  <script type='text/os-data">

    <os:PeopleRequest key="vf" userid="@viewer" groupid="@friends"/>

  </script>
  <script type="text/os-template" requiresData="vf">
    <div repeat="vf">${displayName} is a friend.</div>
  </script>
</Content>

[Lev Epshteyn]

One other note is that I think the spec should specify that unknown data pipelining tags must be ignored by the container.

This should allow backwards compatibility, as well as potentially give containers the option to surface additional data request tags.

[Lev Epshteyn]

Also, I just noticed that the first listing omits (under Detailed Proposal) <script> tags, while the listing under "Embedding in @type="html" content" includes them and specifically says they are required.

Is the omission of the tags in the first instance for brevity? I think it is a bit confusing.

[Adam Winer]

Comments:

- The XML namespace for os: is not defined in here, as best I can see.  "http://opensocial.org/templates" is what I've seen elsewhere, but is odd for a syntax not specific to templates.  I can live with it, though I'd rather see a namespace that resolves to a resource of some sort.

- I agree with Louis that the attributes names should be 1:1 to JSON-RPC parameter names.  In particular:

    - @startPage does not exist;  paging is easy enough without it

    - @filterBy, @filterOp, and @filterValue are all missing

    - os:ActivitiesRequest has a typo (userid -should be userId)

    - @activityIds should be @activityId to match (

- if @userId defaults to @viewer on os:ActivitiesRequest, it should default to @viewer everywhere;  or, it shouldn't default anywhere

- the "Embedding in @type="html" content" text says CDATA is needed, but doesn't use it in the example

[Scott Seely]

The XML Namespace should just match the XMLNS we use in the REST spec: http://ns.opensocial.org/2008/opensocial. Is there any reason why this shouldn’t be true?

 

From: opensocial-an...@googlegroups.com [mailto:opensocial-an...@googlegroups.com] On Behalf Of Adam Winer
Sent: Tuesday, November 18, 2008 3:02 PM
To: opensocial-an...@googlegroups.com
Subject: [opensocial-and-gadgets-spec] Re: [Updated Proposal] Data Pipelining

 

Comments:

[Adam Winer]

On Tue, Nov 18, 2008 at 3:11 PM, Scott Seely <sSe...@myspace.com> wrote:

The XML Namespace should just match the XMLNS we use in the REST spec: http://ns.opensocial.org/2008/opensocial. Is there any reason 

why this shouldn't be true?

It shouldn't be the same namespace, as it's a different XML language.  If you'd use a different XML schema document, you want a different namespace.  Reusing one namespace for multiple languages makes life difficult for tools, for one.

-- Adam

[Evan Gilbert]

Thanks Louis, good feedback.

On Mon, Nov 17, 2008 at 11:23 PM, Louis Ryan <lr...@google.com> wrote:

Some notes:

- Spec does not say for non-proxied non-OSML gadgets how the contents of the preload are made accessible in JS. Something along the lines of opensocial.getPreloaded() which returns DataResponse seems appropriate. A more concise equivalent is also needed in oslite.

I left out the JS APIs in transcription from the original doc. It's been added back in.
 

- Probably best to explicitly state that the attribute names match 1:1 with REST & JSON-RPC parameter names and makeRequest parameter names where a mapping exists

See http://wiki.opensocial.org/index.php?title=Data_Pipelining#Common_request_processing. Feel free to edit if this should be more prominent.
 

- Spec does not state what format the content POSTed for a proxied render is in other than as "POSTed JSON data" I believe the JSON-RPC batch response format is the appropriate one and we should state that.

Agreed, added.
 

- I think filter attribute should be extended to match the REST spec definition of filterBy, filterOp & filterValue. Most of the predefined filters I  believe map to filterOp=<predefined name> e.g. filterOp=@hasApp. Filter operations apply to activites and other types too.

Agreed. I updated the text for people.get to state that these are strings that match the REST spec - however I couldn't actually find where we had defined them in the REST spec.
 

- For <os:DataRequest> I don't think the permutations of method and href work as described. If href is empty and method is "people.get" what are the parameters to the RPC method? I think it would be fine to use the URL encoded form of JSON-RPC parameters in the href and  have the method be "<service>.<operation>" as described in the JSON-RPC spec. Then detection of the special cased RPC call is simply if the method attribute is anything other than the standard HTTP methods. E.g.

<os:DataRequest href="userId=@me&groupId=@self" method="people.get" key="akey" />

I have to admit that this also seems a bit clunky. It may just be cleaner to support <os:RestRequest ...> and <os:RpcRequest> for arbitrary but container resolveable calls and leave DataRequest to just HTTP

One alternative is @method + @params. I'm split between this and <os:RpcRequest> - any preference?
 

-  I think there's a larger list of relevant makeRequest params that should be documented for DataRequest HTTP use. e.g. OAUTH_SERVICE_NAME. We should state whether the intent is to have full 1:1 functionality with makeRequest and if not list the exceptions

The general goal should be 1:1 functionality but I didn't want to blindly add the fields as I wasn't clear if all of them could be statically defined and if all were in current use (vs. legacy support). Also, a number of the parameters deal with feed retrieval, and I wasn't sure if we wanted / needed to support this with preload.

I'll defer to the experts on this one... Louis - if you or someone else has a clear idea what parameters should be exposed, please update the proposal.
 

- Needs to be a description of how containers are allowed to vary behavior and still remain spec compliant. E.g. a container suppressing  viewer information in proxied renders of profile views because it wants to enforce a high degree of cacheability for the proxied content.

Good point. I think we have a few solved issues and one big unsolved issue:
- Containers can already not pass in viewer information (on a given view, or due to permissions). We can just match the results from calling REST APIs when the app is not allowed access to the data
- The proxied content proposal should cover how the data can be cached, based on sign_owner and sign_viewer parameters (or equivalents). The container will probably not send the opensocial_viewer parameter to the 3rd party server if not allowed for a given view. The URL for caching would only include the parameters actually sent to the 3rd party server.
- The tricky issue is that we also need to prevent sending viewer / owner information in the data pipeline if the URL sent to the 3rd party server doesn't contain opensocial_viewer / opensocial_owner.

For this last point, I've made a small update to the proposal:

Elements in a data pipeline referencing the viewer (via @viewer, @me) will return a 403 (Forbidden) error response if the container is unable to also send the opensocial_viewer URL parameter to the 3rd party server. The same holds true for @owner and opensocial_owner URL parameter.

If anyone has better HTTP codes than 403, please update the proposal. On a quick scan I couldn't find a great match.

[Evan Gilbert]

Notes below...

On Tue, Nov 18, 2008 at 8:48 AM, Lev Epshteyn <le...@google.com> wrote:

Evan, this seems to omit a lot of client-side interoperability details that were present in the proposal here: http://wiki.opensocial-templates.org/index.php?title=OpenSocial_Data_Pipelining

Yep, sorry about that. Added back in.
 

Some missing points:

• How does client code access the data in non-proxied scenario (there was a proposal for this in the other spec - via opensocial.data.DataContext[key] )

Same as before.

• At what point is the data available to non-proxied content? Is it guaranteed to be there when the gadget renders (unlikely), when the onLoad callback is made (possible) or do you need to register a listener to be called back (optimal in my view)

I agree that onload is too early, especially given the current implementation where gadgets.util.runOnLoadHandlers() is inlined into the body of the page - we would have to be able to optionally run this async.

The listener API is seems possibly a little heavyweight for this - ideally developers would want access to the raw JSON. I added  a "*" listener for registerListener() to better support the initial load use case, but possibly we should have a data onload handler as well to get the JSON directly.
 

• Can proxied responses include more data markup (for example, a PeopleRequest for specific IDs), and how will the pipelined data be available to merge against this content? (this is a new one)

I was purposely punting on this one to see what comes out of our implementations. My instinct is that we'll want this but it's difficult to commit to the complexity before seeing it in action.

[Evan Gilbert]

Good feedback from all. Batching up the remaning responses...

<script type="text/os-data">
I agree that complex use cases will need to have one, separate data block at the start of the page. However, it would be nice for simple use cases (like Scott's example) to not have to include 2 script blocks, and in general the distinction between type="text/os-data" and "text/os-template" may not be as clear to developers as it is to us.

I'd be happy moving forward with the proposal as it stands, but I also want to outline an alternative for discussion:
- Only have <script type="text/os-template">
- Only top-level data pipelining tags in the first <script type="text/os-template"> block are processed. All other data pipelining tags are ignored
This seems to accomplish both scenarios cleanly and is easy to code. I'm just not clear whether this will be simpler or more complex for developers.

JSON-RPC batch in a separate document
Scott - I can take the action on this... how soon do you think this is needed? It seems like it's not blocking now, but we should have it in place well before we finalize 0.9

unknown data pipelining tags
Agreed that containers should ignore these - added to proposal

<script> tags omitted in Detailed Proposal
oversight - added

Places where we're not 1:1 with JSON-RPC

- @startPage does not exist;  paging is easy enough without it. Did this to match the JSON-RPC doc

- @filterBy, @filterOp, and @filterValue are all missing - added

- os:ActivitiesRequest has a typo (userid -should be userId) fixed

- @activityIds should be @activityId. Did this to match the JSON-RPC doc
For the two places where the proposal matched the JSON-RPC doc - if is incorrect, let's update both. I'm not sure of the source of truth.

CDATA missing from example
Added

Default @userId value
Comment was "if @userId defaults to @viewer on os:ActivitiesRequest, it should default to @viewer everywhere;  or, it shouldn't default anywhere".

Agreed in this case that we shouldn't have a default, and updated the proposal. It's not clear whether @owner or @viewer would be the right default.

I'm not sure I agree with the blanket statement that all defaults should be the same - I'd like to provide intellegent defaults when they make sense.

Namespace
We're going to be sharing the same namespace for OSML, OpenSocial templating, and Data Pipelining. I have no idea what to call this. First person to update the proposal with a reasonable option gets my support.

[Adam Winer]

On Tue, Nov 18, 2008 at 10:14 PM, Evan Gilbert <uid...@google.com> wrote:

Good feedback from all. Batching up the remaning responses...

<script type="text/os-data">

I agree that complex use cases will need to have one, separate data block at the start of the page. However, it would be nice for simple use cases (like Scott's example) to not have to include 2 script blocks, and in general the distinction between type="text/os-data" and "text/os-template" may not be as clear to developers as it is to us.

I'd be happy moving forward with the proposal as it stands, but I also want to outline an alternative for discussion:
- Only have <script type="text/os-template">
- Only top-level data pipelining tags in the first <script type="text/os-template"> block are processed. All other data pipelining tags are ignored
This seems to accomplish both scenarios cleanly and is easy to code. I'm just not clear whether this will be simpler or more complex for developers.

JSON-RPC batch in a separate document
Scott - I can take the action on this... how soon do you think this is needed? It seems like it's not blocking now, but we should have it in place well before we finalize 0.9

unknown data pipelining tags
Agreed that containers should ignore these - added to proposal

I disagree - an error response with notImplemented would be more appropriate and useful.  We need machinery to report errors in any event.

 

<script> tags omitted in Detailed Proposal
oversight - added

Places where we're not 1:1 with JSON-RPC

- @startPage does not exist;  paging is easy enough without it. Did this to match the JSON-RPC doc

- @filterBy, @filterOp, and @filterValue are all missing - added

- os:ActivitiesRequest has a typo (userid -should be userId) fixed

- @activityIds should be @activityId. Did this to match the JSON-RPC doc
For the two places where the proposal matched the JSON-RPC doc - if is incorrect, let's update both. I'm not sure of the source of truth.

In both cases, I was going off of the existing Shindig implementation, not the JSON RPC doc.  I believe Shindig has a bug with activityId - activityIds is better, and is what the spec says.  But I'm confused where startPage came from in the JSON RPC doc, as AFAIK it's not mentioned in the Restful Protocol doc, exposed in the Javascript API, or implemented in Shindig.  I didn't follow the discussions closely when it was proposed - Louis?

 

CDATA missing from example
Added

Default @userId value
Comment was "if @userId defaults to @viewer on os:ActivitiesRequest, it should default to @viewer everywhere;  or, it shouldn't default anywhere".

Agreed in this case that we shouldn't have a default, and updated the proposal. It's not clear whether @owner or @viewer would be the right default.

I'm not sure I agree with the blanket statement that all defaults should be the same - I'd like to provide intellegent defaults when they make sense.

Across all APIs, yes, that'd be an excessive blanket statement.  Limited to @userId in this spec, I think it's reasonable.

Namespace
We're going to be sharing the same namespace for OSML, OpenSocial templating, and Data Pipelining. I have no idea what to call this. First person to update the proposal with a reasonable option gets my support.

http://ns.opensocial.org/2008/markup

-- Adam

 

[Scott Seely]

For JSON-RPC batch being moved to a separate doc—I’ll need that by Dec 1 in order to incorporate it by the 12/4 deadline to create the OS 0.9 draft.

 

From: opensocial-an...@googlegroups.com [mailto:opensocial-an...@googlegroups.com] On Behalf Of Adam Winer
Sent: Tuesday, November 18, 2008 11:24 PM
To: opensocial-an...@googlegroups.com
Subject: [opensocial-and-gadgets-spec] Re: [Updated Proposal] Data Pipelining

 

On Tue, Nov 18, 2008 at 10:14 PM, Evan Gilbert <uid...@google.com> wrote:

[Scott Seely]

Some days, my head isn’t screwed on properly.

 

It seems that we are using the http://ns.opensocial.org/2008/ as a base URL for the REST spec. I suggest http://ns.opensocial.org/2008/template as the URI for os: in the templates spec.

[Evan Gilbert]

On Tue, Nov 18, 2008 at 11:23 PM, Adam Winer <awi...@gmail.com> wrote:

On Tue, Nov 18, 2008 at 10:14 PM, Evan Gilbert <uid...@google.com> wrote:

Good feedback from all. Batching up the remaning responses...

<script type="text/os-data">

I agree that complex use cases will need to have one, separate data block at the start of the page. However, it would be nice for simple use cases (like Scott's example) to not have to include 2 script blocks, and in general the distinction between type="text/os-data" and "text/os-template" may not be as clear to developers as it is to us.

I'd be happy moving forward with the proposal as it stands, but I also want to outline an alternative for discussion:
- Only have <script type="text/os-template">
- Only top-level data pipelining tags in the first <script type="text/os-template"> block are processed. All other data pipelining tags are ignored
This seems to accomplish both scenarios cleanly and is easy to code. I'm just not clear whether this will be simpler or more complex for developers.

JSON-RPC batch in a separate document
Scott - I can take the action on this... how soon do you think this is needed? It seems like it's not blocking now, but we should have it in place well before we finalize 0.9

unknown data pipelining tags
Agreed that containers should ignore these - added to proposal

I disagree - an error response with notImplemented would be more appropriate and useful.  We need machinery to report errors in any event.

Text allowed for 404 or null:
"Containers should ignore unrecognized tags or return error code 404 (Not Found). If a tag is ignored, the JSON structure can either include a null value for the request key or just not include the request key."

How do others feel about this? I like the convenience of null, but possibly the rigor of requiring 404 is better.
 

 

<script> tags omitted in Detailed Proposal
oversight - added

Places where we're not 1:1 with JSON-RPC

- @startPage does not exist;  paging is easy enough without it. Did this to match the JSON-RPC doc

- @filterBy, @filterOp, and @filterValue are all missing - added

- os:ActivitiesRequest has a typo (userid -should be userId) fixed

- @activityIds should be @activityId. Did this to match the JSON-RPC doc
For the two places where the proposal matched the JSON-RPC doc - if is incorrect, let's update both. I'm not sure of the source of truth.

In both cases, I was going off of the existing Shindig implementation, not the JSON RPC doc.  I believe Shindig has a bug with activityId - activityIds is better, and is what the spec says.  But I'm confused where startPage came from in the JSON RPC doc, as AFAIK it's not mentioned in the Restful Protocol doc, exposed in the Javascript API, or implemented in Shindig.  I didn't follow the discussions closely when it was proposed - Louis?

I don't think it affects this spec, as we'll just update to match the official JSON RPC details when they arise. But we should clarify the JSON-RPC

 

 

CDATA missing from example
Added

Default @userId value
Comment was "if @userId defaults to @viewer on os:ActivitiesRequest, it should default to @viewer everywhere;  or, it shouldn't default anywhere".

Agreed in this case that we shouldn't have a default, and updated the proposal. It's not clear whether @owner or @viewer would be the right default.

I'm not sure I agree with the blanket statement that all defaults should be the same - I'd like to provide intellegent defaults when they make sense.

Across all APIs, yes, that'd be an excessive blanket statement.  Limited to @userId in this spec, I think it's reasonable.

Namespace
We're going to be sharing the same namespace for OSML, OpenSocial templating, and Data Pipelining. I have no idea what to call this. First person to update the proposal with a reasonable option gets my support.

http://ns.opensocial.org/2008/markup

Added. Also feel free to edit the Wiki directly - that's why we have a Wiki :)
 

-- Adam

 

[Evan Gilbert]

Only open issue I'm aware with Data Pipelining of is whether unhandled tags get 404 or can return null.

Are there other issues or concerns? This is a key feature for 0.9, so please vote if you like it or say why if you don't.

Evan

[Arne Roomann-Kurrik]

Has this been addressed yet? "TBD - How to choose which container parameters (viewer, owner, relationship) to send to the 3rd party server. This has a big impact on cacheability. " 

I'm also curious about how data gets posted to a third party server.  The spec says "Proxied content requests. This data will be POSTed to the developer party server with requests for proxied content. " but doesn't define the format of the POSTed data.  It would also be extremely useful to define a way for this data to be signed by the container, as there currently isn't a way to pass a list of friends in a verifiable manner from the app to the third party server.

~Arne

--
OpenSocial IRC - irc://irc.freenode.net/opensocial

[Evan Gilbert]

On Sat, Nov 22, 2008 at 2:06 PM, Arne Roomann-Kurrik <kur...@google.com> wrote:

Has this been addressed yet? "TBD - How to choose which container parameters (viewer, owner, relationship) to send to the 3rd party server. This has a big impact on cacheability. " 

I haven't seen a discussion of this yet on the proxied content thread (and i think that's where the discussion should go).

I'm also curious about how data gets posted to a third party server.  The spec says "Proxied content requests. This data will be POSTed to the developer party server with requests for proxied content. " but doesn't define the format of the POSTed data.  It would also be extremely useful to define a way for this data to be signed by the container, as there currently isn't a way to pass a list of friends in a verifiable manner from the app to the third party server.

Think that's covered with: "The results of the data pipelining requests will be sent to the href of the <Content> section as POSTed JSON data. The structure will match the [JSON-RPC] format."
(http://wiki.opensocial.org/index.php?title=Data_Pipelining#Proxied_content)

 

[Kevin Brown]

That answers all of my questions. +1

[Arne Roomann-Kurrik]

On Mon, Nov 24, 2008 at 9:47 AM, Evan Gilbert <uid...@google.com> wrote:

On Sat, Nov 22, 2008 at 2:06 PM, Arne Roomann-Kurrik <kur...@google.com> wrote:

Has this been addressed yet? "TBD - How to choose which container parameters (viewer, owner, relationship) to send to the 3rd party server. This has a big impact on cacheability. " 

I haven't seen a discussion of this yet on the proxied content thread (and i think that's where the discussion should go).

I'm also curious about how data gets posted to a third party server.  The spec says "Proxied content requests. This data will be POSTed to the developer party server with requests for proxied content. " but doesn't define the format of the POSTed data.  It would also be extremely useful to define a way for this data to be signed by the container, as there currently isn't a way to pass a list of friends in a verifiable manner from the app to the third party server.

Think that's covered with: "The results of the data pipelining requests will be sent to the href of the <Content> section as POSTed JSON data. The structure will match the [JSON-RPC] format."
(http://wiki.opensocial.org/index.php?title=Data_Pipelining#Proxied_content)

Is the POST implicitly signed then?
 

 

~Arne

On Fri, Nov 21, 2008 at 6:06 PM, Evan Gilbert <uid...@google.com> wrote:

Only open issue I'm aware with Data Pipelining of is whether unhandled tags get 404 or can return null.

Are there other issues or concerns? This is a key feature for 0.9, so please vote if you like it or say why if you don't.

Evan

--

OpenSocial IRC - irc://irc.freenode.net/opensocial

[Louis Ryan]

On Tue, Nov 18, 2008 at 9:14 PM, Evan Gilbert <uid...@google.com> wrote:

Thanks Louis, good feedback.

On Mon, Nov 17, 2008 at 11:23 PM, Louis Ryan <lr...@google.com> wrote:

Some notes:

- Spec does not say for non-proxied non-OSML gadgets how the contents of the preload are made accessible in JS. Something along the lines of opensocial.getPreloaded() which returns DataResponse seems appropriate. A more concise equivalent is also needed in oslite.

I left out the JS APIs in transcription from the original doc. It's been added back in.
 

- Probably best to explicitly state that the attribute names match 1:1 with REST & JSON-RPC parameter names and makeRequest parameter names where a mapping exists

See http://wiki.opensocial.org/index.php?title=Data_Pipelining#Common_request_processing. Feel free to edit if this should be more prominent.
 

- Spec does not state what format the content POSTed for a proxied render is in other than as "POSTed JSON data" I believe the JSON-RPC batch response format is the appropriate one and we should state that.

Agreed, added.
 

- I think filter attribute should be extended to match the REST spec definition of filterBy, filterOp & filterValue. Most of the predefined filters I  believe map to filterOp=<predefined name> e.g. filterOp=@hasApp. Filter operations apply to activites and other types too.

Agreed. I updated the text for people.get to state that these are strings that match the REST spec - however I couldn't actually find where we had defined them in the REST spec.
 

- For <os:DataRequest> I don't think the permutations of method and href work as described. If href is empty and method is "people.get" what are the parameters to the RPC method? I think it would be fine to use the URL encoded form of JSON-RPC parameters in the href and  have the method be "<service>.<operation>" as described in the JSON-RPC spec. Then detection of the special cased RPC call is simply if the method attribute is anything other than the standard HTTP methods. E.g.

<os:DataRequest href="userId=@me&groupId=@self" method="people.get" key="akey" />

I have to admit that this also seems a bit clunky. It may just be cleaner to support <os:RestRequest ...> and <os:RpcRequest> for arbitrary but container resolveable calls and leave DataRequest to just HTTP

One alternative is @method + @params. I'm split between this and <os:RpcRequest> - any preference?

If people are OK with it it may be simpler to say that regardless of whether the request is executed via REST or RPC the syntax is the RPC URL encoded form. I.e. if method=xxx" is specified then the call is an API call to the containers endpoint and href is interpreted as the query parameter list. Given that there is a trivial 1:1 transformation between the RPC URL encoded syntax and REST this should be fine. If people want to decouple the specs a little bit I can move the URL encoding scheme out of the RPC doc and into this one  

 

 

-  I think there's a larger list of relevant makeRequest params that should be documented for DataRequest HTTP use. e.g. OAUTH_SERVICE_NAME. We should state whether the intent is to have full 1:1 functionality with makeRequest and if not list the exceptions

The general goal should be 1:1 functionality but I didn't want to blindly add the fields as I wasn't clear if all of them could be statically defined and if all were in current use (vs. legacy support). Also, a number of the parameters deal with feed retrieval, and I wasn't sure if we wanted / needed to support this with preload.

I'll defer to the experts on this one... Louis - if you or someone else has a clear idea what parameters should be exposed, please update the proposal.

Doing an update now...

 

- Needs to be a description of how containers are allowed to vary behavior and still remain spec compliant. E.g. a container suppressing  viewer information in proxied renders of profile views because it wants to enforce a high degree of cacheability for the proxied content.

Good point. I think we have a few solved issues and one big unsolved issue:
- Containers can already not pass in viewer information (on a given view, or due to permissions). We can just match the results from calling REST APIs when the app is not allowed access to the data
- The proxied content proposal should cover how the data can be cached, based on sign_owner and sign_viewer parameters (or equivalents). The container will probably not send the opensocial_viewer parameter to the 3rd party server if not allowed for a given view. The URL for caching would only include the parameters actually sent to the 3rd party server.
- The tricky issue is that we also need to prevent sending viewer / owner information in the data pipeline if the URL sent to the 3rd party server doesn't contain opensocial_viewer / opensocial_owner.

For this last point, I've made a small update to the proposal:

Elements in a data pipeline referencing the viewer (via @viewer, @me) will return a 403 (Forbidden) error response if the container is unable to also send the opensocial_viewer URL parameter to the 3rd party server. The same holds true for @owner and opensocial_owner URL parameter.

If anyone has better HTTP codes than 403, please update the proposal. On a quick scan I couldn't find a great match.

403 seems about right...
 

[Louis Ryan]

I don't believe the POST body should be signed for the same reasons its not signed for JSON-RPC

[Louis Ryan]

Updates all look good so I'm +1

[Brian Eaton]

Two concerns about os:DataRequest, one major, one minor.

Major concern: the parameters specified for os:DataRequest are out of
sync with the parameters specified for gadgets.io.makeRequest. All of
the options to makeRequest should be permitted to os:DataRequest. In
the current proposal all of the oauth_* parameters are all missing.
Is there any reason that os:DataRequest wouldn't accept all of the
same parameters as makeRequest?

Minor concern: there is some loss of functionality in data request vs
makeRequest Preloads. Preload tags let you specify user pref
substitutions. This comes in useful for accessing OAuth enabled APIs
that require a user-identifier to be specified in requests.

Yahoo's APIs require the user's GUID be specified:
http://developer.yahoo.com/social/rest_api_guide/web-services-guids.html.
The Netflix OAuth APIs are similar.

Not having the ability to specify preload URLs like
"http://www.example.com/api/__UP_guid__/action" is not going to
completely break compatibility with those APIs, since makeRequest is
still around.

Cheers,
Brian

[Arne Roomann-Kurrik]

Sorry, I only meant to ask whether the request itself would be signed (using the same algorithm as RPC), as opposed to not signed at all, not whether the body would be signed. 

I don't see an explicit way to specify that a PeopleRequest would be signed (whereas DataRequest has the authz parameter).

~Arne

On Mon, Nov 24, 2008 at 4:16 PM, Louis Ryan <lr...@google.com> wrote:

[Adam Winer]

On Mon, Nov 24, 2008 at 4:42 PM, Arne Roomann-Kurrik <kur...@google.com> wrote:
> Sorry, I only meant to ask whether the request itself would be signed (using
> the same algorithm as RPC), as opposed to not signed at all, not whether the
> body would be signed.
>
> I don't see an explicit way to specify that a PeopleRequest would be signed
> (whereas DataRequest has the authz parameter).

The PeopleRequest doesn't go to a third-party server, so signing isn't
relevant. An implementation might use a security token internally,
but that's outside the scope of the spec.

What is important is whether a proxied render (type="html" w/@href)
needs to be signed when it requests viewer or owner data. IMO, yes.

-- Adam

[Adam Winer]

Generally, +1.

I'm unhappy, though, with os:DataRequest. I'd much rather see
os:DataRequest limited to implementing gadgets.io.makeRequest. It's
got loads of attributes (more when Brian Eaton's comments are
addressed) that are irrelevant and confusing for issuing OpenSocial
RESTful/JSON RPC requests. For extensibility, either add a generic
extensibility element, or much better define a generic pattern that
could be used to map in any OpenSocial service, e.g. a photos.get
service has <os:PhotosRequest>.

[Scott Seely]

-1

1: I refuse to +1 something that has TBD items. DataRequest still has a
TBD on the draft at
http://wiki.opensocial.org/index.php?title=Data_Pipelining

2: DataRequest has no information explaining what happens for a POST. A
POST means that data is being passed somewhere. We need something that
can be formatted into a POST in order to actually have the POST be of
any value. Consider reducing the set of supported methods to GET if we
MUST have this tag supported for 0.9. We can figure out POST in v.NEXT
without breaking anyone. Another option: to pass data to a DataRequest
tag, only pass the set of tags nested within the <DataRequest>
</DataRequest> tags to the endpoint.

3: DataRequest needs some text explaining how we will pass things like
{Owner.ID} in a GET request. Alternatively, we can say that integration
with other data is not possible in 0.9 and leave this problem for later.
I doubt that this is useful. Suggestion-- move Dynamic Parameters into
the spec proper as opposed to an issue to resolve at implementation. If
we implement #2 as suggested, then we can punt on passing parameters in
a URL to v.NEXT.

> -----Original Message-----
> From: opensocial-an...@googlegroups.com [mailto:opensocial-
> and-gadg...@googlegroups.com] On Behalf Of Adam Winer
> Sent: Monday, November 24, 2008 5:29 PM
> To: opensocial-an...@googlegroups.com
> Subject: [opensocial-and-gadgets-spec] Re: [Updated Proposal] Data
> Pipelining
>
>

[Scott Seely]

To be clear-- the TBD does not have a suggested resolution either. Is it
even important for us to worry about this right now? I think we can
resolve that issue by handling the POST issue up front.

[Arne Roomann-Kurrik]

On Mon, Nov 24, 2008 at 5:05 PM, Adam Winer <awi...@gmail.com> wrote:

On Mon, Nov 24, 2008 at 4:42 PM, Arne Roomann-Kurrik <kur...@google.com> wrote:
> Sorry, I only meant to ask whether the request itself would be signed (using
> the same algorithm as RPC), as opposed to not signed at all, not whether the
> body would be signed.
>
> I don't see an explicit way to specify that a PeopleRequest would be signed
> (whereas DataRequest has the authz parameter).

The PeopleRequest doesn't go to a third-party server, so signing isn't
relevant.  An implementation might use a security token internally,
but that's outside the scope of the spec.

What is important is whether a proxied render (type="html" w/@href)
needs to be signed when it requests viewer or owner data.  IMO, yes.

I was specifically referring to the proxied content use case:  "The results of the data pipelining requests will be sent to the href of the <Content> section as POSTed JSON data."
I can't seem to find any text indicating that this POST will be signed always, never, or optionally by the user.  Essentially, I want to know that I can Data Pipeline the OWNER's friends to my proxied content page and have that POST be signed (even if the body of the request is not signed, I still want to verify that the request data came from the container).

 

[Adam Winer]

Proxied renders can optionally be signed by owner, viewer, or both.
I'd like to see spec language that asserts:

If data pipelining requests are posted to an @href of the <Content>
section, and data is loaded for the viewer, owner, or both viewer and
owner, a container MUST report an error if the render is not signed
correspondingly (by viewer, owner, or both, and with authz not equal
NONE).

-- Adam

[Lev Epshteyn]

On Mon, Nov 24, 2008 at 8:43 PM, Scott Seely <sSe...@myspace.com> wrote:

2: DataRequest has no information explaining what happens for a POST. A
POST means that data is being passed somewhere. We need something that
can be formatted into a POST in order to actually have the POST be of
any value. Consider reducing the set of supported methods to GET if we
MUST have this tag supported for 0.9. We can figure out POST in v.NEXT
without breaking anyone. Another option: to pass data to a DataRequest
tag, only pass the set of tags nested within the <DataRequest>
</DataRequest> tags to the endpoint.

It seems like a fully flexible implementation of POSTs in DataRequests would need to allow nesting of data pipelining tags, so postponing til v.Next makes sense.

 

3: DataRequest needs some text explaining how we will pass things like
{Owner.ID} in a GET request. Alternatively, we can say that integration
with other data is not possible in 0.9 and leave this problem for later.
I doubt that this is useful. Suggestion-- move Dynamic Parameters into
the spec proper as opposed to an issue to resolve at implementation. If
we implement #2 as suggested, then we can punt on passing parameters in
a URL to v.NEXT.

I agree that moving Dynamic Params into the main spec will make DataRequest much more useful. We should also establish a minimal set of keys that will be accessible. I think this should include at least ViewParams, Prefs and the UIDs of Owner and Viewer.

 

[Evan Gilbert]

Possible resolutions for the open items....

POSTing data in <os:DataRequest>
<os:DataRequest> has @method="post", which is confusing because there is no way to specify the content to POST.

The goal of this was to support key/value parameters that might extend beyond the URL length limits for GETs, and not to support arbitrary POST content. I think this is an important use case, especially with Dynamic Parameters, so I updated the spec to allow for a separate "@params" attribute. This has a bit of duplication with the @href (both can contain params), but the flexibility to specify POST is something that I'd like to keep and I couldn't see a cleaner way. The change:

• @params {string} Additional URL parameters, of format "a=b&c=d". Optional.
• @method {string} HTTP method to use, one of "get" or "post". Defaults to "get". Optional.
  • If method is "post", than the "@params" parameters are sent as POST content. If method is "get", then the params are appended to the URL.
  • Note that POSTing arbitrary content is not supported.

Signing of the POSTed data for proxied content
I think we need to make sure that a malicious party can't spoof a different set of friends to get access to data. I think the right soution to this may be to support a parameter that states whether the container generated this request vs. the JavaScript - the only reason way a malicious party can spoof a request is if they can generate an equivalent gadgets.io.make.request(). I'm going to start a separate thread on this.

TBD in DataRequest
This was addressed with Louis' edit to add the makeRequest parameters of @sign_owner and @sign_viewer.

Dynamic params
Seems that people feel this is required, so moved up to the proposal. Text is:

Data pipelining will support a limited subset of the expression language used in OpenSocial templates (based on JSP EL):

• Expressions of format ${A.B} will be supported, where "A" is the key and "B" is the property.
• The key "ViewParams" is reserved and refers to the parameters passed into the gadget rendering, equivalent to [gadgets.views.getParams()]
• Keys can also map to earlier items in the data pipeline. You can get a viewer and then pass the ID to successive requests.
• <os:PeopleRequest> has a special implicit property, "ids", which is used to send along a comma-delimited list of the IDs returned. This is required to enable passing of friends lists.

Note that we will likely still run into issues during implementation, but this at least gives us a starting point

Signing
There was a suggestion that we MUST require that proxied requests with viewer / owner are signed. I think this is already covered in http://wiki.opensocial.org/index.php?title=Data_Pipelining#Proxied_content

"Elements in a data pipeline referencing the viewer (via @viewer, @me) will return a 403 (Forbidden) error response if the container is unable to also send the opensocial_viewer URL parameter to the 3rd party server. The same holds true for @owner and opensocial_owner URL parameter. "

We've been tossing around a few different syntaxes for how to sign, so I had the text refer to the actual signed parameters that will be sent to the 3rd party server. Let me know if this makes sense

os:DataRequest vs os:RpcRequest
I agree with Adam's comments. How do people feel about splitting into:
- <os:MakeRequest> for makeRequest() equivalent calls
- <os:DataRequest> for RPC data request calls?
(note - "RpcRequest" felt a little off, as it expands to "remote procedure call request")

[Adam Winer]

On Tue, Nov 25, 2008 at 10:51 AM, Evan Gilbert <uid...@google.com> wrote:

Possible resolutions for the open items....

POSTing data in <os:DataRequest>
<os:DataRequest> has @method="post", which is confusing because there is no way to specify the content to POST.

The goal of this was to support key/value parameters that might extend beyond the URL length limits for GETs, and not to support arbitrary POST content. I think this is an important use case, especially with Dynamic Parameters, so I updated the spec to allow for a separate "@params" attribute. This has a bit of duplication with the @href (both can contain params), but the flexibility to specify POST is something that I'd like to keep and I couldn't see a cleaner way. The change:

• @params {string} Additional URL parameters, of format "a=b&c=d". Optional.
• @method {string} HTTP method to use, one of "get" or "post". Defaults to "get". Optional.
  • If method is "post", than the "@params" parameters are sent as POST content. If method is "get", then the params are appended to the URL.
  • Note that POSTing arbitrary content is not supported.

Signing of the POSTed data for proxied content
I think we need to make sure that a malicious party can't spoof a different set of friends to get access to data. I think the right soution to this may be to support a parameter that states whether the container generated this request vs. the JavaScript - the only reason way a malicious party can spoof a request is if they can generate an equivalent gadgets.io.make.request(). I'm going to start a separate thread on this.

TBD in DataRequest
This was addressed with Louis' edit to add the makeRequest parameters of @sign_owner and @sign_viewer.

Dynamic params
Seems that people feel this is required, so moved up to the proposal. Text is:

Data pipelining will support a limited subset of the expression language used in OpenSocial templates (based on JSP EL):

• Expressions of format ${A.B} will be supported, where "A" is the key and "B" is the property.
• The key "ViewParams" is reserved and refers to the parameters passed into the gadget rendering, equivalent to [gadgets.views.getParams()]
• Keys can also map to earlier items in the data pipeline. You can get a viewer and then pass the ID to successive requests.

-1 for this in 0.9.  The specific example is not compelling - you can already sign by viewer (or explicitly request the viewer) for all elements.  More generally, mapping to earlier items eliminates the possibility of concurrency, unless implementations define a dependency graph, which will get harder when have broader support for JSP EL.

• <os:PeopleRequest> has a special implicit property, "ids", which is used to send along a comma-delimited list of the IDs returned. This is required to enable passing of friends lists.

Don't see the need for this property.  os:PeopleRequest has userIds.  We need to define type coercion rules, which aren't hard - we have strings and integers, which are obvious, and arrays of strings, which should be coerced using comma-delimited lists.

Note that we will likely still run into issues during implementation, but this at least gives us a starting point

One immediate need:  all EL values will be URL-encoded when used in os:DataRequest.

Signing
There was a suggestion that we MUST require that proxied requests with viewer / owner are signed. I think this is already covered in http://wiki.opensocial.org/index.php?title=Data_Pipelining#Proxied_content

"Elements in a data pipeline referencing the viewer (via @viewer, @me) will return a 403 (Forbidden) error response if the container is unable to also send the opensocial_viewer URL parameter to the 3rd party server. The same holds true for @owner and opensocial_owner URL parameter. "

+1 

We've been tossing around a few different syntaxes for how to sign, so I had the text refer to the actual signed parameters that will be sent to the 3rd party server. Let me know if this makes sense

@authz, @method, and @format are all uppercase in gadgets.io.makeRequest.  Any reason to change?

We're also still missing the OAuth parameters, though I don't personally have a list of those.

os:DataRequest vs os:RpcRequest
I agree with Adam's comments. How do people feel about splitting into:
- <os:MakeRequest> for makeRequest() equivalent calls
- <os:DataRequest> for RPC data request calls?
(note - "RpcRequest" felt a little off, as it expands to "remote procedure call request")

+1 - but what's the syntax for DataRequest?  service, method, params?  Or just a JSON blob including the method?

[Evan Gilbert]

On Tue, Nov 25, 2008 at 11:06 AM, Adam Winer <awi...@gmail.com> wrote:

On Tue, Nov 25, 2008 at 10:51 AM, Evan Gilbert <uid...@google.com> wrote:

Possible resolutions for the open items....

POSTing data in <os:DataRequest>
<os:DataRequest> has @method="post", which is confusing because there is no way to specify the content to POST.

The goal of this was to support key/value parameters that might extend beyond the URL length limits for GETs, and not to support arbitrary POST content. I think this is an important use case, especially with Dynamic Parameters, so I updated the spec to allow for a separate "@params" attribute. This has a bit of duplication with the @href (both can contain params), but the flexibility to specify POST is something that I'd like to keep and I couldn't see a cleaner way. The change:

• @params {string} Additional URL parameters, of format "a=b&c=d". Optional.
• @method {string} HTTP method to use, one of "get" or "post". Defaults to "get". Optional.
  • If method is "post", than the "@params" parameters are sent as POST content. If method is "get", then the params are appended to the URL.
  • Note that POSTing arbitrary content is not supported.

Signing of the POSTed data for proxied content
I think we need to make sure that a malicious party can't spoof a different set of friends to get access to data. I think the right soution to this may be to support a parameter that states whether the container generated this request vs. the JavaScript - the only reason way a malicious party can spoof a request is if they can generate an equivalent gadgets.io.make.request(). I'm going to start a separate thread on this.

TBD in DataRequest
This was addressed with Louis' edit to add the makeRequest parameters of @sign_owner and @sign_viewer.

Dynamic params
Seems that people feel this is required, so moved up to the proposal. Text is:

Data pipelining will support a limited subset of the expression language used in OpenSocial templates (based on JSP EL):

• Expressions of format ${A.B} will be supported, where "A" is the key and "B" is the property.
• The key "ViewParams" is reserved and refers to the parameters passed into the gadget rendering, equivalent to [gadgets.views.getParams()]
• Keys can also map to earlier items in the data pipeline. You can get a viewer and then pass the ID to successive requests.

-1 for this in 0.9.  The specific example is not compelling - you can already sign by viewer (or explicitly request the viewer) for all elements.  More generally, mapping to earlier items eliminates the possibility of concurrency, unless implementations define a dependency graph, which will get harder when have broader support for JSP EL.

How do we pass along viewer/owner friends? I think this would be a common use case.

• <os:PeopleRequest> has a special implicit property, "ids", which is used to send along a comma-delimited list of the IDs returned. This is required to enable passing of friends lists.

Don't see the need for this property.  os:PeopleRequest has userIds.  We need to define type coercion rules, which aren't hard - we have strings and integers, which are obvious, and arrays of strings, which should be coerced using comma-delimited lists.

This is for the previous item - hard to pass a list of friends otherwise, as you only have person JSON objects.

Very open to alternate suggestions for this use case - this was my best guess at a solution.

[Brian Eaton]

On Tue, Nov 25, 2008 at 10:51 AM, Evan Gilbert <uid...@google.com> wrote:
> TBD in DataRequest
> This was addressed with Louis' edit to add the makeRequest parameters of
> @sign_owner and @sign_viewer.

I don't think this is addressed. makeRequest, Preload, and proxied
content support several parameters for OAuth that haven't been moved
into os:DataRequest.

Maybe we need a content fetching spec? There are multiple parts of
the opensocial spec that all independently describe behavior around
signing and caching of requests. We've got:

gadgets.io.makeRequest: javascript binding
Preload: one XML binding
proxied content: second XML binding
os:DataRequest: third XML binding

We could write a description of opensocial content fetching, then have
those other spots reference it instead of cut and paste.

Cheers,
brian

[Adam Winer]

On Tue, Nov 25, 2008 at 11:10 AM, Evan Gilbert <uid...@google.com> wrote:

On Tue, Nov 25, 2008 at 11:06 AM, Adam Winer <awi...@gmail.com> wrote:

On Tue, Nov 25, 2008 at 10:51 AM, Evan Gilbert <uid...@google.com> wrote:

Possible resolutions for the open items....

POSTing data in <os:DataRequest>
<os:DataRequest> has @method="post", which is confusing because there is no way to specify the content to POST.

The goal of this was to support key/value parameters that might extend beyond the URL length limits for GETs, and not to support arbitrary POST content. I think this is an important use case, especially with Dynamic Parameters, so I updated the spec to allow for a separate "@params" attribute. This has a bit of duplication with the @href (both can contain params), but the flexibility to specify POST is something that I'd like to keep and I couldn't see a cleaner way. The change:

• @params {string} Additional URL parameters, of format "a=b&c=d". Optional.
• @method {string} HTTP method to use, one of "get" or "post". Defaults to "get". Optional.
  • If method is "post", than the "@params" parameters are sent as POST content. If method is "get", then the params are appended to the URL.
  • Note that POSTing arbitrary content is not supported.

Signing of the POSTed data for proxied content
I think we need to make sure that a malicious party can't spoof a different set of friends to get access to data. I think the right soution to this may be to support a parameter that states whether the container generated this request vs. the JavaScript - the only reason way a malicious party can spoof a request is if they can generate an equivalent gadgets.io.make.request(). I'm going to start a separate thread on this.

TBD in DataRequest
This was addressed with Louis' edit to add the makeRequest parameters of @sign_owner and @sign_viewer.

Dynamic params
Seems that people feel this is required, so moved up to the proposal. Text is:

Data pipelining will support a limited subset of the expression language used in OpenSocial templates (based on JSP EL):

• Expressions of format ${A.B} will be supported, where "A" is the key and "B" is the property.
• The key "ViewParams" is reserved and refers to the parameters passed into the gadget rendering, equivalent to [gadgets.views.getParams()]
• Keys can also map to earlier items in the data pipeline. You can get a viewer and then pass the ID to successive requests.

-1 for this in 0.9.  The specific example is not compelling - you can already sign by viewer (or explicitly request the viewer) for all elements.  More generally, mapping to earlier items eliminates the possibility of concurrency, unless implementations define a dependency graph, which will get harder when have broader support for JSP EL.

How do we pass along viewer/owner friends? I think this would be a common use case.

Developers that need the combination of viewer/owner friends with a makeRequest to their server can use proxied rendering.  For 0.9, I'd rather be minimal and punt complexity (like inter-request dependencies) to v.Next.

• <os:PeopleRequest> has a special implicit property, "ids", which is used to send along a comma-delimited list of the IDs returned. This is required to enable passing of friends lists.

Don't see the need for this property.  os:PeopleRequest has userIds.  We need to define type coercion rules, which aren't hard - we have strings and integers, which are obvious, and arrays of strings, which should be coerced using comma-delimited lists.

This is for the previous item - hard to pass a list of friends otherwise, as you only have person JSON objects.

Very open to alternate suggestions for this use case - this was my best guess at a solution.

Proxied rendering is what I'd use.  YMMV.

-- Adam

[Lane LiaBraaten]

@Evan - Can you give an example of "Keys can also map to earlier items in the data pipeline. You can get a viewer and then pass the ID to successive requests."?

@Adam - Can you expand on the "possibility of concurrency"?  (sounds like an O'Bama book to me =P)

-Lane

[Lane LiaBraaten]

On Tue, Nov 25, 2008 at 6:23 PM, Lane LiaBraaten <llia...@google.com> wrote:

@Evan - Can you give an example of "Keys can also map to earlier items in the data pipeline. You can get a viewer and then pass the ID to successive requests."?

@Adam - Can you expand on the "possibility of concurrency"?  (sounds like an O'Bama book to me =P)

s/O'Bama/Obama/

...last time I try to make a joke on the list.

Adam - to be clear, you're just -1 that one bullet point, right?

-Lane

[David Byttow]

On Evan's behalf, I believe he was referring to something like this:

<script type="text/os-data">
  <os:PersonRequest key="user" userid="@me"/>
  <os:DataRequest key="scores" href="http://developersite.com/getscores?user=${user}"/>
</script>

Granted it's a bit contrived and one could argue that the developer would get passed in the ID and thus be able to just look-up the scores directly given the passed in user.

On Tue, Nov 25, 2008 at 6:23 PM, Lane LiaBraaten <llia...@google.com> wrote:

[Adam Winer]

On Tue, Nov 25, 2008 at 6:57 PM, David Byttow <david...@google.com> wrote:

On Evan's behalf, I believe he was referring to something like this:

<script type="text/os-data">

  <os:PersonRequest key="user" userid="@me"/>
  <os:DataRequest key="scores" href="http://developersite.com/getscores?user=${user}"/>
</script>

Yes, that's an example.  For viewer/owner, it's not compelling.  For viewer friends (for example) it's more interesting:

  <os:PeopleRequest key="friends" userIds="@me" groupId="@friends"/>
  <os:DataRequest key="scores" href="http://developersite.com/getscores?user=${friends}"/>

... which might download scores for all friends.   More generally, the following would be legit:

  <os:ViewerRequest key="viewer"/>

  <os:DataRequest key="players" href="http://developersite.com/getsomeuserlist?user=${viewer}"/>

  <os:PeopleRequest key="playerPeople" userIds="${players.ids}"/>

  <os:DataRequest key="scores" href="http://developersite.com/getscores?users=${playerPeople}"/>

Without inter-dependencies, all requests can be loaded concurrently by the server (not "must", just "can").  With the last example, though, you'd have no choice but load each of the 4 serially.

I don't yet see that this extra complexity buys us much, especially not for proxied rendering, and would rather that we re-evaluate for v.Next.

W/regard to the -1 - yes, it's just for that bullet point.  I'm +1 otherwise.

-- Adam

[Louis Ryan]

Im with Adam on this point. I would rather address the issue of references across data requests with a more general solution focused on composability that will work in all forms of the protocol. Theres no reason that code executing a people request and then a data request dependent on it should not be able to leverage request composition at the protocol level, building it in as a special affordance for just pipelining now I think is the wrong choice.