Proposal for OpenSocial 1.0 iteration

[Lane LiaBraaten]

Hey Everyone,

It's been about 6 months since we published OpenSocial v0.9 and the new features are becoming widely available in production - generally our cue to kick off a new iteration of the spec process. However, as we've made rapid progress expanding the capabilities of OpenSocial, the specification itself has suffered.  Since publishing v0.9 we've all uncovered many inaccuracies, contradictions, and obsolete sections, resulting in a spec that is hard to interpret and nearly impossible to implement consistently across dozens of containers.

Rather than go through the normal round of spec proposals and prototypes, I propose we focus on two things to get our specification back in order and ready for future revisions.

First, let's clean up the spec documents.  I'd like to go beyond the bandaids of fixing typos and formatting.  The organization of the spec makes it hard to track down where things are actually defined.  We've got duplicated content (that's sometimes contradictory) and obsolete concepts bloating the documents. And we're lacking specificity - some things are just left up to the reader/implementor!

Second, let's formalize what we mean by 'extensions'. It's clear that containers have varied and ever-changing requirements and we started to introduce the concept of extensions in v0.9.  We need to be more rigorous about defining extensions and the process by which they're adopted into the core spec.

IMO, getting these two things right will leave us with a version of OpenSocial that we're happy to call 1.0 - a revamped and sustainable specification that can help solve compatibility issues by providing a clear and complete definition of OpenSocial.

In short, the proposal is "For the next spec iteration, let's fix the existing spec rather than add new features, and clarify our framework for extensions so we can better accommodate new features in future revisions."

On process, I think we can treat this like any other decision, so send in your +/-1's and we'll leave the discussion open for at least 5 days.

Cheers,
Lane

[Scott Seely]

We’ll also need someone to step up as spec editor. I thought I’d be able to do this, but my past month has shown me that I won’t have the bandwidth to do the  job properly.

[goosemanjack]

I think this is a wise approach for 1.0. I certainly have a list of
items for clarification/deprication. Not an official announcment, but
we should be launching the entirety of 0.9 over the next quarter (I
know, we've all heard this song before). That should help the
discussion as well since we'll be able to compare and contrast the
implementations. I've been holding off on sending in proposals until
we're live since an object example helps the discussion.

I am +1 on your proposal, Lane, with the exception that I'd like to
see two of the 1.0 feature proposals already on the table come up for
vote:

* The os:Var extension appears very useful and we're likely to
impl with or shortly after 0.9
* The javascript API for reprocessing data tags is very important.

If anyone needs I can dig up the original threads on these two items.

I can probably take on the spec editor role provided there are no
surprises with our launch of 0.9.
--
clc

[Arne Roomann-Kurrik]

I've been talking offline with Lane about what we wanted to see from this revision and wanted to summarize my points for the rest of the community.

I started thinking about the problems with the spec and the specification process as I saw them during the 0.9 phase:

 * Spec complexity

   The spec has too many dependencies right now.  The RPC spec refers to the REST spec which refers back to the core spec.  The osapi spec claims to map 1:1 to the RPC spec, but has contradictory examples and documentation.  Finding out the complete parameter set accepted by a single RPC, REST or osapi call requires flipping through three documents.

 * Too many features moving at once

   There was just too much going on for 0.9 for any one person to keep on top of.  We introduced features which incorporated changes across several documents at once (see: Spec complexity).   Features were modified during the voting and prototype phases, and the spec docs themselves were changing from other, interrelated features.  The result was errors in documentation and examples that live in the spec today.

 * Arbitrary timelines

   We wound up having to cut a lot of functionality for proxied content because of an arbitrary deadline.  Why?  When did pushing a new version of the spec become more important than having a really useful specification for proxied content? 

 * Prototyping chain of responsibility was undefined

   Features were dropped because nobody bothered to prototype them.  That doesn't make a lot of sense to me - if an application developer wants a change to the way apps work, why do they have to make modifications to server code for it to be included?  If you need to get buy-in from a container to get a prototype made, why bother going to the spec group first, then?

 * XXXL Scope

   Some people wanted REST and others wanted RPC.  Some people wanted XML, others JSON, others ATOM.  Instead of compromising and coming to an agreement, we have 6 permutations of the protocol and content-type for servers and client libraries to support.  Few, if any, suppport all of these.

 * "Mix & match" attitude to the spec

   Because of the way that features get woven into many different portions of the spec, it becomes very difficult to determine the entire scope of any given feature.  For example, osapi.* has been working in some form on PHP and Java shindig, but osapi.http was overlooked in both versions because it didn't have a corresponding RPC endpoint defined in the spec anywhere.  Because the spec is difficult/impossible to comprehend fully, it is almost required to pick and choose portions of the spec to implement.

These issues are significant, but I believe they're quite solvable, even in the next (1.0) iteration of the spec.  If I were king, I'd love to see the following:

1.) To address complexity, we need to "refactor" the spec into something that can be more easily read and understood by a single person.  Lane and I have started brainstorming what that could look like, and hope to have a sample available for review here soon.  The biggest change will be to pull together REST, RPC, osapi, and Data Pipelining definitions into one consistent data access portion of the spec.

2.) Because of the large number of text changes needed for #1, I believe that this revision should not change anything specified by the 0.9 spec.  Keeping track of stylistic changes in a document is possible, and keeping track of semantic changes to a document is possible, but doing both at once is asking for trouble.  We want changes to the spec to be well reviewed, but making the changes to complex is a good way to make sure mistakes make their way in.

3.) We should divorce the timelines for the core spec and logical blocks of changes, or extensions to the spec.  The example I like to use the most is OAuth's body signing extension: http://oauth.googlecode.com/svn/spec/ext/body_hash/1.0/oauth-bodyhash.html.  This extension was allowed to come to maturity on its own, isolated from the timeline of the OAuth core spec.  It was written in a response to a problem which manifested on some OpenSocial containers, iterated on, and implemented quickly.  OAuth 1.0 did not need to be versioned or changed in any way.  Eventually, when OAuth v.NEXT comes out, maybe this extension will be seen as important enough to roll into the core spec.  At that point, it will be a mature proposal with several production implementations and strongly defined language, making incorporation so much easier.  

4.) For an extension to be rolled into the main spec, it should be implemented to full compliance in production on at least two infrastructures.  This will remove the need to block the main spec while prototypes are built for significant new features or groups of features. Containers wanting to implement new features still have a specification to refer to.

5.) Extensions will be blocked on implementation, so we will hopefully see realistic extension scopes, as opposed to trying to get every possible technology included (see: XXXL Scope).  I expect at least a few extension documents to be written descriptively, not prescriptively.

6.) I know there are people who will disagree with me here, but I believe that having a smaller, more comprehensible core spec with extensions will be more beneficial for compatibility than rolling everything into one large spec document.   Honestly, we don't have a system for enforcing compliance, and some containers are going to pick and choose portions of the spec to implement.  If we can deliver the spec in more manageable chunks, then hopefully it'll be more realistic for a container to implement an entire chunk at a time, opposed to the "part of this, part of that" approach that's in use today. 

I know that what I'm describing superficially seems like there will be no new features for OpenSocial until the next whole iteration (~6 months), but let me try to squash that sentiment here - if we can agree to separate the core and new features, then I expect that the 1.0 revision can be done in isolation and completed much faster than 0.9 was.  I also expect that extension drafts can begin right away and operate independently, ensuring that new features are sufficiently thought out and prototyped before we even need to consider putting them into core.  

Chris, I believe that those features (and a couple more) could be rolled up into an extension proposal.  There's also another option we could discuss - to make Templates an extension doc by itself.  The number of extensions to OS templates currently in production seems to indicate that that feature could likely benefit from iterating on its own for a while longer.

~Arne

 

[Tim Moore]

I think these are all great proposals.

+1 to everything proposed by Lane & Arne, and especially to Scott's
suggestion of a designated spec editor. The current version reads
like a bit of a patchwork (not for no reason, I know). Some clean-up
would be greatly appreciated. I've got some notes on the gadgets spec
in particular that I need to type up into a patch.

One thing that I think needs to be decided is the future of gadgets as
a separate spec. Right now it's kind of implied that the gadgets spec
is independent of the rest of OpenSocial, but there are subtle
dependencies and inconsistencies between the two. Each spec refers to
concepts that are only defined in the other, so neither one can really
be said to be more "core" with the other extending it. And there are
often multiple, incompatible ways of doing the same things, such as
makeRequest vs. osapi.http and UserPrefs vs. AppData. It would be
good to clarify the direction this should go in and restructure the
documents accordingly.

I have no strong opinion on the two features mentioned by Chris, they
sound fine to me.

-- Tim

> OAuth's body signing extension:http://oauth.googlecode.com/svn/spec/ext/body_hash/1.0/oauth-bodyhash....

[Arne Roomann-Kurrik]

On Fri, Sep 4, 2009 at 12:08 PM, Tim Moore <tmo...@atlassian.com> wrote:

I think these are all great proposals.

+1 to everything proposed by Lane & Arne, and especially to Scott's
suggestion of a designated spec editor.  The current version reads
like a bit of a patchwork (not for no reason, I know).  Some clean-up
would be greatly appreciated.  I've got some notes on the gadgets spec
in particular that I need to type up into a patch.

One thing that I think needs to be decided is the future of gadgets as
a separate spec.  Right now it's kind of implied that the gadgets spec
is independent of the rest of OpenSocial, but there are subtle
dependencies and inconsistencies between the two.  Each spec refers to
concepts that are only defined in the other, so neither one can really
be said to be more "core" with the other extending it.  And there are
often multiple, incompatible ways of doing the same things, such as
makeRequest vs. osapi.http and UserPrefs vs. AppData.  It would be
good to clarify the direction this should go in and restructure the
documents accordingly.

I agree - we really have two specifications here - the data formats and access methods, and then the whole gadget/view/proxied content/template side.  I would like to see how people feel about dividing the spec along those lines, as opposed to the (now) arbitrary and fuzzy OpenSocial/Gadgets line.

~Arne

[Chris Chabot]

I really like this overall direction the spec really could do with a good cleanup, and as it's mentioned in other parts of this thread, there's a lot of potential improvements to be made here.

I would however like to go for a 'minimal chances' instead of a complete functionality freeze, there's a few new items that I feel should definately be part of the 1.0 spec (or as extensions, more on that later):

• Pub-Sub, IBM's working hard to come up with a pubsub proposal that would be compatible with the OpenAjax pubsub mechanism, this would do wonders for the interoperability between their own iWidget's and OpenSocial gadgets, and would help a lot to get them to support OpenSocial in a lot of their products. Since pubsub was dropped from the 0.9 spec we've always said that it would be something we'd address in v.Next
• Templating language, right now the spec states that it's "somewhat like JSP, and supports some of its features". Perhaps this would fall under 'cleaning the spec' but I very much prefer to have a bit more exact and descriptive language there.
• osx:Flash, navigateTo, var.. we shouldn't have essential bits of the spec live in undocumented limbo & roll these into the official os: space & spec them
  
• Payments API, lots of containers and app developers alike have hold off on this because it wasn't officially part of the spec yet, it should be an optional part (or extension), but giving it the 'official' label would do it a lot of good.
• ActivityStrea.ms, and this is I think the largest but also most important issue for this spec iteration & future direction of OpenSocial, and something I feel we should do something with. Personally I would much prefer ripping out the old activities and replacing it with the activitystream counterpart (with the old format still being supported in older opensocial features / shindig magic wrappers), especially to prevent what happened with the JSON-RPC, REST (xml, atom, json) resulting complexities; However I'm also willing to settle for a more realistic extension spec :)
  

I would further more also be hugely in favor of removing what we've been calling depreciated for a while now, and where possible make things MUST instead of MAY's where we can, JSON-RPC for instance would be a great candidate for that. Inconsistencies between implementations are one of our largest challanges and where we can move to reduce them, we really shouldn't avoid making the tough decisions to make that happen.

On the topic of extensions, I'm finding my self caught between thinking that's a great way to allow containers to experiment and develop new concepts outside of the spec process, and fearing that it will become a way to punt on the harder to solve issues. Given that there's a lot of gadget developers that cite portability issues between containers as a reason not to invest to much in that, anything that moves us away from the 'any app, anywhere' long term vision shouldn't be encouraged in my opinion.. it's hard enough to target a multitude of containers as it is, and if they all support different payment/templating/proxied content/data pipelining/pubsub api's, well then we've put the final nail in the coffin of the portability dream and resided to that fact that gadgets can only practically be created for one specific platform (which, face it, given that the largest container only represents about 15% of the opensocial landscape, means *you* won't get most of the popular and great apps)

So extensions is great, because (as kevin brown pointed out), we simply can't have the expertise to define all the social operations on the web, and distilling down the spec to more of the essentials is fine too; Just as long was we have a very careful description and process in place to decide if something should be a extension or not, it really shouldn't be an easy escape and a tool for letting go of our goal for platform consistency.

[Chris Chabot]

On Fri, Sep 4, 2009 at 8:46 PM, Arne Roomann-Kurrik <kur...@google.com> wrote:

I've been talking offline with Lane about what we wanted to see from this revision and wanted to summarize my points for the rest of the community.

I started thinking about the problems with the spec and the specification process as I saw them during the 0.9 phase:

 * Spec complexity

   The spec has too many dependencies right now.  The RPC spec refers to the REST spec which refers back to the core spec.  The osapi spec claims to map 1:1 to the RPC spec, but has contradictory examples and documentation.  Finding out the complete parameter set accepted by a single RPC, REST or osapi call requires flipping through three documents.

What is your proposed solution to this btw? As someone who's implemented a lot of this I completely agree with your assessment, but is this a suggestion to fix up the documentation, the spec, or completely rework how the data models & interfaces are described?
 

 * Arbitrary timelines

   We wound up having to cut a lot of functionality for proxied content because of an arbitrary deadline.  Why?  When did pushing a new version of the spec become more important than having a really useful specification for proxied content? 

This is a very interesting point, I do agree with you that arbitrary deadlines aren't always a good thing, but on the other hand I also talk to a couple of very large enterprise organizations (lockheed, ibm, etc) who are willing to implement OpenSocial in a new or a whole line of existing products, but arbitrary release cycles means they can't align their organization to it.. They need army's of lawyers to audit every line of code for it's legality, army's of product teams to integrate the changes into their products, and have a massive sales, support etc organization as well that needs to know what's going to happen when.. If we want to take the step from grass-roots to the enterprise, we actually need a completely predictable release cycle.

Now if we decide to go that way (and I think it would be a great win for OpenSocial if we did), and we decide on doing a spec revision every X months, and release an implementation of said spec an N months after the spec was finalized, we'll have to make the choice on a case by case basis if it's better to 'ship with what we have now' or remove it from the spec until the next release cycle because it couldn't be completed in time.. Not always the easiest thing to live with, but again, getting OpenSocial into the rest of the world is a very nice thing to accomplish too :)

 

 * Prototyping chain of responsibility was undefined

   Features were dropped because nobody bothered to prototype them.  That doesn't make a lot of sense to me - if an application developer wants a change to the way apps work, why do they have to make modifications to server code for it to be included?  If you need to get buy-in from a container to get a prototype made, why bother going to the spec group first, then?

It sounds like this is 3 separate issues all rolled into one delicious complaint :)

1) Prototyping, I think we all see the reason why this is required, you really need a proof of concept to know if something works in practice, so I wouldn't be in favor of removing this from the spec process, otherwise we'd likely repeat the 0.8 -> 0.8.1 type of revisions (which prompted it to be a part of the spec process)

2) Who is responsible for the prototyping? I don't think this is a problem we can easily solve, most of the teams working on implementations (myspace, php & java shindig as well as independent implementations by avatarsunited, idtail, etc) are resource constrained in one way or another, and each one of those will always look at what is the most efficient way to spend their resources for their own business.

If a great case was made why a new feature is important, one would hope that it would be important to all of those implementers to get it out there asap, but I think that the fact that no implementation of 0.9 is complete yet (though the major 3 hope to have this done in Q3/Q4) shows that we are working with a resource constrained model.

Now this doesn't mean that we should give in and not push for having everything prototyped, but getting buy-in so that it's prioritized by at least one of the implementers will probably never be avoidable.

3) Why then bother with the spec list? I think it's a bit like democracy, it's the best everything we tried, but in no way perfect.. Having a way for all these different types of socially interested entities be involved in the spec processes has led to a spec that can be used for a multitude of things, without this open spec process we'd probably would've had exactly what we set out to avoid, per site specific implementations of API's. I agree that there's a lot of improvement that can be made, but I'm not at all convinced that since we don't get a perfect score, it should be abandoned :)

 

 * XXXL Scope

   Some people wanted REST and others wanted RPC.  Some people wanted XML, others JSON, others ATOM.  Instead of compromising and coming to an agreement, we have 6 permutations of the protocol and content-type for servers and client libraries to support.  Few, if any, suppport all of these.

See this is what bothers me about the extensions suggestion (and before that about the MAY's in the spec), MAY's and extensions are intended for border cases, for things that are likely not to be implemented in most implementations... however it often got used as a way to punt on hard choices, and as a result we have this plethora of protocol options.

Following this reasoning we should try to move away from extensions and MAY's instead of embracing it.. however a lot of people do give the impression they would prefer this.. Not sure what to make of that yet.

In my opinion we should make things like the payments, albums & media API's extensions (since not enough of the social landscape will support all of them), and move things like templating, proxied content, json-rpc etc into the core spec and make them MUST's.

Anything else will only cause this to become worse and not better, so I'm slightly confused about the mixed message here.

 

 * "Mix & match" attitude to the spec

   Because of the way that features get woven into many different portions of the spec, it becomes very difficult to determine the entire scope of any given feature.  For example, osapi.* has been working in some form on PHP and Java shindig, but osapi.http was overlooked in both versions because it didn't have a corresponding RPC endpoint defined in the spec anywhere.  Because the spec is difficult/impossible to comprehend fully, it is almost required to pick and choose portions of the spec to implement.

A easier to read, consistent and condensed spec would help a lot to prevent such issues, but it's also good to keep in mind that no implementations are 0.9 complete yet, so in some ways it's to early to call.

I would however be in favor of having language in the spec that to call your self 'OpenSocial X' you need to implement everything of said spec, as long as it is indeed readable and implementable :)

 

These issues are significant, but I believe they're quite solvable, even in the next (1.0) iteration of the spec.  If I were king, I'd love to see the following:

1.) To address complexity, we need to "refactor" the spec into something that can be more easily read and understood by a single person.  Lane and I have started brainstorming what that could look like, and hope to have a sample available for review here soon.  The biggest change will be to pull together REST, RPC, osapi, and Data Pipelining definitions into one consistent data access portion of the spec.

+1, and please lets make RPC a MUST and not a MAY :)
 

2.) Because of the large number of text changes needed for #1, I believe that this revision should not change anything specified by the 0.9 spec.  Keeping track of stylistic changes in a document is possible, and keeping track of semantic changes to a document is possible, but doing both at once is asking for trouble.  We want changes to the spec to be well reviewed, but making the changes to complex is a good way to make sure mistakes make their way in.

3.) We should divorce the timelines for the core spec and logical blocks of changes, or extensions to the spec.  The example I like to use the most is OAuth's body signing extension: http://oauth.googlecode.com/svn/spec/ext/body_hash/1.0/oauth-bodyhash.html.  This extension was allowed to come to maturity on its own, isolated from the timeline of the OAuth core spec.  It was written in a response to a problem which manifested on some OpenSocial containers, iterated on, and implemented quickly.  OAuth 1.0 did not need to be versioned or changed in any way.  Eventually, when OAuth v.NEXT comes out, maybe this extension will be seen as important enough to roll into the core spec.  At that point, it will be a mature proposal with several production implementations and strongly defined language, making incorporation so much easier.  

Please do keep in mind that while this is a great idea, it's also got a a huge pandora's box potential.

 

I believe that those features (and a couple more) could be rolled up into an extension proposal.  There's also another option we could discuss - to make Templates an extension doc by itself.  The number of extensions to OS templates currently in production seems to indicate that that feature could likely benefit from iterating on its own for a while longer.

That sounds great in theory, but in practice that will probably mean that it's not usable by gadget developers yet... I mean if orkut supports templating language r23534 and myspace supports templating language r01, which one is a gadget dev supposed to use? Which tags are safe to use?

The OAuth body has extension method worked because it wasn't depended on while it was being developed and was only fully launched when the spec it's self was done... With templating however we are already telling people to use it, so you're trying to rebuild a plane in mid flight... do we really want to tell gadget dev's that the templating spec they are using might change as their gadget is already live for millions of end-users and risk random breakage? Or how do you propose we'd deal with such situations?

And with a separate still iterating spec extention, chances are that all major implementations (myspace, java & php shindig) will work differently, so all dependent containers will have a subtly different templating implementation, and most of the new additions would only work on 1 or 2 (out of the 50+) containers .. I can't imagine what could be less motivating for a developer to start using it, any developer in his right might would just avoid it all together.

Let alone trying to sell this to the enterprise's that are looking to adopt OpenSocial as a standard, 'yes you can use this svn snapshot, and it might work, partially, but differently from the rest of the OpenSocial world', that's a pitch I don't look forward to doing :)

If you feel templating isn't done yet, we should remove it from the spec, and roll it back in once the templating spec is actually done, anything else is probably going to do more damage then good

[Jon]

I'm one of the few on this group who use gadgets in a non-opensocial
context. The more I talk to our internal developers, the more uses we
find for the gadget framework as ways to bring third party content to
pages with ease. Today there are not enough of the non-opensocial
developers to justify 2 projects, but I believe that a good separation
of the fundamental gadget infrastructure from the applications will
enable additional applications to be built upon gadgets.

If the opensocial group is amenable to a separation of core gadget
functionality from opensocial, we are here to help, contribute and
serve as a test case for that split.

Our very visible use is: http://developer.ebay.com/products/selling-manager-applications/,
gadgets also appear at several other places in our site, allowing 3rd
party developers a chance to add content to the pages.

Jon

[Mark W.]

A few additional thoughts on some of the points Chris has made...

As he pointed out, we are in the process of getting ready to propose
some of the work we've been doing around pub/sub to OpenSocial. We've
been looking to become more engaged in OpenSocial and pub/sub is an a
capability that would really help create powerful mashups. Hopefully,
with some of the work we've done around the Open Ajax technology, this
will also help bring the two communities together and in doing so,
strengthen both.

I also agree with Chris that we need a way to work with extensions.
We've come across several use cases having a well defined extension
mechanism would be very helpful. It's important that extensions don't
become a way to punt when something becomes hard. It's possible that
we could work some text into the spec around this. A while ago, I
worked on the OVF specification (from DMTF) and we discussed a similar
issues. I like the idea of requiring implementations, but I'm a bit
hesitant b/c some of them may not be publicly accessible, e.g. ship in
a product behind a firewall.

I'm also in favor of reducing the complexity of specifications. The
collection of material can be very confusing and difficult to grok.

As far as a spec editor goes... One person to lead the effort sounds
great. I'm more than happy to work closely with the lucky volunteer
and do whatever is necessary to reduce the load. Hopefully, this will
be one direct way I can help contribute to this effort.

-Mark W.

On Sep 6, 4:40 am, Chris Chabot <chab...@google.com> wrote:
> I really like this overall direction the spec really could do with a good
> cleanup, and as it's mentioned in other parts of this thread, there's a lot
> of potential improvements to be made here.
>
> I would however like to go for a 'minimal chances' instead of a complete
> functionality freeze, there's a few new items that I feel should definately
> be part of the 1.0 spec (or as extensions, more on that later):
>

>    - Pub-Sub, IBM's working hard to come up with a pubsub proposal that

>    would be compatible with the OpenAjax pubsub mechanism, this would do
>    wonders for the interoperability between their own iWidget's and OpenSocial
>    gadgets, and would help a lot to get them to support OpenSocial in a lot of
>    their products. Since pubsub was dropped from the 0.9 spec we've always said
>    that it would be something we'd address in v.Next

>    - Templating language, right now the spec states that it's "somewhat like

>    JSP, and supports some of its features". Perhaps this would fall under
>    'cleaning the spec' but I very much prefer to have a bit more exact and
>    descriptive language there.

>    - osx:Flash, navigateTo, var.. we shouldn't have essential bits of the

>    spec live in undocumented limbo & roll these into the official os: space &
>    spec them

>    - Payments API, lots of containers and app developers alike have hold off

>    on this because it wasn't officially part of the spec yet, it should be an
>    optional part (or extension), but giving it the 'official' label would do it
>    a lot of good.

>    - ActivityStrea.ms, and this is I think the largest but also most

[Lane LiaBraaten]

Lots of good feedback in this thread. The comments/caveats/suggestions seem to fall into three categories:
 1) Things related to "cleaning up" the spec
   - Condense REST, RPC, osapi, and Data Pipelining definitions into one, "data access" section of the spec
   - Be more specific about what parts of JSP is supported by templates
   - Gadget XML reference (I'm adding this one myself :)
   - Deprecate/remove opensocial.* (gadgets.io.makeRequest vs osapi.http is related)
   - Make JSON-RPC a MUST instead of a MAY
   - UserPrefs vs AppData
   - Gadgets and OpenSocial as separate specs
 2) Extension/Spec process
   - How do we allow for extensions while maintaining compatibility across containers?
   - Tracking maturity of an extension (from idea, to prototype, to production/official)
   - Moving to a regular release cycle for the OpenSocial spec
   - Integrating extensions into the core spec
 3) Individual extensions we need to work on
   - JS API for reprocessing data pipelining tags
   - PubSub
   - Templates extensions (osx:flash, parseJson, variable)
   - Payments
   - ActivityStrea.ms
 
Each of these warrant further discussion and the items in 1) and 2) are akin to the "proposals" we'd see in any spec iteration: they'll need to be discussed and we'll ultimately need a patch to the spec document that we can vote on.

How we handle the items in 3) depends largely on the outcome of 2). I'd like to reiterate Arne's point, that even if we don't make functional changes to the core spec in this iteration, we can still make progress on implementing extensions.

On process, we need to tally votes (so far we have +1's from Chris Cole, Tim Moore, and myself) and leave this thread open for 5 days (i.e. until 10/14).  However, I don't think we need to postpone discussions related to some of the topics brought up already, so feel free to kick off some new threads to keep discussion focused (I'm about to start one on extensions).

Cheers,
Lane

[Mark W.]

Lane,
From you note that started this thread, here’s what I think you are
asking us to +/- 1 on:

<<In short, the proposal is "For the next spec iteration, let's fix
the existing spec rather than add new features, and clarify our
framework for extensions so we can better accommodate new features in

future revisions.”>>
Based on the above +1.

Regarding the cleanup, please count me in to help editing, structure,
et. please count me in.

-Mark W.

> ...
>
> read more »

[Mark W.]

Chris,

<<What is your proposed solution to this btw? As someone who's
implemented a
lot of this I completely agree with your assessment, but is this a
suggestion to fix up the documentation, the spec, or completely rework
how
the data models & interfaces are described? >>

From my perspective, cleaning up the documentation and the spec are in
scope for 1.0. Changing the data models and interfaces should be
avoided. I'm all for stabilizing what we've got now. As an aside,
regarding the data model...In the Enterprise OpenSocial meet up, there
were some interesting ideas about how your profile is really an
aggregate of information. This should be explored, but not for 1.0
(IMHO).

<< If we want to take the step from grass-roots to the enterprise, we
actually need a completely predictable release cycle. >>

Agree completely. I'm one of the "large enterprises" with "armies of
lawyers" and having predictability of release cycles is a must.
Getting a road map of 1.0 and 1.x allows me to work with my product
teams to know what version they should plan on picking up and
supporting. Also, if we have a "must have" requirement, this lets us
plan on how we can work it into the spec and then opensource.

<<Prototyping chain of responsibility was undefined...>>
I agree with Chris C's point on the need for prototyping. I've got one
caveat/question/et... In some instances, the implementation of the new
capability may be bound to a container and not easy to separate into a
stand alone prototype. Granted, we haven't hit this situation yet, but
we could easily. Take group support. We've all talked about how
important this is and we'd all like to have it. It's conceivable,
since many containers have something like this today, that a prototype
can't easily be distilled from the guts of something that's working.
As a community, we have a choice to make on how we get a prototype in
these circumstances.

<<See this is what bothers me about the extensions suggestion (and
before that
about the MAY's in the spec), MAY's and extensions are intended for
border
cases, for things that are likely not to be implemented in most
implementations... however it often got used as a way to punt on hard
choices, and as a result we have this plethora of protocol options. >>

Lane's started a wiki page on extensions (http://wiki.opensocial.org/
index.php?title=Extensions) which stared from this post (http://
groups.google.com/group/opensocial-and-gadgets-spec/browse_thread/
thread/edcede8d134002b5/26b3483599aa8b16?show_docid=26b3483599aa8b16).

In general, I agree with your comment on MAY's in the spec. This only
makes programming harder. To me though, this is independent of
extensions, per se. We need a mechanism for the spec to grow and
evolve without changing the underlying programming model from major
version to major version. I think we also want to avoid massive
extension proliferation as well. Your idea of treating media and
payments as extensions sounds good, as does requiring proxy et...

One thing not mentioned is the REST APIs. These are very important
from an integration perspective. As we go forward, we should think
about what it means to be opensocial compliant if you ONLY do the REST
APIs.

I'm looking forward to working with everyone on the 1.0 and 1.next. I
"plus one'd" this on another response.

Thanks
-Mark W.

> >http://oauth.googlecode.com/svn/spec/ext/body_hash/1.0/oauth-bodyhash....

> >  This extension was allowed to come to maturity on its own, isolated from
> > the timeline of the OAuth core spec.  It was written in a response to a
> > problem which manifested on some OpenSocial containers, iterated on, and
> > implemented quickly.  OAuth 1.0 did not need to be versioned or changed in
> > any way.  Eventually, when OAuth v.NEXT comes out, maybe this extension will
> > be seen as important enough to roll into the core spec.  At that point, it
> > will be a mature proposal with several production implementations and
> > strongly defined language, making incorporation so much easier.
>
> Please do keep in mind that while this is a great idea, it's also got a a
> huge pandora's box potential.
>
> > I believe that those features (and a couple more) could be rolled up into
> > an extension proposal.  There's also another option we could discuss - to
> > make Templates an extension doc by itself.  The number of extensions to OS
> > templates currently in production seems to indicate that that feature could
> > likely benefit from iterating on its own for a while longer.
>
> That sounds great in theory, but in practice that will probably mean that
> it's not usable by gadget developers yet... I mean if orkut supports
> templating language r23534 and myspace supports templating language r01,

> which one is a gadget dev...
>
> read more »

[Evan Gilbert]

+1

We should also be moving forward on getting new features added, but we can do these as extensions that we iterate on separately from the core spec.

[Mark W.]

Evan,
One of the themes that came up at the Enterprise OpenSocial summit was
having a bit more predictability in the spec time lines. This will be
a huge help in terms of planning et. One thing I suggested is that as
we layout the time line for 1.0, we establish a high level target date
for 1.1. For example if we want to get 1.0 out the door in November,
then we could establish March/April as the target for 1.1. This way,
if you have a feature that you are working on that isn't quite ready
for the current version, you have an idea when the next opportunity
will be. Maybe this is more important this round because there won't
be new features in 1.0.

Thoughts?

-Mark W.

On Sep 16, 1:08 pm, Evan Gilbert <uid...@google.com> wrote:
> +1
> We should also be moving forward on getting new features added, but we can
> do these as extensions that we iterate on separately from the core spec.
>

> ...
>
> read more »

[goosemanjack]

A couple comments on points made so far.

RE: Protoyping

Prototyping was a huge PITA with 0.9 because the scope was just
ginormous to move from the JavaScript/REST paradigm to the declarative
server/client processed tag paradigm of DP/OSML. We had to question
and rework many assumptions about our core application design, which
greatly delayed the release. That said, now that the Shindig and
MySpace implementations both have this support in place we should be
able to iterate more quickly on prototypes. I am very strongly in
favor of keeping the prototype requirement in-place for 1.0.

RE: Arbitrary deadlines vs. letting it bake 'til it's done.

We have several commercial partners, so it's more beneficial to us to
have arbitrary deadlines at this stage than it is to simply let things
bake. If we didn't have the Google IO deadline last spring we'd
probably still be debating 0.9 while the scope continued to balloon.
I don't think November is reasonable for signoff on the 1.0 spec since
internally we're focused on getting 0.9 launched. Next spring,
however, would give us ample time to work through cleanup issues and
gather feedback from the early adopters of 0.9 to feed back into the
1.0 spec. I would propose a final draft deadline for 1.0 of early
February 2010 with a ratification deadline for 1.0 of early May 2010.

RE: Extensions

One aspect of extensions that we haven't really talked about is tag
libraries. Extensions are not likely to be sharable between Shindig
and MySpace, other than as interface contract definitions which will
need discrete implementations. Tag libraries, on the other hand,
could be sharable between platforms. We're not launching 0.9 with
external library support, though it's high on our list to bring online
after the initial launch. The current extension discussion is pretty
well locked down to the core platform implementers being able
implement extensions. Rich tag library support could be a way for
third parties to contribute to the various OpenSocial platforms. Tag
library support may be too much for 1.0, but I see a lot of value in
it, esp if others on the Shindig project are interested as well.

--
clc

> ...
>
> read more »

[Mark W.]

Re: Prototyping: I thought one of the ideas for 1.0 was that there
would be no new features. Wouldn't this mean that our prototypes would
consist mostly of the cleanup etc.? It sounds like I'm missing
something subtle in the dev process around prototypes. I'm also
strongly in favor of keeping the prototype requirement and didn't mean
to imply that I wasn't. I don't know if, from a prototype standpoint,
you are referring to the two items that you put as a caveat in your
initial post on this thread (os:Var and tag reprocessing).

RE: Deadlines vs. the e-z bake oven
I agree with the idea of putting deadlines in place and working as a
community towards those dates. I'm not wed to any specific dates for
1.0, per se, but I've got a few questions about the timeline in
general. If 1.0 has no new features, and does not go final until 2Q10,
then when will new features be able to make it into the spec? Late
2010 or early 2011? For products with traditional delivery methods,
that means at best a 2Q2011 delivery, more realistically 3Q2011. So I
don't know if I have any great suggestions here, but this is something
that we need to consider and try to balance.

RE: Extensions
I think there's a lot of very powerful "goodness" that can come out of
tag libraries, but agree it's probably too much for 1.0. I also think
we need to get the extension mechanism right, because we don't want to
have to try and fix it or worse, re-do it--especially since 1.0
implies that we've got a commitment to the APIs.


So... Where do we stand?

-Mark W.

> ...
>
> read more »

[Arne Roomann-Kurrik]

I think some of what I said has been misconstrued a bit, so I'm going to try to clarify in a shorter post:

1.) I am not in favor of removing the prototype requirement to get into the core spec.  New features should be proposed, documented in an extension document, prototyped, and implemented in production container before we consider them for spec inclusion.  This allows features to develop organically and only be considered for inclusion when they're ready.  If we only consider features which are in production for inclusion, then the spec process is not held up while we're waiting for a hacked together prototype to work. 

I also feel that this more accurately represents the way that things are currently working.  Both MySpace and orkut have custom features which they developed independently, presumably because they needed them before a new version of OpenSocial could be written.  The spec should allow for this, and have a direct path to import custom production features back into the core.

2.) I want to release the spec in regular intervals, relatively quickly.  The spec voting process should be a series of questions along the lines of "Do we feel that the os:Var implementation, as documented in <link to extension doc> and in production on MySpace is ready for spec inclusion?".  If yes, the extension doc is merged with the core spec, the extension itself is deprecated, and a new version of OpenSocial is cut with the feature.  If no, then the extension is left to mature until the next OS version and a new version of OpenSocial is cut without the feature.

~Arne

[Lane LiaBraaten]

There are 5 +1s on this proposal, so I'd like to close this thread and spawn new ones to continue the discussion.  Most of it relates to extensions, which already has a thread and a wiki page - feel free to add and comments or content here: http://wiki.opensocial.org/index.php?title=Extensions (n.b. I currently owe several updates to the current content of this page).

On the topic of timelines...the spec iteration needs timelines or we'd probably still be on v0.7 :P  The 0.9 iteration taught us that prototyping is hard to do within the confines of the spec iteration, hence the motivation for extensions to allow "baking" concepts outside the spec iteration timeline, but I digress...

Since we're agreed on the general scope, I think we can start going through the normal spec process: suggest proposals, vote on the scope, prototype, publish draft, final vote.

Here's a timeline that gets us to 1.0 just before Thanksgiving.  Basically, we'll collect proposals next week (I don't think we need long for this since we have a pretty good list already [1]).  Then spend October creating the spec patches for these proposals (this is equivalent to the prototyping phase) and finalize things in early November.

9/21 - 9/25 : List proposals (1 week)
9/28 - 10/2 : Vote on list of proposals (scope of spec changes) (1 week)
10/5 : 11/6 : Write up spec patches (5 weeks)
11/11: Publish draft (couple days)
11/11 - 11/18: Vote to accept draft (1 week)
11/20: Publish official OpenSocial v1.0 spec (couple days)

Sound good?

Legal reminder: Please submit your CLA: http://wiki.opensocial.org/index.php?title=Specification_Process#Contributions_.26_Intellectual_Property

-Lane



[1]  - Condense REST, RPC, osapi, and Data Pipelining definitions into one, "data access" section of the spec

   - Be more specific about what parts of JSP is supported by templates
   - Gadget XML reference (I'm adding this one myself :)
   - Deprecate/remove opensocial.* (gadgets.io.makeRequest vs osapi.http is related)
   - Make JSON-RPC a MUST instead of a MAY
   - UserPrefs vs AppData
   - Gadgets and OpenSocial as separate specs

   - Formalize "extensions"

-Lane

[Tim Moore]

On Sep 17, 10:40 am, goosemanjack <cc...@myspace.com> wrote:
>
> RE: Arbitrary deadlines vs. letting it bake 'til it's done.
>
> We have several commercial partners, so it's more beneficial to us to
> have arbitrary deadlines at this stage than it is to simply let things
> bake.  If we didn't have the Google IO deadline last spring we'd
> probably still be debating 0.9 while the scope continued to balloon.
> I don't think November is reasonable for signoff on the 1.0 spec since
> internally we're focused on getting 0.9 launched.  Next spring,
> however, would give us ample time to work through cleanup issues and
> gather feedback from the early adopters of 0.9 to feed back into the
> 1.0 spec.  I would propose a final draft deadline for 1.0 of early
> February 2010 with a ratification deadline for 1.0 of early May 2010.

Yeah, I would agree with this. A lot of people are still catching up
with 0.9. Packaged software needs a longer lead time than web sites.
We're just finishing up our 0.8 implementation. We can't really even
think about implementing 0.9 until Shindig 1.1 is released. I think
1.0 would benefit a lot from giving people time to work extensively
with 0.9---both container developers and app developers---so we have a
better idea where the spec needs clean up.

Feature work doesn't have to wait for 1.0 to finalize if it takes
place in extensions, so I don't see a longer cycle for 1.0 as a major
problem for products that want to ship new features next year.

-- Tim

[Lane LiaBraaten]

I think we can get a 1.0 spec out in November and be looking at 1.1 (with capability changes based on 0.9 feedback) in the 2010 Q2 timeframe.

-Lane

[Evan Gilbert]

On Thu, Sep 17, 2009 at 6:01 AM, Mark W. <weit...@us.ibm.com> wrote:

Evan,
One of the themes that came up at the Enterprise OpenSocial summit was
having a bit more predictability in the spec time lines. This will be
a huge help in terms of planning et. One thing I suggested is that as
we layout the time line for 1.0, we establish a high level target date
for 1.1. For example if we want to get 1.0 out the door in November,
then we could establish March/April as the target for 1.1. This way,
if you have a feature that you are working on that isn't quite ready
for the current version, you have an idea when the next opportunity
will be. Maybe this is more important this round because there won't
be new features in 1.0.

Thoughts?

(sorry for the delayed response, I was having a problem posting)

A like the idea, few thoughts on nuts & bolts:

People should definitely have a feature ready by the date, but it's hard putting a schedule on consensus. think I'd like a hard deadline on feature submission and a softer deadline on discussions.

Should this cycle apply to extensions? There is some benefit in letting extensions have their own cycles (trying to line up 20 different smaller specs will be harder), and I'd imagine that different parts of the spec have different iteration cycles. More mature extensions may need fewer changes and can iterate to every 6-12 months, experimental features may need uprevs every 2 months

[goosemanjack]

Please allow for more than just one week for proposing 1.0 spec
changes. I'm currently out of the country and have extremely limited
access and motivation for contributing until early October.
--
clc


On Sep 26, 11:10 am, Evan Gilbert <uid...@google.com> wrote:

> ...
>
> read more »- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -

[Lane LiaBraaten]

Okay - let's keep "open for proposals" status until the end of rest of this week.

-Lane

[Randy Hudson]

Why not call the next version 0.10?

[Lev Epshteyn]

Isn't 0.10 less than 0.9? :)

[Lane LiaBraaten]

Not when you're versioning software - and since we've modeled spec development after software development, I certainly understand the 0.9 --> 0.10 progression.

By the same token, 0.10 would indicate an incremental change.  However, many of the proposals suggest fundamental changes (redefining compliance, formalizing extensions, etc.), hence the 1.0.

-Lane

On Oct 1, 2009 1:48 PM, "Lev Epshteyn" <le...@google.com> wrote:

Isn't 0.10 less than 0.9? :)

On Thu, Oct 1, 2009 at 3:41 PM, Randy Hudson <randy....@gmail.com> wrote: > > > Why not call th...

[Chris Chabot]

By the same token 'osapi', 'templating', 'data-pipeling' were pretty fundamental changes too, really it were 2 completely new ways of developing opensocial apps replacing the old ways.

So by that logic 0.9 should've been 1.0 already?

[Lev Epshteyn]

I think the changes we're currently discussing for v.Next-whatever-its-called are ones that move it to be a mature baked standard. Specifically, minimizing new features and maximizing "fixing" the spec where its broken or fuzzy - these seem like the very things that one does when flipping the important psychological bit from 0.9 to 1.0.

To me, this seems like a very fundamental change, much more so than the delta between 0.8 and 0.9.

[Mark W.]

I think a 1.0 also implies a degree of support and commitment to the
APIs. For example, I would not expect API "breakage" between 1.0 and
1.1, 1.2, 1.n. I agree with Lev, getting to 1.0, in part because of
this commitment, seems more significant than going from 0.8 to 0.9,
regardless of the "wad-o-stuff" that got in there.

-Mark W.

On Oct 1, 6:52 pm, Lev Epshteyn <le...@google.com> wrote:
> I think the changes we're currently discussing for
> v.Next-whatever-its-called are ones that move it to be a mature baked
> standard. Specifically, minimizing new features and maximizing "fixing" the
> spec where its broken or fuzzy - these seem like the very things that one
> does when flipping the important psychological bit from 0.9 to 1.0.
>
> To me, this seems like a very fundamental change, much more so than the
> delta between 0.8 and 0.9.
>

> On Thu, Oct 1, 2009 at 6:43 PM, Chris Chabot <chab...@google.com> wrote:
> > By the same token 'osapi', 'templating', 'data-pipeling' were pretty
> > fundamental changes too, really it were 2 completely new ways of developing
> > opensocial apps replacing the old ways.
>
> > So by that logic 0.9 should've been 1.0 already?
>

> > On Thu, Oct 1, 2009 at 10:57 PM, Lane LiaBraaten <lliab...@google.com>wrote:
>
> >> Not when you're versioning software - and since we've modeled spec
> >> development after software development, I certainly understand the 0.9 -->
> >> 0.10 progression.
>
> >> By the same token, 0.10 would indicate an incremental change.  However,
> >> many of the proposals suggest fundamental changes (redefining compliance,
> >> formalizing extensions, etc.), hence the 1.0.
>
> >> -Lane
>
> >> On Oct 1, 2009 1:48 PM, "Lev Epshteyn" <le...@google.com> wrote:
>
> >> Isn't 0.10 less than 0.9? :)
>

> >> On Thu, Oct 1, 2009 at 3:41 PM, Randy Hudson <randy.hud...@gmail.com>