Realigning around the shift url [challenge]

[Mushon Zer-Aviv]

So...

As hinted in my previous mail, over the last (many) years my perspective on ShiftSpace has… ahm… shifted. When we started and for many years after what led me in thinking about a metaweb platform is that there's a lot of heavy lifting involved in metaweb development and what we need to do is to solve most of that heavy lifting and remove the hurdles away from the app (previously known as "space") developers way. These hurdles included hosting, discovery, sharing, development standards, user management, permissions, UI, the paste to connect a shift to a page, proxying/unique url, plugin architecture, installation, a distributed data scheme... (I could continue) a very long list of features that are all indeed based on real user requirements but have turned § into a huge behemoth that is hard to develop in a small team and hard to hack on without subscribing to the whole stack as a given. In my view this has been our main point of failure and it has mounted tons of work that required more and more advanced skills.

My proposed thought experiment suggests reviewing all of these features and distilling the core functionality required for the metaweb. In my opinion, it is the metaweb paste+the shift url.

The metaweb paste = the magical glue that connects content+controller+context

The shift url = the package to refer to, to share, to process.

I would like to come up with a way to have a url that represents a shift as the core of the ShiftSpace platform and have any other feature be built on top of it. Not unlike how the <140chars is the core of Twitter and can be processed by any client or system. I would like this core unit to also take a distributed system into account and have as little as possible points of failure.

The shift url should contain:

[§ protocol] + [§ host namespace] + [context uri] + [x-path to pin on page] + [controller (app) uri] + [shift data/pointer]

Just as an illustration, one way of implementing this could look like:

http://s.s.    shiftspace.org/    nytimes.com/   #p[EskTut],h[EskTut]+   api.markkit.com+   bit.ly/iWw3dfh87

A client running the plugin would know to interpret an s.s.domain.com as a potential § server and to ignore the namespace and serve the shift in its original context while a client not running the plugin would use the proxy at http://s.s.shiftspace.org/ to interpret the shift.

At none of these points is a centralized database required, all of the shift interpretation can happen in the front end. This is going on a protocol approach to managing shifts. Once we have that as a makeshift standard (pun unintended believe it or not) we can build endless ways to interpret that add more features, integrate external services (oEmbed, Hashify, Telehash, whatever...), have endless ways of building apps and endless ways of running proxies and what not...

That was my pitch, but it'a actually not only my pitch, I am not just here to describe things theoretically. I have been drinking a substantial amount of beer (the official § refreshment) with Zohar Arad, a crazy Israeli hacker who has collaborated with us in the past working on some front-end code for §.org. I would like to re-introduce Zohar to the list and to ask him to please make a bit more practical sense out of the ideas I have been floating around here (including correcting mistakes I might have made in trying to explain this). Zohar also has an idea for a stack of software he would like to use for this proof of concept and it would be great if you could give us some feedback on both the idea and the tech schema.

Take it away Zohar...

--

Mushon Zer-Aviv
Shual.com - design studio
§  ShiftSpace.org - an opensource layer above any website
¶  Mushon.com - blog
× @mushon - Tweet me

[Dan Phiffer]

Wow, it's amazing that I'm awake with these emails coming in. I raise my virtual glass to you sir. I am sober here but rather sleepy, which may simulate some of the pleasant effects of our official beverage.

BTW, I don't think this has come up on the list already, but Mushon is going to be a dad! You guys should look at Galia's impressive belly on Flickr if you haven't already... Mozel tov! http://www.flickr.com/photos/galiaoffri/

So, I had a lot of enthusiasm about hacking on ShiftSpace back in March. But then I went to SXSW and then I was super busy catching up with work stuff. I like this thread idea and I'd like to start from thinking about ShiftSpace as a social context before thinking about its technology. For me it's a potential means of connecting with other people about media. For example, I'm still kind of murky on this new Fatah/Hamas deal. Aside from a handful of tweets, my knowledge is limited to what I heard on Democracy Now or NPR or whatever. What are some good articles I should read? How are other people thinking about it? That is my itch. All of my existing tools for social/critical media discovery/creation feel unsuited to the variety of ways I use the web.

I think Mushon is right about the importance of the "metaweb paste," that's my biggest point of frustration with the current approach. I had a conversation with Joe recently about using ShiftSpace for "serious" scholarship and he pointed out how untrustworthy our software must feel to somebody like a PhD candidate, somebody staking their career on information being retrievable. Imagine that user at a key moment when they're trying to synthesize an idea based on a ShiftSpace annotation -- what are the chances the shift will still work? When I shift a page today, I have very little faith that I'll be able to retrieve it tomorrow. Or that it will even work if I reload the page a moment later. Solving this problem is probably the biggest priority for building a compelling piece of software. I would like to use ShiftSpace on a daily basis and recommend it to my friends and acquaintances.

I know there's room for improvement in the range coder, but our current approach is fundamentally limited by the "changing page" problem. I don't want ShiftSpace content to be limited to a handful of old-school sites that don't have the budget for a modern redesign. Some alternative approaches: cache the full page source code and store it alongside the shift. Or save a full page screenshot of the shift. Maybe both? Maybe offer them as an option? Some shifts are more important to retain than others. But it's often hard to know, when you're creating something, the degree to which you'll need to retrieve it down the line. I would like every shift to be automatically and reliably archived in some form.

Maybe that archiving function is part of some proxy-based design? Personally I like the trend of some newer start-ups to make all URLs short URLs. I like that sites like vhx.tv and mlkshk.com are more sharable on sites like Twitter without relying on a third-party service.

Also, nice photo on TechCrunch! And I'm curious what you guys think about the Knight-Mozilla partnership...
http://blog.mozilla.com/blog/2011/02/07/knight-mozilla-news-technology-partnership-announced/

Have you thought about resubmitting your proposal from last time?

-Dan

On Apr 29, 2011, at 2:50 AM, Mushon Zer-Aviv wrote:

> Hi fella§,
> The last wave of discussions on the dev list proved:
> • We're still on top of the latest social/web trends
> • These trends and possibilities have changed tremendously over the last few years
> • We seem to still enjoy ShiftSpace as an innovative web playground
> When Dan and I started ShiftSpace back at the end of 2005 Javascript wasn't even popular and browsers couldn't do much. We thought of ShiftSpace as a metaweb platform focused on discoverability—click [Shift]+[Space] to see what's on this page. This leading concept has lead the path for ShiftSpace and has focused much of our work on managing and delivering shifts. The more involved and sophisticated our code become, the more requirements were mounted on anyone wanting to play the game in the ShiftSpace way.
>
> I don't think these were indeed mistakes but I think the past 6 month break was a good opportunity to think things over and rethink what is the itch we're trying to scratch by doing ShiftSpace, and if the itch has changed how can we rethink the way we scratch it?
>
> Rather than continuing previous fringe-tech threads I would like to challenge all of us (lurkers included) to propose new ideas for how would they build § knowing what we know today. The format would be to propose ideas and to follow up with fresh code (not necessarily built on a fork of the § repo). You can either answer to this thread or better—start a new one to focus the discussion around your proposal (maybe include the [challenge] brackets in there too).
>
> The goal is not necessarily to have the team decide what *we* should do, but rather throw a lot of §paghetti at the wall and later see what sticks and what we and others gravitate towards.
> Lastly, for those who haven't noticed, Techcrunch has chose the § table as the leading image for their 2011 Disrupt Hackathon. That's how geeky we are! We have the word hackathon spelled big over our foreheads… :)
>
> Let the ideas flourish and let the hacks begin!
> --
> Mushon Zer-Aviv
> <shual_gray.gif> Shual.com - design studio

> § ShiftSpace.org - an opensource layer above any website
> ¶ Mushon.com - blog
> × @mushon - Tweet me
>
>

> --
> You received this message because you are subscribed to the Google Groups "ShiftSpace-dev" group.
> To post to this group, send email to ShiftSp...@googlegroups.com.
> To unsubscribe from this group, send email to ShiftSpace-de...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ShiftSpace-dev?hl=en.

> <shual_gray.gif> Shual.com - design studio

> § ShiftSpace.org - an opensource layer above any website
> ¶ Mushon.com - blog
> × @mushon - Tweet me
>
>

> --
> You received this message because you are subscribed to the Google Groups "ShiftSpace-dev" group.
> To post to this group, send email to ShiftSp...@googlegroups.com.
> To unsubscribe from this group, send email to ShiftSpace-de...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ShiftSpace-dev?hl=en.

[David Nolen]

On Fri, Apr 29, 2011 at 3:38 AM, Mushon Zer-Aviv <mus...@shual.com> wrote:

The shift url should contain:

[§ protocol] + [§ host namespace] + [context uri] + [x-path to pin on page] + [controller (app) uri] + [shift data/pointer]

Just as an illustration, one way of implementing this could look like:

http://s.s.    shiftspace.org/    nytimes.com/   #p[EskTut],h[EskTut]+   api.markkit.com+   bit.ly/iWw3dfh87

I totally encourage people going down this path, however I think mucking around with URLs (which are already data-infested) like this means that organizations/developers will likely never adopt it. As a developer I would never use such a system or encourage people to use such a system.

So you won't see me express any interest in this direction at all.

David 

[David Nolen]

On Fri, Apr 29, 2011 at 4:54 AM, Dan Phiffer <d...@phiffer.org> wrote:

I think Mushon is right about the importance of the "metaweb paste," that's my biggest point of frustration with the current approach. I had a conversation with Joe recently about using ShiftSpace for "serious" scholarship and he pointed out how untrustworthy our software must feel to somebody like a PhD candidate, somebody staking their career on information being retrievable. Imagine that user at a key moment when they're trying to synthesize an idea based on a ShiftSpace annotation -- what are the chances the shift will still work? When I shift a page today, I have very little faith that I'll be able to retrieve it tomorrow. Or that it will even work if I reload the page a moment later. Solving this problem is probably the biggest priority for building a compelling piece of software. I would like to use ShiftSpace on a daily basis and recommend it to my friends and acquaintances.

I know there's room for improvement in the range coder, but our current approach is fundamentally limited by the "changing page" problem. I don't want ShiftSpace content to be limited to a handful of old-school sites that don't have the budget for a modern redesign. Some alternative approaches: cache the full page source code and store it alongside the shift. Or save a full page screenshot of the shift. Maybe both? Maybe offer them as an option? Some shifts are more important to retain than others. But it's often hard to know, when you're creating something, the degree to which you'll need to retrieve it down the line. I would like every shift to be automatically and reliably archived in some form.

ShiftSpace was always biased towards the social side of things even though it has really amazing implications for the single user use case. If ShiftSpace always ran locally most of what is described be trivial. With decentralizing, snapshots of the web could be cached across users machines w/o all the issues around scalability that would be incurred in a centralized scenario.

David 

[Mushon Zer-Aviv]

David,

Thanks for showing your encouragement in your own special way ;)

Do you understand the problem that this hacky solution is trying to solve?

We can have a non-semantic url instead (like shiftspace.org/bdefu347) which will require a standardized database, but that adds a point of failure and requires every ShiftSpace implementation to follow our database schema. What I'm suggesting does not require a centralized § server and allows to activate this url on any domain once you change the [§ host namespace]. Moreover, I think we need to suggest a more open way of developing for §, our way or the highway has not worked for us so far and is something I am really trying to avoid.

I am not a webapp developer, but I really don't understand if Google can use a url like this:

http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=274+Morgan+Ave,+Brooklyn,+NY+11211,+USA&aq=1&sll=32.059882,34.785118&sspn=0.080742,0.16119&ie=UTF8&hq=&hnear=274+Morgan+Ave,+Brooklyn,+Kings,+New+York+11211&t=h&z=16&iwloc=A

...why can't we? Please enlighten me.

Basically, if Google, Bing Maps, Yahoo and OpenStreetMap would've standardized their url structure we could have used them interchangeably. In § case this is even more important as if we set the url structure as the standard we can basically have any § implementation work with any other one seamlessly.

I do intend to explore this direction and you can obviously choose to play along or not, but as you know I really value your perspective and I would really like to understand your arguments against this approach and your alternatives to it.

On another note:

Zohar has been hacking through the weekend and has hit some walls with the approach to the proxy, he would be posting here about his experience and questions. I hope some of you who were involved with proxying can help shed some insights or code references on this far-from-obvious challenge.

See you later in the IRC channel,

cheers,

Mushon Zer-Aviv
Shual.com - design studio
§  ShiftSpace.org - an opensource layer above any website
¶  Mushon.com - blog
× @mushon - Tweet me

--

[David Nolen]

On May 1, 2011, at 3:38 AM, Mushon Zer-Aviv wrote:

David,

Thanks for showing your encouragement in your own special way ;)

Do you understand the problem that this hacky solution is trying to solve?

We can have a non-semantic url instead (like shiftspace.org/bdefu347) which will require a standardized database, but that adds a point of failure and requires every ShiftSpace implementation to follow our database schema. What I'm suggesting does not require a centralized § server and allows to activate this url on any domain once you change the [§ host namespace]. Moreover, I think we need to suggest a more open way of developing for §, our way or the highway has not worked for us so far and is something I am really trying to avoid.

I don't see how your proposal has anything to do with whether people will have to implement our database schema. People can do whatever they want as long as they follow the HTTP based API, this is  true today.

Basically, if Google, Bing Maps, Yahoo and OpenStreetMap would've standardized their url structure we could have used them interchangeably. In § case this is even more important as if we set the url structure as the standard we can basically have any § implementation work with any other one seamlessly.

Many websites change the hash component of the url with JavaScript which is the only place our data could really live w/o a proxy. Proxies present their own issues for user experience, security and scalability.

I've already outlined in past emails how a proxy might be done properly using postMessage.

David

[Mushon Zer-Aviv]

On 5/1/11 7:47 PM, David Nolen wrote:

I've already outlined in past emails how a proxy might be done properly using postMessage.

This is David 11 months ago (more in that thread and in following ones):

       Weekly Update 6/1/10: The Peer-to-Peer Web Cometh!

http://hacks.mozilla.org/2010/06/beyond-html5-database-apis-and-the-r...

A great read and an exciting direction for the web. I was dreading
having to distribute our complicated server setup to clients in order
to make ShiftSpace P2P. We have an evolving plan now,

ShiftSpace P2P =  window.applicationCache + IndexedDB + HTML5 workers
+ HTML postMessage

All other technologies need not apply :)

:D

:D

:D

Sorry, but I'm a bit excited.

Did anything happened that chilled your excitement, or did you simply end up getting more excited about other stuff?

[David Nolen]

Haha.

It's a ton of work that I'd rather not do by myself and I'm skeptical about the scalability of serving proxied pages - someone needs to run some benchmarks on the idea.

I'm more excited about a real decentralization strategy.

David

[Mushon Zer-Aviv]

One person's scalability challenge is person's business model.

Imagine having each shift and its page cached for X hours after being created, maybe getting the countdown zeroed every time it is viewed. But then if you want to keep all of your shifts stored, you pay a fee. It can be a model that would support a §-based service and would justify maintaining AWS costs or something.

Just thinking out loud here (I have a loud keyboard)

Mushon Zer-Aviv
Shual.com - design studio
§  ShiftSpace.org - an opensource layer above any website
¶  Mushon.com - blog
× @mushon - Tweet me

[David Nolen]

On May 1, 2011, at 1:44 PM, Mushon Zer-Aviv wrote:

One person's scalability challenge is person's business model.

Imagine having each shift and its page cached for X hours after being created, maybe getting the countdown zeroed every time it is viewed. But then if you want to keep all of your shifts stored, you pay a fee. It can be a model that would support a §-based service and would justify maintaining AWS costs or something.

Exactly. I would do the work given the time and money. As I don't see any time or money forthcoming, I'm only going to tackle the problems I care about solving in my own free time.

David