[proposal for UJ/0.2] Extensions and capabilities

[Dmitry Tantsur]

Hi and happy new year!=)

I'd like to discuss some ways for servers to provide additional
functionality.
I suggest to add one more command to UJ/0.2:

CAPABILITIES:
This command is used to get information about additional functionality
provided by storage node. POST fields:
"protocol": "UJ/0.2"
"method": "CAPABILITIES"
Possible responses:
200 OK + a JSON string that decodes to exactly the following
fields:
"value": list of strings which describe additional node
capabilities (see below)
400 Bad Request (if what you posted was unparsable. preferably add
a plain text explanation in the response body)
503 Service Unavailable (if the server is too busy)

Each capability should be in form: "command:property/X.Y" (each part
being optional).
1st part allows to add new commands. For compatibility purpose any
extension command should begin with some special symbol (say "_").
2nd part allows to add new parameters to existing commands. For
compatibility purpose any extension parameter should begin with some
special symbol (say "_"). Without extension parameters command should
behave as stated in the protocol.
3rd part is an optional extension version.

Briefly:
- Should be possible to add server-specific commands in a standard way
- Should be possible to add server-specific parameters to commands in
a standard way
- Extensions should not affect clients that are not aware of them
- If possible, extensions should be standardized.

Examples:
- capabilities ['_FOO'] - server supports "_FOO" command
- capabilities ['GET:_bar'] - server supports "_bar" parameter for
"GET" command
- capabilities ['_FOO/0.1', 'GET:_bar/1.0'] - both with versions
- capabilities [':_bar'] - server supports "_bar" parameter for any
command

Regards,
Dmitry.

[Daniel Gröber]

I think having a way to extend the protocol while retaining compatibility with
clients that do not support this extension is indeed very important. I am
thinking file upload, video encoding and such. We cannon force servers to
implement these things, but for certain applications they are necessary and if
we don't give the app devs a way to do this stuff while complying with the
protocol they would probably just use a centralized service for this or not use
unhosted at all.

As for the proposal itself: The idea is very good, but i think we should do this
in a more HTTP'isch way.

Quote from RFC 2616:
> The OPTIONS method represents a request for information about the
> communication options available on the request/response chain
> identified by the Request-URI. This method allows the client to
> determine the options and/or requirements associated with a resource,
> or the capabilities of a server, without implying a resource action
> or initiating a resource retrieval.

So i think instead of making this a command we should use the HTTP option
method. We need a way for the client to find the exact POST URI for Unhosted
anyways so we can use this to solve two problems.

Daniel

[Dmitry Tantsur]

That's good, but is this method common? Is it supported by e.g. Apache
+PHP (our reference implementation)?

On 1 янв, 22:55, Daniel Gröber <darkbo...@gmail.com> wrote:
> I think having a way to extend the protocol while retaining compatibility with
> clients that do not support this extension is indeed very important. I am
> thinking file upload, video encoding and such. We cannon force servers to
> implement these things, but for certain applications they are necessary and if
> we don't give the app devs a way to do this stuff while complying with the
> protocol they would probably just use a centralized service for this or not use
> unhosted at all.
>
> As for the proposal itself: The idea is very good, but i think we should do this
> in a more HTTP'isch way.
>
> Quote from RFC 2616:
>
> > The OPTIONS method represents a request for information about the
> > communication options available on the request/response chain
> > identified by the Request-URI. This method allows the client to
> > determine the options and/or requirements associated with a resource,
> > or the capabilities of a server, without implying a resource action
> > or initiating a resource retrieval.
>
> So i think instead of making this a command we should use the HTTP option
> method. We need a way for the client to find the exact POST URI for Unhosted
> anyways so we can use this to solve two problems.
>
> Daniel
>

[Daniel Gröber]

Well, it is in the HTTP standard so it better be supported ;)

But the bigger problem is if browsers support OPTIONS with AJAX (as far as I know most older versions of M$IE don't). For these cases we might have to add a compatability command to the regular POST api that accepts the same parameters as the OPTIONS version.

[Dmitry Tantsur]

Thus, maybe introduce a POST command now and research OPTIONS support
for 0.3?

On 2 янв, 02:20, Daniel Gröber <darkbo...@gmail.com> wrote:
> Well, it is in the HTTP standard so it better be supported ;)
>
> But the bigger problem is if browsers support OPTIONS with AJAX (as far as I
> know most older versions of M$IE don't). For these cases we might have to
> add a compatability command to the regular POST api that accepts the same
> parameters as the OPTIONS version.
>

[Daniel Gröber]

Sound good. I will include this in my rewrite of the 0.2 specification :)

[Michiel de Jong]

When you open up Firebug, you see that Firefox is already using
OPTIONS to query the CORS headers.

i was always against making the protocol extensible, but maybe you
guys are right.

however, wouldn't it be easier if people would indicate the
capabilities needed in the version number of the protocol? so if you
add file uploads, or whatever, you name it UJ/0.2.1 or
UJ/0.2.<projectName>. otherwise, we might as well define only the
OPTIONS command, and then everybody can do whatever they feel like,
and still call it 'the same protocol'. but it would be many different
protocols, wouldn't it? and then if someone wants to play Exploding
Cats, it will always say 'your unhosted storage node doesn't support
the Cat.Explode command, please migrate to our node which is much
better'. do you see the problem?

i don't know. how do other protocols do this? i'll think through the
pros and cons some more.

[Michiel de Jong]

Hi,

I'm still not very convinced that this is a good idea.

On Sat, Jan 1, 2011 at 11:41 PM, Dmitry Tantsur <divius...@gmail.com> wrote:

I'd like to discuss some ways for servers to provide additional
functionality.

Did you have anything specific in mind? Let's think about the consequences.

If your server provides additional functionality, then either apps start relying on this for some of their features, or they don't.

This means we are creating a distinction between 'haves' and 'havenots' for the end-users of this app.

this reminds me of what internet explorer did. it started introducing special extra features. some websites started using these, meaning that little by little the web became less and less compatible with netscape.

isn't that the opposite of what we want?

i can see how maybe we want to allow production servers to support experimental features that are not part of the mainstream protocol yet. but as soon as these become useful to people, they should be incorporated in the next version of the protocol as soon as possible.

if you have a feature that you want to offer, then it should become part of the next version of the protocol, i don't care if there are some nodes that over version 2 of the protocol and others offer version 9. as long as this is a linear path, and not a tree with many branches.

otherwise, sooner or later, apps will be divided into 'explorer-only' and 'netscape-only'.

maybe we should do something like debian's distinction between 'stable' and 'unstable'? then in unstable you can extend whatever you want, and experiment with it. as soon as one of these suggestions are seen as useful, it can go into stable (but 'stable' would by definition never be extensible itself).

[pir...@gmail.com]

It's true that it's a problem if someone does extensions incompatibles
with others in a embrace-extense-extins way, but in any case it's
useful. I have found developing the access control system that
Javascript is more powerful that i thought with a little of ingenious,
but there's a lot of things that can be optimized with server side
computing. I think that a good option at least at this moment is not
think too much about it and just see what extensions/commands are
necesary when we get v0.9, and then think about if the extension
system has had been necesary or not, and if so, develop it in a way
where apps can check for expecific functions like javascript
done-in-the-good-way or OpenGL extensions work.

By the way, i have read at Wikipedia
(http://en.wikipedia.org/wiki/XMLHttpRequest#The_open_method) that the
GET, POST, HEAD, PUT, DELETE and OPTIONS methods are supported in the
XMLHttpRequest, and that it's left for the browser if it support any
other methods but there's nothing that forbid it, so if we left away
the thing about old versions of Explorer, i think that the point about
REST and OPTIONS is awailable, at least for v0.3.

Oh, another thing: in HTTPS the headers are also encripted, so we
could put the authentification there and we could get a RESTful
interface without problems and also we could get a end-to-end
encripted communication and storage system... :-)

2011/1/2 Michiel de Jong <mic...@unhosted.org>:

--
"Si quieres viajar alrededor del mundo y ser invitado a hablar en un
monton de sitios diferentes, simplemente escribe un sistema operativo
Unix."
– Linus Tordvals, creador del sistema operativo Linux

[Daniel Gröber]

From: Michiel de Jong <mic...@unhosted.org>
Subject: Re: [unhosted] [proposal for UJ/0.2] Extensions and capabilities
Date: Sun, 2 Jan 2011 10:20:40 +0800

> Hi,
>
> I'm still not very convinced that this is a good idea.
>
> On Sat, Jan 1, 2011 at 11:41 PM, Dmitry Tantsur <divius...@gmail.com>wrote:
>
>> I'd like to discuss some ways for servers to provide additional
>> functionality.
>>
>
> Did you have anything specific in mind? Let's think about the consequences.
>

Just think of a unhosted YouTube like app. The unhosted storage nodes for use
with this app would have to support video trans-coding for this to work. Now
obviously you cant just make video encoding part of the protocol as it is very
complex and resource intensive. So you'd need a way to see if a storage node
supports trans-coding of video uploads.

> If your server provides additional functionality, then either apps start
> relying on this for some of their features, or they don't.
>
> This means we are creating a distinction between 'haves' and 'havenots' for
> the end-users of this app.
>

Well there is simply no way around this. People wanting to do slightly more
complicated apps the likes of social networks, multimedia publishing, what have
you will need a way to access special processing power that only certain nodes
provide.

It might be possible to separate the stuff unhosted already does from special
processing of data (especially as soon as browsers support native extensions
(that are seamlessly installed [see http://code.google.com/p/nativeclient/]) and
most of this stuff can be done in the browser). But right now app devs are most
likely to once again use proprietary protocols to talk to their own servers.

> this reminds me of what internet explorer did. it started introducing
> special extra features. some websites started using these, meaning that
> little by little the web became less and less compatible with netscape.
>

Well then lets hope browsers adopt native client as soon as possible as I just
don't see a way around this if they don't.

> isn't that the opposite of what we want?
>

True.

> i can see how maybe we want to allow production servers to support
> experimental features that are not part of the mainstream protocol yet. but
> as soon as these become useful to people, they should be incorporated in the
> next version of the protocol as soon as possible.
>
> if you have a feature that you want to offer, then it should become part of
> the next version of the protocol, i don't care if there are some nodes that
> over version 2 of the protocol and others offer version 9. as long as this
> is a linear path, and not a tree with many branches.
>
> otherwise, sooner or later, apps will be divided into 'explorer-only' and
> 'netscape-only'.
>
> maybe we should do something like debian's distinction between 'stable' and
> 'unstable'? then in unstable you can extend whatever you want, and
> experiment with it. as soon as one of these suggestions are seen as useful,
> it can go into stable (but 'stable' would by definition never be extensible
> itself).

That probably won't work as people are just going to add stuff to the protocol
the way they want anyways. All we can do is offer a framework to make this
possible without breaking everything.

[Ravi Pinjala]

Here's the thing: if somebody wants to create an incompatible
protocol, they could just as easily do that by ignoring Unhosted
completely. If somebody writes an Unhosted app that depends on a
feature that only they provide, they're not any better off than if
they had made up their own protocol, and if somebody offers a server
that has useful new features, app authors won't adopt them unless
they're widely implemented.

Internet Explorer was a special case, because Microsoft figured out a
way out of the chicken-and-egg situation by distributing the browser
with Windows (so their new features were widely-implemented enough for
website authors to start using them). The best protection against that
is to have several independent implementations, and it seems like
Unhosted is already going down that path. :D

I've been working on a similar project for a few months (an extensible
REST storage protocol, http://bitbucket.org/pstatic/webfs - I'm
planning to implement UJ in it once the spec settles down a bit) so
I'm biased a bit toward extensibility. ;) One thing you might consider
is defining a base protocol that lets clients discover what extensions
are available, and rolling up useful extensions into future versions
of the protocol as required extensions, to keep things from becoming
too fragmented.

--Ravi

[Michiel de Jong]

On Sun, Jan 2, 2011 at 10:28 AM, pir...@gmail.com <pir...@gmail.com> wrote:

It's true that it's a problem if someone does extensions incompatibles
with others in a embrace-extense-extins way, but in any case it's
useful.

oh yes, i know it's very useful for the developer to change the protocol to fit his application. that's exactly the danger. it is too useful, and too tempting.

 

I have found developing the access control system that
Javascript is more powerful that i thought with a little of ingenious,
but there's a lot of things that can be optimized with server side
computing.

that's exactly what we don't want. 

 

I think that a good option at least at this moment is not
think too much about it

very bad option i would say. :) now is the moment to think about it.

 

and just see what extensions/commands are
necesary when we get v0.9,

yes, i proposed to do that in 'unstable' versions of the protocol, where experimental features are allowed. but still keeping a 'stable' version that everybody should always implement.

 

and then think about if the extension
system has had been necesary or not,

with necessary you probably mean that developers will want to use it. i don't doubt they will. my question is, should we let them, or should we try to impose some unity.

 

and if so, develop it in a way
where apps can check for expecific functions like javascript
done-in-the-good-way

? - you mean javascript 'the good parts'? i don't see the relation between that and the protocol. for clarity, the protocol defines the communication with the unhosted storage node. it doesn't define how you implement your application to work with it. you could even write a desktop client-app without using javascript, and still use the protocol.

 

or OpenGL extensions work.

? - same question - maybe you mean WebGL? but still then i don't understand what that has to do with the protocol for the unhosted storage

 

anyway, the fact that people see so many uses for extending the protocol is, to me, a reason not to make it extensible rather than the other way around.

[Michiel de Jong]

On Sun, Jan 2, 2011 at 10:45 AM, Ravi Pinjala <pst...@gmail.com> wrote:

Here's the thing: if somebody wants to create an incompatible
protocol, they could just as easily do that by ignoring Unhosted
completely. If somebody writes an Unhosted app that depends on a
feature that only they provide, they're not any better off than if
they had made up their own protocol, and if somebody offers a server
that has useful new features, app authors won't adopt them unless
they're widely implemented.

exactly! thank you. :) so if people want to do stuff that's not in the protocol, let them do it outside the protocol. as long as they don't "fork" our protocol and splinter it.

 

I've been working on a similar project for a few months (an extensible
REST storage protocol, http://bitbucket.org/pstatic/webfs - I'm
planning to implement UJ in it once the spec settles down a bit) so
I'm biased a bit toward extensibility. ;)

great project! yes, unhosted is really a cloud-storage protocol, but aimed at per-user cloud storage instead of per-website cloud storage.

so... could unhosted be one of the interfaces that can be discovered through webfs? or did i not understand that correctly? maybe that belongs in a separate thread, or we can chat about it in #unhosted on irc.freenode.net if you're around.

 

One thing you might consider
is defining a base protocol that lets clients discover what extensions
are available, and rolling up useful extensions into future versions
of the protocol as required extensions, to keep things from becoming
too fragmented.

yes, that is more or less what i'm proposing. with the only difference that in think we should explicitly any extensions as experimental. so that no apps can count on storage nodes having these extensions installed.

if an app uses anything but the latest 'stable' version of the protocol, then that app should be labeled (for instance by app channels) as 'experimental', or the specific feature inside the app should be labeled as an 'experimental feature'. that way people can showcase stuff they're developing and we can vote for which things we include and which ones we don't.

i'm in #unhosted on irc.freenode.net if anybody want to talk about this on chat and try me with convincing arguments. :)

[Dmitry Tantsur]

On 2 янв, 04:14, Michiel de Jong <mich...@unhosted.org> wrote:
> When you open up Firebug, you see that Firefox is already using
> OPTIONS to query the CORS headers.
>
> i was always against making the protocol extensible, but maybe you
> guys are right.
>
> however, wouldn't it be easier if people would indicate the
> capabilities needed in the version number of the protocol? so if you
> add file uploads, or whatever, you name it UJ/0.2.1 or
> UJ/0.2.<projectName>. otherwise, we might as well define only the
> OPTIONS command, and then everybody can do whatever they feel like,
> and still call it 'the same protocol'. but it would be many different
> protocols, wouldn't it? and then if someone wants to play Exploding
> Cats, it will always say 'your unhosted storage node doesn't support
> the Cat.Explode command, please migrate to our node which is much
> better'. do you see the problem?

And what? And their 0.2.1 will be incompatible with somebody other's
0.2.1? And with mainstream 0.2 too?
Reality is simple: user want to extend protocols they use. And the
more extensible the fewer of them will invent new incompatible
protocols.
The idea is that UJ/0.2 with extensions is still UJ/0.2. One can use
additional features, one can ignore them. This is not a different
protocol.

If you app use UJ/0.2.1 it will be rejected by any node without this
extension (e.g. by your node).
And maybe some extensions are not that critical for the app.

Examples:
1. A node can support user-uploaded videos. Applications can use it
but also can easily go without.
2. A node supports special value-fetching command (e.g. SQL-like),
Applications can use it to speed up, but can work with standard "GET"
command.

I'm strictly against creating new versions of a protocol for every
small feature. Otherwise we will just end up having lots of different
0.2.1 nodes and a few real 0.2 with all of them being incompatible.

[pir...@gmail.com]

>> and just see what extensions/commands are
>> necesary when we get v0.9,
>
> yes, i proposed to do that in 'unstable' versions of the protocol, where
> experimental features are allowed. but still keeping a 'stable' version that
> everybody should always implement.

And being always be able to fall-back to the (latest) stable version
of the protocol? I agree with that :-)

>> and then think about if the extension
>> system has had been necesary or not,
>
> with necessary you probably mean that developers will want to use it. i
> don't doubt they will. my question is, should we let them, or should we try
> to impose some unity.
>
>>
>> and if so, develop it in a way
>> where apps can check for expecific functions like javascript
>> done-in-the-good-way
>
> ? - you mean javascript 'the good parts'? i don't see the relation between
> that and the protocol. for clarity, the protocol defines the communication
> with the unhosted storage node. it doesn't define how you implement your
> application to work with it. you could even write a desktop client-app
> without using javascript, and still use the protocol.
>
>>
>> or OpenGL extensions work.
>
> ? - same question - maybe you mean WebGL? but still then i don't understand
> what that has to do with the protocol for the unhosted storage

No, i was talking about programming styles: if you are a good
javascript on OpenGL programmer, before using a OpenGL extension or
browser-dependent javascript feature you first politely ask for it,
and if it doesn't exist you are able to fall back to something more
estandar (pure OpenGL functions or try to another browser-dependent
javascript feature) or notify the user that you can't achieve this
instead of simply crash. This is the way i have designed the access
control system, so before you access the data you can check for it,
another thing is that the programmers are idiots and use the API in a
bad way accessing the data directly without doing a little of safe
checking...

In any case i like the stable-unstable thing, only that this way we
need somthing to specify the extensions required/offered, and this
being dependent for the corresponding stable version. Maybe setting
the require stable version plus a list of used/required extensions
would be sufficient:

{"version":0.2,"extensions":"abc,def"}

and maybe adding to that list of extensions what stable versions are compatible:

{"version":0.2,"extensions":{"abc":1.0,"def:">0.7"}}

[Michiel de Jong]

I think there has been a confusion. this came up in the irc channel yesterday as well, while talking with guybrush.

If someone want to build one closed system of application and storage node, then obviously, the only reason to have a protocol is for software compatibility, and then it makes sense to design an extensible protocol.

If, however, someone wants to offer just the application, and not offer any storage nodes, then people will come to use that application, and bring their own storage node. That is the case in which extensions would throw out the baby with the bathwater.

 So I understand that people are asking for a way to offer experimental features in storage nodes, and of requiring them in experimental applications, before these features go through the approval of this mailing list and make it into the stable protocol.

Since I think feature requests should always be driven by applications that need them ('feature pull'), and never by storage nodes trying to compete by adding value ('feature push'), i don't see the use in a command that makes the storage node list its capabilities. rather, it makes more sense to me to do it the other way around: an app decides it could be better if a storage node would have a certain feature. This should be a feature listed on our wiki, in a list of official, numbered, feature requests. These feature requests should not be modified - anybody should be able to add one. It would have to be a numbered lists, so that you get for instance 'FR-17' if your experimental feature is feature request number 17. Now the app asks the storage node 'do you have experimental support for FR-17?' by specifying "UJ/FR-17" instead of "UJ/0.2" in the 'protocol' field. in fact, it could save itself a roundtrip by directly trying and catching Bad Request responses.

So that's my proposal. We make a numbered list of feature requests on the wiki. There, every app developer can add their 'Cat.Explode()' command, so that it gets documented and has unique number assigned to it. Storage node implementers can choose to implement these experimental feature requests. Each time we feel it's time (maybe every 6 months or so), we issue a new version of the stable base.

Would that work for people?

Cheers,

Michiel

[Michiel de Jong]

On Mon, Jan 3, 2011 at 8:57 AM, Michiel de Jong <mic...@unhosted.org> wrote:

These feature requests should not be modified - anybody should be able to add one.

Sorry, i meant to say they should not be moderated. although modifying them after having defined them would obviously not be desirable either ;)

[Daniel Gröber]

Long email ahead but please bear with me I have some pretty cool ideas further
down ;)

From: Michiel de Jong <mic...@unhosted.org>

Subject: Re: [unhosted] Re: [proposal for UJ/0.2] Extensions and capabilities
Date: Mon, 3 Jan 2011 08:57:26 +0800

> So I understand that people are asking for a way to offer experimental
> features in storage nodes, and of requiring them in experimental applications,
> before these features go through the approval of this mailing list and make it
> into the stable protocol.

I personally am not asking for a way to offer experimental features but rather a
way to provide features that cannot be dictated by the protocol (because of
eg. resource constraints).

> Since I think feature requests should always be driven by applications that
> need them ('feature pull'), and never by storage nodes trying to compete by
> adding value ('feature push'), i don't see the use in a command that makes
> the storage node list its capabilities.

For unhosted to be any use for modern web applications we need a way for
application developers to ask the server what it can do. Not all servers are
going to be able to provide WebSockets, Video encoding or whatever. This could
of course be solved by making these things a protocol requirement or simply
having only one storage node implementation. But I just don't think that is
reasonable.

> rather, it makes more sense to me to
> do it the other way around: an app decides it could be better if a storage
> node would have a certain feature. This should be a feature listed on our
> wiki, in a list of official, numbered, feature requests. These feature
> requests should not be modified - anybody should be able to add one. It
> would have to be a numbered lists, so that you get for instance 'FR-17' if
> your experimental feature is feature request number 17.

There is actually something interesting that the OpenID and WebFinger/XRD guys
have. Namespaces for elements that are simply URIs. So instead of having a
centralized place for the specification of protocol features/elements they add a
namespace URI to all (non standard[or even to standard]) elements. That
namespace URI then potentially points to a website describing where to find the
specification for this element.

***

But now a different idea. What if instead of having only one single storage node
(server) per app we make it possible for a node[1] to explicitly state that it
supports (for example) only multimedia encoding and video streaming. Now some
other server could state that it supports storage of large files and has a
key-value store.

The user could now add both of these servers to a application he wishes to use
that requires the capabilities of those servers. The application then just looks
up the server it needs to use for a certain task.

Now you might be asking: "Daniel, are you crazy!? We cannot make our users go
hunting for a hundred servers if an app they want to use needs a lot of
capabilities." And you would be absolutely right with this argument. If it
weren't for the glory of distributed (automatic) service discovery.

By that i mean some sort of distributed protocol that all Unhosted server would
be required to speak. With this protocol a server would register itself in the
network when it goes up and that registration is accompanied by a list of
capabilities this server provides in addition to some other meta-data (free, ad
supported, paid server etc.).

Back to an app that needs video streaming + encoding and video archiving
(storage of large files). Once the user has specified a single server the app
can display a list of all servers that support the features the app needs. The
user can then choose one (or one might be chosen for him?) and can then start
using the application.

The only open question now is how do we actually specify these
capabilities. Now what i wrote further up about what OpenID/WebFinger is
interesting. If you identify capabilities by the URI to the specification (like:
'http://example.com/spec-example-1.0') developers can not only immediately find
the stuff they need to implement to get their app working. But we also have a
distributed specification where no one controls the specification of additional
capabilities :D

I really think making Unhosted a framework for extensions is a better idea than
making it a static protocol that cannot be extended in useful ways. Because
after all what we want to achieve is freeing the users from the monopoly
companies currently have on the web. And if a application needs to do something
that would not be possible with unhosted they would just continue doing what
they do right now. And we obviously don't want that. So we should make unhosted
sort of the path of least resistance ;)

Of course specifying a protocol if you need a new capability is a lot of work
but therefor a company can offload storage and processing to their users
instead of having to provide it.

Something that would also be nice to make this offloading work is having a
implementation that is as complete as possible (extension wise) that can run on
a plug server or some other sort of cheap hardware and is REALLY easy to
install. Just like 'Plug in and play' ;)

Tell me what you think.

---

[1] The term storage node is really no longer appropriate here as a what I am
describing would potentially do much more than just store data. So i think we
should call it just 'node' or 'Unhosted node'

[Ravi Pinjala]

Okay, this is really cool. :D

As long as we're thinking about discovery, there are two cases that we
should consider. First, discovery on the local network - zeroconf
solves this pretty well, and I for one think it'd be insanely cool if
you started an unhosted app and it could automatically detect and
start using storage and other services that are running on the local
network. (Yeah, I just stepped out of the browser - this could work
with a service running on localhost on a well-known port that does the
discovery and lets web-based apps access the data, or something) If we
had this, plug-style extensibility would become really compelling -
plug in the new device, and the app automatically starts using the new
service.

For remote service discovery, I suppose we'd have to ask the user for
their server preferences. To avoid making the user choose servers for
every random extension an app wants, what if we have the app provide
defaults? Then the author would be responsible for hosting any
nonstandard behavior that they want for their app.

--Ravi

[Curtis j Schofield]

On Sat, Jan 1, 2011 at 7:41 AM, Dmitry Tantsur <divius...@gmail.com> wrote:
> Hi and happy new year!=)
>
> I'd like to discuss some ways for servers to provide additional
> functionality.
> I suggest to add one more command to UJ/0.2:
>
> CAPABILITIES:
> This command is used to get information about additional functionality
> provided by storage node. POST fields:
>    "protocol": "UJ/0.2"
>    "method": "CAPABILITIES"
> Possible responses:
>    200 OK + a JSON string that decodes to exactly the following
> fields:
>        "value": list of strings which describe additional node
> capabilities (see below)
>    400 Bad Request (if what you posted was unparsable. preferably add
> a plain text explanation in the response body)

maybe 422 ?
https://secure.wikimedia.org/wikipedia/en/wiki/List_of_HTTP_status_codes

>    503 Service Unavailable (if the server is too busy)
>

in my understanding - it is a piece of infrastructure like HAPROXY or NGINX etc
that will return a 503 code - What does server too busy look like?

> Each capability should be in form: "command:property/X.Y" (each part
> being optional).
> 1st part allows to add new commands. For compatibility purpose any
> extension command should begin with some special symbol (say "_").
> 2nd part allows to add new parameters to existing commands.  For
> compatibility purpose any extension parameter should begin with some
> special symbol (say "_"). Without extension parameters command should
> behave as stated in the protocol.
> 3rd part is an optional extension version.
>
> Briefly:
> - Should be possible to add server-specific commands in a standard way
> - Should be possible to add server-specific parameters to commands in
> a standard way
> - Extensions should not affect clients that are not aware of them
> - If possible, extensions should be standardized.
>
> Examples:
> - capabilities ['_FOO'] - server supports "_FOO" command
> - capabilities ['GET:_bar'] - server supports "_bar" parameter for
> "GET" command
> - capabilities ['_FOO/0.1', 'GET:_bar/1.0'] - both with versions
> - capabilities [':_bar'] - server supports "_bar" parameter for any
> command

does this above syntax means that

GET /capabilities
returns JSON in a hash structure that represents
a list of actions and method and parameters

{
'capabilites =>
{ 'foo', => {
'GET' => [,'id','bar']
, 'POST' => ['bar']
},
,...
}
'version' => '0.2UJsSW+fishsticks'
}

--
make haste slowly \
festina lente  \
-
mobile  +1_415_632_6001
cur...@robotarmyma.de
http://robotarmyma.de

[Ravi Pinjala]

(finally getting around to replying to this...)

Yep, webfs can host unhosted or any other HTTP-based protocol. The
discovery mechanism is even flexible enough to support all versions of
the unhosted protocol simultaneously, and let the client choose one,
but I probably won't be able to do that because my free time is pretty
limited right now :( That's why I'm waiting for the protocol to
stabilize.

--Ravi

[pir...@gmail.com]

> Since I think feature requests should always be driven by applications that
> need them ('feature pull'), and never by storage nodes trying to compete by
> adding value ('feature push'), i don't see the use in a command that makes
> the storage node list its capabilities. rather, it makes more sense to me to
> do it the other way around: an app decides it could be better if a storage
> node would have a certain feature. This should be a feature listed on our
> wiki, in a list of official, numbered, feature requests. These feature
> requests should not be modified - anybody should be able to add one. It
> would have to be a numbered lists, so that you get for instance 'FR-17' if
> your experimental feature is feature request number 17. Now the app asks the
> storage node 'do you have experimental support for FR-17?' by specifying
> "UJ/FR-17" instead of "UJ/0.2" in the 'protocol' field. in fact, it could
> save itself a roundtrip by directly trying and catching Bad Request
> responses.

This wouldn't end up in a "do our homework" situation? Apart from
that, maybe an app require several experimental features, so just
"UJ/FR-17" is insufficient... Good idea, but in this case, is better
what i proposed about base version and list of extensions with it's
necesary version/revision:

{"version":"UJ/0.2","extensions":{"FR-17","FR-22":">0.3.1"}}

> So that's my proposal. We make a numbered list of feature requests on the
> wiki. There, every app developer can add their 'Cat.Explode()' command, so
> that it gets documented and has unique number assigned to it. Storage node
> implementers can choose to implement these experimental feature requests.
> Each time we feel it's time (maybe every 6 months or so), we issue a new
> version of the stable base.
> Would that work for people?

Ubuntu style? ;-)

[Michiel de Jong]

On Mon, Jan 3, 2011 at 1:37 PM, Daniel Gröber <dark...@gmail.com> wrote:

Long email ahead but please bear with me I have some pretty cool ideas further
down ;)

Finally, these arguments (including also Ravi's reply) are beginning to make some sense to me. You can always count on Daniel for some insightful guru meditation. :) I can see where you are going with this. I will be offline for about 48 hours now. It's an entirely different take from what I had been thinking of until now, though. I'll think about it in the airplane and reply when I arrive in Holland.

Cheers!

Michiel.

[Dmitry Tantsur]

BTW,
One of the possible extensions that is not directly connected with
commands - whether server supports keep-alive connections.
I mean we can use one connection for several requests but I'm not sure
it's supported by all server software.

Ideas?

On 3 янв, 14:34, Michiel de Jong <mich...@unhosted.org> wrote:

[Thad Guidry]

Think...Web sockets, web sockets, web sockets... if you want to stay alive ?
http://en.wikipedia.org/wiki/WebSockets

But encrypted tunnels might be a problem ? Not sure if TLS is a requirement.

2cents,

-Thad
http://www.freebase.com/view/en/thad_guidry

[Michiel de Jong]

Hi,

I thought about Daniel's latest suggestion, and I agree it makes sense to make the protocol modular. The first two modules will then be the KeyValue store ('KV') and the Messages system ('MSG'). People can then add all sorts of per-user unhosted services, like video, filesystems, instant messages (WebSockets), etcetera, without going through the central authority of this mailing list. They could simply make these modules discoverable per-user, through WebFinger. I do think we should encourage modules to support data migration, and that it makes sense to have the account-management actions centralized in an ACCT module that has a special superior role. So any node that offers anything, should at least provide ACCT support for it as well. Otherwise, everything is open.

I think this can lead to an open yet coordinated progress. We should simply make sure that good, modular, unhosted storage node software (supporting lots of modules) is always available. Then it will be easy for people to always have the latest modules, just like it's easy for people nowadays to always have the latest browsers features, even though there is a decentralized and speedy progress in these.

So each node should support at least the ACCT module, and can then choose to offer any other modules it wants. When registering unhosted storage from within an app, the app will look up the WebFinger with which you're logging on. In there, there should be a list of storage nodes, each with min&max version numbers per module that is supported. Not all modules need to be on the same storage node, you can combine three or four storage nodes to cover your different needs. If a certain module is not supported at all, then the app can failover to its per-app default node.

This should remove the need for feature requests as well. if you can't find the options you need in a module, just fork and create your own module that does support those special features you need.

The generic service discovery is solved by WebFinger. Declaring where a user's unhosted data should go, is controlled by whoever controls the email address's domain name. Per module, it makes reference to a globally unique URL specifying the protocol for that module. So the only centralized dependency is DNS. If there is no WebFinger on the email address then that corresponds to having no modules anywhere, so all modules will have to be provided by the app's default node.

So this means we should split up the protocol definition into per-module definitions (ACCT, KV, and MSG for now). Only the ACCT will keep a centralized character, that all apps and unhosted storage nodes should abide by. For the rest, both apps and storage nodes are free to introduce the modules they want, as long as they create a unique module name for it.

An added advantage of this modular approach is that versioning will also be per-module. So if a module is good the way it is, its version number doesn't need to increase any further, even though other modules do evolve separately. Let me know if everybody can find what they need in this proposal, and whether there are any worries about how this decision would affect progress and unison. This is probably the biggest and most important decision we'll ever make on this mailing list, so everybody please think it through thoroughly, and everybody please contribute any insights you may have from having dealt with similar decisions elsewhere.

As for keep-alive/WebSockets, i think it may make sense to do an IM module for instant messages, separate from the MSG module. That way, unhosted storage nodes that don't have WebSockets can choose the support the MSG module but not support the IM module.

Cheers,

Michiel

[Ravi Pinjala]

Two suggestions right away:

- Use a full URL for the module name, not just a short name. This
allows people to make up module names without having to worry about
collisions, and also makes the protocol a bit more self-documenting
because you can stick documentation at that URL. (I see this used a
lot for XML namespaces, it's an idea good enough to steal ;)

- Adding an "IM" module would seem to be moving in the direction of
having a new module for every application. That's definitely not what
we want. Are there smaller pieces that you could build an IM
application on top of? Like, maybe have the data structure on the
server be a text-based log, and add capabilities somehow to read the
last N lines of a text file, and append lines to a text file. (Just an
example, not saying this is how it should be done!) Unhosted will be
more useful in the long run if it provides general capabilities that
can be used for a lot of different apps.

--Ravi

[Michiel de Jong]

good points, both of them.

i didn't mean IM as in the application, more a way to do the MSG module (a basic SEND and RECEIVE), but with either long polling or WebSocket callbacks on the RECEIVE command, so that messages arrive as soon as they are created. optionally, this could i think be browser-to-browser (is this true?) and you can use the unhosted storage node only to set up the connection. But yeah, something as basic as possible, that could be used for any realtime events (like events in games for instance), not just chat.

[pir...@gmail.com]

> through WebFinger. I do think we should encourage modules to support data
> migration, and that it makes sense to have the account-management actions
> centralized in an ACCT module that has a special superior role. So any node
> that offers anything, should at least provide ACCT support for it as well.

I'm agree with the fact to make a modular protocol, but i think that
ACCT should be just a plain module like others, just for homogeneity,
and let the communications protocol as the common factor by itself,
and only maybe let the ACCT module as a requeriment to be an "UnHosted
certified" implementation. Maybe somebody can develop a chat app or a
ChatRoulette app and only need the MSG/IM/VIDEO/whatever module, but
since he is not storing data that could be migrated nor registration,
why he need a server with ACCT support? Just need a server that
provided the required service, no more, and the fact that it uses the
UnHosted communications protocol would easy up to integrate it with
the platform, although it would be no-certified server (it doesn't
need to, it just need only the communications protocol in fact).

I know maybe this is a little against UnHosted philosophy, but it's
just my opinnion... :-/

> So the only centralized dependency is DNS.

This is not fully true, just yesterday i learn about two
descentralized DNS that could be useful to us :-)

https://www.42registry.org/ A project to create a first level domain
(.42) just for open source projects (yes, the 42 was selected as a
meaningful joke for the project ;-) )
http://global-anycast.net/index.php/Main_Page

[Ricardo Gladwell]

On Wednesday, January 5, 2011 7:29:44 AM UTC, Michiel de Jong wrote:

Declaring where a user's unhosted data should go, is controlled by whoever controls the email address's domain name. Per module, it makes reference to a globally unique URL specifying the protocol for that module. So the only centralized dependency is DNS. If there is no WebFinger on the email address then that corresponds to having no modules anywhere, so all modules will have to be provided by the app's default node.

Isn't this a problem with using webfinger: the entity that controls the user's email address domain determines where a user's unhosted data goes, not the user themselves. Shouldn't the user be able to determine this themselves?

Also doesn't this rely on support for webfinger? Only Google seems to support the .known-files standard at the moment. Also how are you going to convince GMail (never mind other email providers like Hotmail) to support unhosted webfinger?

-- Ricardo Gladwell

[Michiel de Jong]

Good question. 

my motivation for this was mainly that if someone is going to set up an unhosted storage server for you, then chances are that they will also set up email for you. Like for instance your employer, or your university, your hobby club, or whoever. If you yourself own a domain name, then you could host WebFinger for it. In other words, people who have unhosted storage, are likely to also have (access to) WebFinger.

For hotmail users, right now my loginapp defaults to a default per-app storage node. I wanted to change it to make it default to a per-app default WebFinger provider.

The main motivations behind using email addresses as the 'hook' are that:

- people are used to using them as login identifiers

- it is free in theory. so anyone who cares, can use an email address that's not hotmail.

- we can still do per-app failovers if people do log in with hotmail addresses.

- i think google will make its WebFinger accounts editable at some point. Maybe simply an option to add custom xml-entries into your google profile LRRD/XRD doc. (if there are any google employees here, that's a nice 20%-time project: user-editable WebFinger profiles!)

- there aren't a lot of other options as far as i can see. i don't think we can do a lot better than WebFinger (unless you consider ways of improving DNS itself)

But please reply anybody who can think of improvements!

[Jan Wildeboer]

Am 2011-01-07 um 13:31 schrieb Michiel de Jong:

> Good question.

[...]

No improvements to offer, just some nasty questions to think about, inline.

> The main motivations behind using email addresses as the 'hook' are that:
> - people are used to using them as login identifiers

ACK, however quite some (younger, inexperienced) people think the same of their facebook name. E-Mail is not the only UID people think of nowadays.

> - it is free in theory. so anyone who cares, can use an email address that's not hotmail.

Do we have a defined way of transferring ownership of unhosted content when the user switches his email address for whatever reason (switches provider, employer, etc)? Can we have 1:n email addresses for the same content (AKA sharing between friends)? If my mail account gets "stolen" for whatever reason - can I block the unhosted content via some fallback mechanism?

> - we can still do per-app failovers if people do log in with hotmail addresses.
> - i think google will make its WebFinger accounts editable at some point. Maybe simply an option to add custom xml-entries into your google profile LRRD/XRD doc. (if there are any google employees here, that's a nice 20%-time project: user-editable WebFinger profiles!)

Take a look at webID(FOAF+SSL for some good input. Also look at what mozilla sync AKA weave does. Adding an unhosted vestor to the weave BLOB should be a no-brainer and chances are higher that Mozilla peeps can pull it of faster as google changing their webfinger implementation.

> - there aren't a lot of other options as far as i can see. i don't think we can do a lot better than WebFinger (unless you consider ways of improving DNS itself

The design should allow a modular approach - if better solutions are available, it should be possible to add them to unhosted in a simple way IMHO.

> But please reply anybody who can think of improvements!

Jan

[Ricardo Gladwell]

On Friday, January 7, 2011 12:31:32 PM UTC, Michiel de Jong wrote:

my motivation for this was mainly that if someone is going to set up an unhosted storage server for you, then chances are that they will also set up email for you.

Seems like a lot of additional infrastructure overhead, maintaining an SMTP/IMAP server is no simple thing.
 

Like for instance your employer, or your university, your hobby club, or whoever. If you yourself own a domain name, then you could host WebFinger for it. In other words, people who have unhosted storage, are likely to also have (access to) WebFinger.

Given that most users have a webmail account, you're asking unhosted users to maintain at least two email addresses *and* remember which unhosted-enabled email address to use when logging into an unhosted app. This seems like a lot to ask of users who might already be sceptical of using your new service.
 

For hotmail users, right now my loginapp defaults to a default per-app storage node. I wanted to change it to make it default to a per-app default WebFinger provider.

The main motivations behind using email addresses as the 'hook' are that:

- people are used to using them as login identifiers

I agree using email address as the hook is reasonable, I'm just not sure webfinger is the right protocol for service discovery. Maybe something like XMPP might be a more user-centric protocol?

-- Ricardo Gladwell

[Michiel de Jong]

On Fri, Jan 7, 2011 at 2:20 PM, Jan Wildeboer <j...@wildeboer.net> wrote:

ACK, however quite some (younger, inexperienced) people think the same of their facebook name. E-Mail is not the only UID people think of nowadays.

i think one important first decision is whether we want to reinvent DNS or not. I think a lot of people want to reinvent DNS, but that this should not be seen as part of unhosted. so in other words, unhosted should rely on 'DNS or some equivalent alternative', and externalise that part. So we need some sort of mapping from a globally unique user id onto two components: a 'domain' part that decodes to an IP address, and a 'local user' part that exists inside the domain. I see no room for any drastic deviation from what basically would be email. 

 

> - it is free in theory. so anyone who cares, can use an email address that's not hotmail.

Do we have a defined way of transferring ownership of unhosted content when the user switches his email address for whatever reason (switches provider, employer, etc)?

yes, or at least, this is easy enough.

 

Can we have 1:n email addresses for the same content (AKA sharing between friends)?

not at the moment, we can think about that.

 

If my mail account gets "stolen" for whatever reason - can I block the unhosted content via some fallback mechanism?

not at the moment, we can think about that.

 

> - we can still do per-app failovers if people do log in with hotmail addresses.
> - i think google will make its WebFinger accounts editable at some point. Maybe simply an option to add custom xml-entries into your google profile LRRD/XRD doc. (if there are any google employees here, that's a nice 20%-time project: user-editable WebFinger profiles!)

Take a look at webID(FOAF+SSL for some good input. Also look at what mozilla sync AKA weave does. Adding an unhosted vestor to the weave BLOB should be a no-brainer and chances are higher that Mozilla peeps can pull it of faster as google changing their webfinger implementation.

correct me if i'm wrong, but from what i've been reading i think

- FOAF+SSL requires state on the device (an installed client-cert), which we would like to avoid.

- mozilla weave uses https://auth.services.mozilla.com as its root server, which is good if you use it as a bookmark server, but not so elegant as an alternative to DNS in our login procedure. i think we should try to reuse ideas, designs and implementations from mozilla weave, instead of using them as a layer to build upon.

 

> - there aren't a lot of other options as far as i can see. i don't think we can do a lot better than WebFinger (unless you consider ways of improving DNS itself

The design should allow a modular approach - if better solutions are available, it should be possible to add them to unhosted in a simple way IMHO.

yes, we were thinking of using WebFinger to discover the modules. I can't think of a more minimal bootstrap for module loading than WebFinger. It is really just a 5-line xml file that serves as a redirector to where the real stuff lives.

 

we should all keep our eyes open for possible improvements, but for now i don't see any alternatives to WebFinger, other than designing something ourselves that would basically be doing nothing different from WebFinger, but doing it in a non-compatible way.

[Michiel de Jong]

On Fri, Jan 7, 2011 at 2:39 PM, Ricardo Gladwell <ricardo....@gmail.com> wrote:

On Friday, January 7, 2011 12:31:32 PM UTC, Michiel de Jong wrote:

my motivation for this was mainly that if someone is going to set up an unhosted storage server for you, then chances are that they will also set up email for you.

Seems like a lot of additional infrastructure overhead, maintaining an SMTP/IMAP server is no simple thing.

you could run WebFinger without running SMTP and imap. the term 'email address' would become a misnomer, but it would work.

 

 

Like for instance your employer, or your university, your hobby club, or whoever. If you yourself own a domain name, then you could host WebFinger for it. In other words, people who have unhosted storage, are likely to also have (access to) WebFinger.

Given that most users have a webmail account, you're asking unhosted users to maintain at least two email addresses *and* remember which unhosted-enabled email address to use when logging into an unhosted app. This seems like a lot to ask of users who might already be sceptical of using your new service.

yeah, true. but any identifier you use, whether you call it 'an email address' or 'a nickname' or whatever, would be either their existing email address, or if not, something extra to remember. and i think by using default failovers, people could use their webmail-address as a unique identifier, and then let that point to a different place in a per-app directory. just not a global directory.

 

 

For hotmail users, right now my loginapp defaults to a default per-app storage node. I wanted to change it to make it default to a per-app default WebFinger provider.

The main motivations behind using email addresses as the 'hook' are that:

- people are used to using them as login identifiers

I agree using email address as the hook is reasonable, I'm just not sure webfinger is the right protocol for service discovery. Maybe something like XMPP might be a more user-centric protocol?

-- Ricardo Gladwell

please elaborate on the xmpp suggestion! how would that work?

[Ricardo Gladwell]

On Friday, January 7, 2011 2:28:08 PM UTC, Michiel de Jong wrote:

you could run WebFinger without running SMTP and imap. the term 'email address' would become a misnomer, but it would work.

So, this would be an email-like string identifier, consisting of a unique user name and a domain name, separated by the "@" symbol, which would be use for unhosted service discover? A sort of "unhosted address" that might or might not match up to an actual email address?

If so, that is fine, the only problem I can see would be that it would confuse users when their unhosted address is not also an email address.
 

please elaborate on the xmpp suggestion! how would that work?

Not sure on the specifics but rather than do a webfinger look-up, you'd send some sort of XMPP message to the user's email address which would reply with the user's services. Perhaps an XMPP extension already exists for service discovery?

-- Ricardo Gladwell

[Daniel Gröber]

XMPP, that is an insanely awesome idea!

[Michiel de Jong]

On Fri, Jan 7, 2011 at 4:04 PM, Daniel Gröber <dark...@gmail.com> wrote:

XMPP, that is an insanely awesome idea!

 

please elaborate on the xmpp suggestion! how would that work?

Not sure on the specifics but rather than do a webfinger look-up, you'd send some sort of XMPP message to the user's email address which would reply with the user's services. Perhaps an XMPP extension already exists for service discovery?

-- Ricardo Gladwell

i still don't get it. XMPP to which server? how would hotmail not support WebFinger, but support replying to XMPP requests?

[Ricardo Gladwell]

On Friday, January 7, 2011 3:08:22 PM UTC, Michiel de Jong wrote:

i still don't get it. XMPP to which server? how would hotmail not support WebFinger, but support replying to XMPP requests?

It wouldn't, but XMPP support is more widespread than WebFinger support and most users now have an XMPP account, whether that is through their Facebook account, or GMail account.

But you didn't address my other remarks/questions?

-- Ricardo Gladwell

[Daniel Gröber]

Instead of requesting a email address form the user you could request a JID.

[Michiel de Jong]

Sorry, the answers to your other questions were:

On Fri, Jan 7, 2011 at 3:48 PM, Ricardo Gladwell <ricardo....@gmail.com> wrote:

So, this would be an email-like string identifier, consisting of a unique user name and a domain name, separated by the "@" symbol, which would be use for unhosted service discover?

yes

 

A sort of "unhosted address" that might or might not match up to an actual email address?

yes

 

If so, that is fine, the only problem I can see would be that it would confuse users when their unhosted address is not also an email address.
 

and yes. that's the non-ideal part of calling it an email address. but i thought the confusion would be greater if we call it 'Unhosted Id' or something similar. You can set up the mailserver to always forward to a secondary email address, or to always bounce with an informative message. i don't know, it's not ideal but it can be overcome.

[Michiel de Jong]

On Fri, Jan 7, 2011 at 4:22 PM, Daniel Gröber <dark...@gmail.com> wrote:

Instead of requesting a email address form the user you could request a JID.

yes, but how do I specify where my unhosted storage node is? you would still need some sort of link from the JID to the IP address of a storage node.

[Ricardo Gladwell]

This might do something similar for WebFinger service discovery with XMPP:

http://xmpp.org/extensions/xep-0030.html

[Thad Guidry]

Hmm.... I sort of like it. http://tools.ietf.org/html/rfc3920 -
Section 3.5 and 4.3

I wonder if the authentication could somehow use my OpenID ? It seems
to require the @ symbol ?

--
-Thad
http://www.freebase.com/view/en/thad_guidry

[Thad Guidry]

NICE. I also like the fact that you can bind to a specific resource with XMPP:

http://tools.ietf.org/html/rfc3920#section-7

--
-Thad
http://www.freebase.com/view/en/thad_guidry

[Ricardo Gladwell]

Not sure but I believe there is a way to handle OAuth over XMPP.

[Nate Benes]

Yep, check out XEP 235

http://xmpp.org/extensions/xep-0235.html

-Nate

[Ricardo Gladwell]

The only problem is finding client software that supports the extensions above.

[pir...@gmail.com]

> If so, that is fine, the only problem I can see would be that it would
> confuse users when their unhosted address is not also an email address.

Not really, if for the register process you need an email address, you
can always redirect the emails you receive at ni...@unhosted.node.org
to us...@email.com... I don't believe a only forward email system
should be too much difficult to implement inside unhosted code...

Related to the XMPP thing, i thought about it but as a basis for the
talked before unhosted IM extensions...