remoteStorage spec version 12.04 - first proposals
Michiel de Jong
1) - format of the lrdd link:
right now we do:
<Link rel="remoteStorage" template="https://example.com/storage/{category}/" auth="https://example.com/auth.html" api="CouchDB" />
but it has been brought to our attention that inventing link attributes like that is not really allowed. Also i think we don't need the '/{category}/' in the template, i think we can dictate that the template should always end in that, so that we don't have to specify it. so then I think it would be more correct to say:
<resource class="remoteStorage">
<Link rel="version" href="https://www.w3.org/community/unhosted/wiki/RemoteStorage#CouchDB-12.04" />
<Link rel="storage" href="https://example.com/storage" />
<Link rel="auth" href="https://example.com/auth.html" />
</resource>
notice how the string 'CouchDB' became a link that points inside the spec doc by calling it 'version' instead of 'api'. this way, we're also documenting authoratively what we mean by this webfinger entry, and we're making sure we can version it. makes this all a bit more sensible i think. comments welcome!
2) - read-only access
this was an idea from Garret. basically in OAuth, the scope would specify whether it's either read-only or read-write access that we're requesting (see also point 9 about the syntax of this). i think that's a nice addition and should be not too hard to implement for people. comments welcome!
3) - per-category data formats. Right now we're doing this here https://github.com/unhosted/website/wiki/categories we should move this to the w3c wiki, and mention it in the spec. So far the spec only prescribes how providers should behave. i think we can add a paragraph stating that RPs are expected to obey the per-category data format specs. comments welcome!
4) - adding an ACLs. i'm against this. we already have the 'public' category for allowing other people to read directly from your storage without your mediation. i see no need for allowing other people to write to it. the only thing you can do with that is either sending a message to someone (which is a separate issue, see below), and "writing on someone's wall". if someone wants to write, they can write on their own wall. they can then message me, and my friends saying 'hey, look, i wrote this'. but there's no need to use my storage for your data. and if we keep it this way, then everything is simpler. it's clear who should pay for disk quota, it's clear who is responsible for publishing something, it's clear who is authorative to say who can read something, and it's clear who is authoritive to delete something. comments welcome!
5) - 'inbox'. i'm not sure about this one. i'm certain we need messaging (again, see below). but if the recipient is not online, then the message can wait on the sender's side. that's what skype does. the message is delivered as soon as both contacts coincide online at the same time. a nice side-effect of this is that when the message is delivered, the recipient can reply in real-time (because you know the sender is online at the moment the message is delivered). i guess it depends on the type of application. if we see there is a concrete need for this in concrete applications, then we can add it in the next version, but i wouldn't add it in this one. comments welcome!
6) messaging - i propose adding xmpp-over-websockets. what i'm not sure about is whether that would then require all remoteStorage providers to hire an erlang engineer. also, i think we can do the same thing as with WebDAV, just say that it's one of the options for messaging, and allow multiple ones as long as it's reasonable to ask a client-side app to implement all of them. i'll write a separate thread about this.
7) fakefinger - we have been using fakefinger on Libre Docs. It's basically just unhosted's own central user database where /if/ you don't have webfinger (this is only a fallback of last resort), you create a record pointing to your remoteStorage, on the basis of BrowserID. I think it's time to write down what we're doing there. we could also add other alternatives to webfinger, like swd and xmpp disco and maybe dev-identity will propose a 'persona profile' standard at some point, although i doubt whether that's actually useful for anything. people should just implement webfinger, always. i'll write a separte thread about this.
8) a way to migrate your data - right now, there is no in-band way to migrate all your data from one storage to another. there are two things missing:
- list all categories
- list all items in a category
they are easy to add in all three stable api's (assume the instance address is /storage):
- CouchDB: we should formalize the mapping category->database, item->document that we're using. then, /storage/all_dbs and /storage/category/all_docs will work.
- WebDAV: we should formalize /storage/category/item as where we store each item, then PROPFIND /storage/ and PROPFIND /storage/category/ will return a 207 with an xml doc that contains all the category/item names.
- simple: here we need to add it. to keep it simple, i would say GET /storage/ and GET /storage/category/ (both ending in a trailing slash) can return a JSON array of strings.
comments welcome!
9) OAuth scopes format: we need new OAuth scopes for read-only, and for listing, and for messaging. we could also improve how auth to the 'public' category works, i think. so instead of just saying 'sandwiches' to implicitly mean read/write access, we could do 'sandwiches:rw'. we could use 'l' for listing, 'p' for public (and then the token would work for writing on /{storage}/public/{category}/{item}), and 'm' for messaging (see other thread). and then the empty string would mean all categories. so what is now the 'public' scope would become ':p', and ':rw' would be read/write access to all your categories. scope ':l' would be needed to list all categories. comments welcome!
If you think we should make any changes that are not among these 9 points, then also please reply! The current spec is here as you hopefully all know:
https://www.w3.org/community/unhosted/wiki/RemoteStorage
Cheers,
Michiel
nil
yesterday on #unhosted?
Melvin Carvalho
Hi! it's almost April so since we update our spec (which is our main product) in April and in October, let's start rounding up what we want to change in the spec this time. here are a few ideas:
1) - format of the lrdd link:
right now we do:
<Link rel="remoteStorage" template="https://example.com/storage/{category}/" auth="https://example.com/auth.html" api="CouchDB" />
but it has been brought to our attention that inventing link attributes like that is not really allowed. Also i think we don't need the '/{category}/' in the template, i think we can dictate that the template should always end in that, so that we don't have to specify it. so then I think it would be more correct to say:
<resource class="remoteStorage">
<Link rel="version" href="https://www.w3.org/community/unhosted/wiki/RemoteStorage#CouchDB-12.04" />
<Link rel="storage" href="https://example.com/storage" />
<Link rel="auth" href="https://example.com/auth.html" />
</resource>
notice how the string 'CouchDB' became a link that points inside the spec doc by calling it 'version' instead of 'api'. this way, we're also documenting authoratively what we mean by this webfinger entry, and we're making sure we can version it. makes this all a bit more sensible i think. comments welcome!
Would this change mean the XRD goes from invalid to valid?
2) - read-only access
this was an idea from Garret. basically in OAuth, the scope would specify whether it's either read-only or read-write access that we're requesting (see also point 9 about the syntax of this). i think that's a nice addition and should be not too hard to implement for people. comments welcome!
3) - per-category data formats. Right now we're doing this here https://github.com/unhosted/website/wiki/categories we should move this to the w3c wiki, and mention it in the spec. So far the spec only prescribes how providers should behave. i think we can add a paragraph stating that RPs are expected to obey the per-category data format specs. comments welcome!
4) - adding an ACLs. i'm against this. we already have the 'public' category for allowing other people to read directly from your storage without your mediation. i see no need for allowing other people to write to it. the only thing you can do with that is either sending a message to someone (which is a separate issue, see below), and "writing on someone's wall". if someone wants to write, they can write on their own wall. they can then message me, and my friends saying 'hey, look, i wrote this'. but there's no need to use my storage for your data. and if we keep it this way, then everything is simpler. it's clear who should pay for disk quota, it's clear who is responsible for publishing something, it's clear who is authorative to say who can read something, and it's clear who is authoritive to delete something. comments welcome!
5) - 'inbox'. i'm not sure about this one. i'm certain we need messaging (again, see below). but if the recipient is not online, then the message can wait on the sender's side. that's what skype does. the message is delivered as soon as both contacts coincide online at the same time. a nice side-effect of this is that when the message is delivered, the recipient can reply in real-time (because you know the sender is online at the moment the message is delivered). i guess it depends on the type of application. if we see there is a concrete need for this in concrete applications, then we can add it in the next version, but i wouldn't add it in this one. comments welcome!
6) messaging - i propose adding xmpp-over-websockets. what i'm not sure about is whether that would then require all remoteStorage providers to hire an erlang engineer. also, i think we can do the same thing as with WebDAV, just say that it's one of the options for messaging, and allow multiple ones as long as it's reasonable to ask a client-side app to implement all of them. i'll write a separate thread about this.
7) fakefinger - we have been using fakefinger on Libre Docs. It's basically just unhosted's own central user database where /if/ you don't have webfinger (this is only a fallback of last resort), you create a record pointing to your remoteStorage, on the basis of BrowserID. I think it's time to write down what we're doing there. we could also add other alternatives to webfinger, like swd and xmpp disco and maybe dev-identity will propose a 'persona profile' standard at some point, although i doubt whether that's actually useful for anything. people should just implement webfinger, always. i'll write a separte thread about this.
Will fakefinger be JSON or XML based?
Jan-Christoph Borchardt
> 5) - 'inbox'. i'm not sure about this one. i'm certain we need messaging
> (again, see below). but if the recipient is not online, then the message
> can wait on the sender's side. that's what skype does. the message is
> delivered as soon as both contacts coincide online at the same time. a nice
> side-effect of this is that when the message is delivered, the recipient
> can reply in real-time (because you know the sender is online at the moment
> the message is delivered). i guess it depends on the type of application.
> if we see there is a concrete need for this in concrete applications, then
> we can add it in the next version, but i wouldn't add it in this one.
> comments welcome!
I’m highly _for_ an inbox.
"What Skype does" is really annoying. People shouldn’t be pressed to
be online at the same time, forcing real-time communication as a
supposed nice side-effect.
A concrete need _is_ messaging. The spec will be valid until October,
in the meantime we will rely either on
* temporary hubs on the app side (like for Libre Docs) – which is not
purely client-side
* people being online at the same time, which is simply not acceptable
If we wait for it until the next version, communication apps are
effectively blocked for now.
Thad Guidry
I need to know about the real messaging scenario here.
Michiel de Jong
Thad Guidry
-Thad
http://www.freebase.com/view/en/thad_guidry
Michiel de Jong
So, in your mind...do you think that messages are generic ? and so notifications, messages, & chat, basically all forms could share the same low level service ? That form should be XMPP, as an extensible protocol, in my opinion...it's decentralized, and the idea of XMPP on top of WebSockets is really the future movement.
yes, i wrote about that in the other thread (the one called bla bla '- messaging')
Michael Williams
non-trivial app, so here's a promising messaging protocol that I
recently stumbled upon: http://about.psyc.eu/.
If we could integrate this with remote storage, that would be
beautiful. Yes email works, but we could do better, right?
Cheers,
Michael
Michiel de Jong
I agree messaging is very important if we want to make a creative
non-trivial app, so here's a promising messaging protocol that I
recently stumbled upon: http://about.psyc.eu/.
If we could integrate this with remote storage, that would be
beautiful. Yes email works, but we could do better, right?
Cheers,
Michael
Yeah, sure! but we have to think how we can get it onto websockets, as they are the only way (apart from http long polling) to get into the browser.
sender's browser -(A)> sender's relay server -(B)> recipient's delivery server -(C)> recipient's browser
The psyc message format http://www.psyc.eu/mmp.html can be used once this route is established. A and C would use websockets, B could use whatever it is psyc uses. We can pretty much define psyc-over-websockets to take the same format as psyc-over-tcp, maybe we need to say something about character encoding, but that would be about it. then you would just have to write a compatible client in javascript.
It's also possible that the recipient's delivery server accepts connections directly from the sender's browser, given that the recipient is local to it. That begs the question why we would use anything other than websockets for step B. in fact. if both sender and recipient are using a browser, then we probably never even need a step B. We just need the recipient's socket hub. If you don't have a sockethub yourself, you can either use a generic one (we may run one, maybe surfnet and riseup want to run one as well). otherwise, you will just always play the sender role in conversations.
Now apart from the fact that different people might run different client apps in their browsers, and it's nice if we can allow two or three different protocols, so that there is more chance that people's clients will have some protocol that both can speak, it's obviously interesting to send to and receive from people whose clients are not in the browser (yet). That just means that your socket hub should speak tcp as well as websocket. afaik all protocols we might want to be compatible with (smtp, xmpp, psyc, sip even?) go over tcp, so if we build a gateway server that tunnels between a websocket and a tcp socket, the rest of the protocol implementation can be done inside the browser, right?
Cheers!
Michiel
elf Pavlik
> I agree messaging is very important if we want to make a creative
> non-trivial app, so here's a promising messaging protocol that I
> recently stumbled upon: http://about.psyc.eu/.
>
> If we could integrate this with remote storage, that would be
> beautiful. Yes email works, but we could do better, right?
Jon Spriggs
application, is it worth talking to them to see how they're doing it?
--
Jon "The Nice Guy" Spriggs
elf Pavlik
> Given the Jappix guys have got a working XMPP web/javascript
> application, is it worth talking to them to see how they're doing it?
https://buddycloud.org/wiki/Main_Page
guys here have some nice web client and xmpp component both with coffeescript node etc. component uses pubsub for creating personal and topic channels, i've also seen some geo nodes under development etc. =)
Melvin Carvalho
I agree messaging is very important if we want to make a creative
non-trivial app, so here's a promising messaging protocol that I
recently stumbled upon: http://about.psyc.eu/.
If we could integrate this with remote storage, that would be
beautiful. Yes email works, but we could do better, right?
Not sure how much psyc will be used going forward, now that we have libretroshare.
The best integration of chat that I've seen so far is from the socialswarm / secushare guys that have ported libretroshrare into the browser via dooble / interace. Very impresive stuff.
Thad Guidry
Even so-called private messages from one person to another may better be modeled as minimal multicast contexts, because quite often graphical chat applications like to provide an "Add person to this chat" function, which would otherwise require a cumbersome transformation of protocol methods.
"
Michiel de Jong
Jan-Christoph Borchardt
"couchdb" to avoid any confusion.
Michiel de Jong
i think the urgent ones are:
- listing category names
- listing category contents
- per-category subfolders under 'public'
and also nice to have:
- for symmetry and also usefulness we might as well add 'folders'
inside the categories whose content you can list.
- read-only access to categories
to indicate a storage implements these new possibilities, we can
announce api types 'simple-12.04', 'webdav-12.04', 'couchdb-12.04'.
and OAuth scopes would then be ':r', to read all categories, ':rw' to
read/write all categories, and then per-category '<category>:r',
'<category>:rw'.
and then '<category>:rw' would be needed to write to public/<category>.
the api of remoteStorage.js would then also either get extra methods
get/set/remove-PublicItem, or and extra parameter which may default to
'private'. we'll see.
I would also rewrite the spec into two parts:
- one generic one describing Personal Data Services in general, and
our use of webfinger/swd and OAuth2.
- one describing the actual functionality of the storage, and its
possible REST APIs
Then we can later write a 'sockethub' spec that reuses the PDS spec,
and is just another personal data service that an unhosted web app can
discover and connect to.
is that ok for everyone?
Cheers,
Michiel
Michiel de Jong
space-separated scopes. now that we're already doing this in all
providers and all clients of the WebDAV, CouchDB and simple APIs,
let's keep doing it there, and make sure we do it right on both sides
for the new api types.
Jan-Christoph Borchardt
difference, should it? Or does that need to be specified?
You said:
- per-category subfolders under 'public'
When we talked I thought we agreed on the other way around – »public«
subfolders per category
Also, minor
'simple-12.04', 'webdav-12.04', 'couchdb-12.04'.
Should be 2012.04, it’s already hard to understand with Ubuntu version numbers.
Michiel de Jong
<j...@unhosted.org> wrote:
> If comma-separated or space-separated ideally should make no
> difference, should it? Or does that need to be specified?
OAuth specifies space-separated, we have just been doing it wrong.
>
> You said:
> - per-category subfolders under 'public'
> When we talked I thought we agreed on the other way around – »public«
> subfolders per category
Yeah but now i'm not so sure if it's necessary or even beneficial to
throw the whole folder structure around. it's also easier to implement
the way it is now, with only one place where public access is allowed
(one policy exception), and not unlimited numbers of small places
(lots of small policy exceptions). also, the public data will in
general not be human-readable so i think if you make a file manager
(as an unhosted app or hosted on the storage, like ownCloud has) then
it might be ok to have one special 'public' folder instead of having
them show up everywhere and having to click through 'private' to get
to the data you're looking for.
>
> Also, minor
> 'simple-12.04', 'webdav-12.04', 'couchdb-12.04'.
> Should be 2012.04, it’s already hard to understand with Ubuntu version numbers.
ok we can do that. actually i'm thinking now whether we should use a
version number instead because with the modular spec we won't have to
do scheduled updates anymore. we could also use -0.4 because it's the
4th iteration of the spec, and then plan to leave it at this version
forever, unless we find some serious problem (like now, the fact that
you can't write a migration app and that you can't protect one
category of public items from other categories)
Jan-Christoph Borchardt
> On Tue, Apr 17, 2012 at 9:55 PM, Jan-Christoph Borchardt
> <j...@unhosted.org> wrote:
>> If comma-separated or space-separated ideally should make no
>> difference, should it? Or does that need to be specified?
>
> OAuth specifies space-separated, we have just been doing it wrong.
Ok.
>>
>> You said:
>> - per-category subfolders under 'public'
>> When we talked I thought we agreed on the other way around – »public«
>> subfolders per category
>
> Yeah but now i'm not so sure if it's necessary or even beneficial to
> throw the whole folder structure around. it's also easier to implement
> the way it is now, with only one place where public access is allowed
> (one policy exception), and not unlimited numbers of small places
> (lots of small policy exceptions). also, the public data will in
> general not be human-readable so i think if you make a file manager
> (as an unhosted app or hosted on the storage, like ownCloud has) then
> it might be ok to have one special 'public' folder instead of having
> them show up everywhere and having to click through 'private' to get
> to the data you're looking for.
Private stuff would be directly in the category folder, and »public«
would be a special folder inside the category. I agree that having
both a »public« and »private« folder inside each category would be
stupid.
From a mental model it makes more sense to have the public stuff be
part of the category, because being »public« doesn’t say anything
about the document. It’s nonsensical as a group, next to the other
topic-based categories. We should correct it now while we still can.
>
>>
>> Also, minor
>> 'simple-12.04', 'webdav-12.04', 'couchdb-12.04'.
>> Should be 2012.04, it’s already hard to understand with Ubuntu version numbers.
>
> ok we can do that. actually i'm thinking now whether we should use a
> version number instead because with the modular spec we won't have to
> do scheduled updates anymore. we could also use -0.4 because it's the
> 4th iteration of the spec, and then plan to leave it at this version
> forever, unless we find some serious problem (like now, the fact that
> you can't write a migration app and that you can't protect one
> category of public items from other categories)
Yeah, whatever is best there. The version numbers of the APIs won’t
increase with the remoteStorage / remoteStorage.js versions, right?