Camlistore Sharing Example

[Brad Fitzpatrick]

This is an example walk-though of (working) sharing on Camlistore.   Brett and I got this working last night (the basic "have a link" use case with no addition auth)

Let's say I have a private blobserver:

http://camlistore.org:3179/

And I have a file, "Hi.txt".

Its bytes are blob sha1-3dc1d1cfe92fce5f09d194ba73a0b023102c9b25

Its metadata (inode, filename, etc) is blob sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d

You don't have access to those, even though you know their names.  Verify the 401 errors:

http://camlistore.org:3179/camli/sha1-3dc1d1cfe92fce5f09d194ba73a0b023102c9b25

http://camlistore.org:3179/camli/sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d

(hm, those are returning Unauthorized errors, but no Content Body... will fix later)

Note also that any errors you get from my private blob server always delay for at least 200 ms to mask timing attacks that could otherwise reveal the existence or non-existence of a blob on my private server.

Now I want to share Hi.txt with you, so I create a share blob (e.g camput --share <blob>).

I've created this, and its name is sha1-071fda36c1bd9e4595ed16ab5e2a46d44491f708

Note that you can fetch it without authentication, because my blobserver knows I have it and that it's a share blob that doesn't require auth (authType == "haveref" ... like "Share with others that have the link")

Here's you getting the blob:

$ curl http://camlistore.org:3179/camli/sha1-071fda36c1bd9e4595ed16ab5e2a46d44491f708

{"camliVersion": 1,

  "authType": "haveref",

  "camliSigner": "sha1-f019d17dd308eebbd49fd94536eb67214c2f0587",

  "camliType": "share",

  "target": "sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d",

  "transitive": true

,"camliSig":"iQEcBAABAgAGBQJNQJGuAAoJEIUeCLJL7Fq1EuAIAL/nGoX8caGaANnam0bcIQT7C61wXMRW4qCCaFW+w67ys5z4ztfnTPKwL9ErzMF8Hd32Xe/bVcF6ZL38x/axqI7ehxN8lneKGQNoEdZDA9i752aAr0fkAba6eDehoOj9F4XxOzk3iVrq445jEXtu/+twamHV3UfRozWK1ZQb57dM+cRff47M/Y6VIBRSgW2BrABjuBs8G6PiKxycgh1mb+RL8f9KG+HB/yFuK37YJqZ0zU2OTRp6ELiOgTxbeg99koV9Duy4f4mQgxQgli46077Sv/ujzIeVbmdFL3OenGEzQnyKG0fhf8fa5WkED0XfH7zibAHLiSq3O7x11Q0406U==ANug"}

Note the "target" and "transitive".

Now we present this proof of access in subsequent requests in the "via" parameter, with the in-order path of access.

Here's the first hop to the metadata, in which we discover the blobRef of the bytes of the file (in this case, just one part is the whole file bytes...)  I already told you this earlier in the email, but assume you're just discovering this now.

$ curl http://camlistore.org:3179/camli/sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d?via=sha1-071fda36c1bd9e4595ed16ab5e2a46d44491f708

{"camliVersion": 1,

  "camliType": "file",

  "contentParts": [

    {

      "blobRef": "sha1-3dc1d1cfe92fce5f09d194ba73a0b023102c9b25",

      "size": 14

    }

  ],

  "fileName": "Hi.txt",

  "size": 14,

  "unixGroup": "camli",

  "unixGroupId": 1000,

  "unixMtime": "2011-01-26T21:11:22.152868825Z",

  "unixOwner": "camli",

  "unixOwnerId": 1000,

  "unixPermission": "0644"

}

Now let's get the final bytes of the file:

$ curl http://camlistore.org:3179/camli/sha1-3dc1d1cfe92fce5f09d194ba73a0b023102c9b25?via=sha1-071fda36c1bd9e4595ed16ab5e2a46d44491f708,sha1-0e5e60f367cc8156ae48198c496b2b2ebdf5313d

Hello, Camli!

That's it.

Now imagine different authType parameters (passwords, SSL certs, SSH, openid, oauth, facebook, membership in a group, whatever... )

[Kenton Varda]

Thanks for the explanation.  Things are somewhat clearer now.

Questions:

- Aren't the URLs going to get pretty unwieldy for deeply-nested objects?

- Is this access revocable?  I would argue that it doesn't need to be because the data is immutable, and leaking the share blob is equivalent to leaking all the data it references.  But it wasn't clear to me that you agreed with this logic.

- How do I delegate access to one sub-object to someone else, without delegating the whole tree that you gave me?

- How do I take an object that you shared with me (which is possibly just one sub-object of the share) and compose it with others into a new object, and then share that?

[Brad Fitzpatrick]

On Wed, Jan 26, 2011 at 3:38 PM, Kenton Varda <temp...@gmail.com> wrote:

Thanks for the explanation.  Things are somewhat clearer now.

Questions:

- Aren't the URLs going to get pretty unwieldy for deeply-nested objects?

In some cases, but we don't think that's a problem.  Old browsers had ~2k limits, but these won't be typically hit directly by browsers.  But in general things won't get that deep.  You could also imagine putting this in an http header or having a stateful protocol instead.  For now we like the stateless simplicity.

 

- Is this access revocable?  I would argue that it doesn't need to be because the data is immutable, and leaking the share blob is equivalent to leaking all the data it references.  But it wasn't clear to me that you agreed with this logic.

Yes, it will be in the future when the share blobref we give isn't the share JSON blobref itself, but the permanode + become claim + shareref (collectively: "object" or "share object").  Then we can issue a revocate claim against that permanode.  But that requires the blobserver doing a search server query to look for revocations.  Since that's not done yet we're cheating and doing just single blobrefs.  In the future the blobservers will also support taking a share blobref directly but then do an index lookup to find a permanode FOR that blobref, and then act as that share object's permanode was given, and proceed with checking for revocations.

This matters because the data *isn't* necessarily immutable:  we want to also be able to share a dynamic query object too, give people access to e.g. "the 20 most recent blog posts I've written".

   

- How do I delegate access to one sub-object to someone else, without delegating the whole tree that you gave me?

Your blob server could do that, with your personal blob server holding the share object and doing the proxying.  My blob server is oblivious to it.   (that said, this isn't a use case we've discussed much yet, so I don't feel too strongly, other than that this should be a) possible, b) done simply without complicating the root sharer)

 

- How do I take an object that you shared with me (which is possibly just one sub-object of the share) and compose it with others into a new object, and then share that?

"object" in camli terminology is a mutable object and involves hitting search servers with indexes.  a "blob" is the immutable thing.  If you want to re-share a blob subgraph, just clone it and make your own blobserver have a share blob/object that you grant to others.  If you want to re-share a blob object from me, you'll need to proxy, as above.

So far all the above is only about immutable blob graphs, though.  Things will get more fun and interesting when objects and search servers mature.

[Kenton Varda]

Proxying is a fine approach theoretically -- in fact, capability-based security is built on proxying in theory.  But the performance will be disappointing, taking either a lot of bandwidth (to sync upfront) or having high latency (to proxy on-demand).

After thinking about it a few days, I think I've come up with a capability-based approach that is almost as easy to use as my original "make get and set public" suggestion, but defends against probing.  You seem pretty set on doing things the way you're doing them (and as a fellow opinionated system designer I can't blame you for that), but just for the sake of sharing ideas, here's my approach:

- Within a single physical store, you can create multiple logical stores.  They behave exactly as if they were independent stores, but they are backed by the same physical storage, so identical blobs can be shared.  These sub-stores are cheap and so can be created and deleted rapidly.

- Each sub-store is independently access-controlled.  I'd use capability-based authorization but you could do identity-based as well.  If you have read access to a store, you can read *any* blob in the store.  If you have write access, you can write any blob -- but the hash has to be valid, of course.

- If you have read access to one store and write to another, you can sync content from the former store to the latter.  Syncing a blob implies syncing all blobs that it references transitively.  Nothing actually gets copied, of course; the store just updates lists of which blobs are accessible from which sub-stores.  (As an optimization, this sync can happen lazily as the tree is actually traversed.  In this case, trying to jump directly to the root to a deep leaf without accessing the intermediate blobs wouldn't work, but no reasonable use case would do that anyway.)

So, to share with someone, you create a new sub-store for things to be shared with them, and then you sync stuff into it.

- It's not necessary for URLs to express the full path to the target, because the URL identifies which sub-store is being used, and the system knows which blobs are visible in which stores.

- Access to a sub-store is revocable by the usual means for whatever authorization system it uses.

- Delegating access to a sub-object can be done by creating a new sub-store and syncing that object into it.

- Shared objects can be composed in their sub-store, or by syncing them into another sub-store.

Basically, it's semantically the same as the proxying approach you suggest, but without all the inefficiency.  It may sound a little weird at first, but if you think about it, syncing is already a fundamental part of the camlistore model.  If you can effectively accomplish higher-level tasks in terms of an operation you already have, rather than build a whole new system for it, why not do it?  Kill two birds with one stone, etc.

[Kenton Varda]

Brad/Brett, do you have any thoughts on this approach?  I might end up implementing it myself at some point, since I think I could do so while retaining compatibility with regular camlistore apps.  Would you accept this into the main repo or would I be maintaining a fork?

[Brad Fitzpatrick]

Kenton, sorry for the slow reply.

I want to make sure any feature we add to the blobserver layer is absolutely necessary and/or makes other things vastly simpler as a result.

Fortunately I think we might need (or want) this sub-store thing for a couple other things.  (Brett and I have been calling it "partitions" but perhaps that's not the best name either.)  Let me ignore sharing for a minute and discuss some of the other needs so we can work on unifying all the requirements.

Brett and I have debating a good way to handle the "resumed enumerate" for the full syncer process (mirroring to other blob servers of a user) as well as the resumed enumerate (and hanging GET or other real-time update mechanism) for the search server to index all the blobs in real-time and not have to do a full enumerate on restart.  (you should be able to kill the search indexer, change config, restart, and not have to re-enumerate the world....)

Partitions might help us here.  Still ignoring the sharing use cases, here is the partition design we think we'd like for the reliable queues for mirroring and indexing:

-- a blob server may have multiple partitions, but always has the "main" partition.

-- each partition has attributes on read, put, delete access.

-- the "main" partition's "delete" bit is set false (as it is now:  no delete command.  that's a trusted command for GC only)

-- the enumerate command takes an optional ?partition=<name> parameter, defaulting to "main".

-- new delete command, requiring a blobref and partition, checking permission (does that partition allow deletes?)

-- blobserver can be configured to mirror blobs in one partition (e.g. "main") to other partitions (e.g. "sync_queue" and "mirror_s3_queue")

-- the enumerate command gets a new ?hang_on_empty_seconds=<n> parameter.  if there are no results, the GET will hang for that many max seconds for blobs to appear in that partition.

Now the indexer and mirror tools are loops of enumerating their queue partitions and deleting blobs (in those partitions only) as they handle them.  Of course behind the scenes they're just refcounted/hardlinked etc.

This way we don't have to deal with time synchronization or global ordered, etc.

But it also means we now have a concept of light-weight partitions that we could possibly reuse for sharing.

 

This is still tentative for now.  I'm very reluctant to specify or implement any more at the blobserver layer until more is working end-to-end.  It's easy to take the best working piece and keep adding crap on to it before we have anything else.  ("when all you have is a hammer, everything looks like a nail", etc)

[Kenton Varda]

Good features solve multiple problems.  :)

One tweak I'd make to what you said:  Instead of specifying ?partition=<name> as a URL parameter, why not map the partition to an entirely separate URL?  I like this approach because it means that different partitions can be treated like completely separate stores, which I think simplifies the tools that work with them.  This way, apps written against camlistore can support partitions even if they haven't been explicitly designed to do so.

I think that every high-level operation that can be performed between two partitions should work between completely separate stores as well -- just not always as efficiently.  In your indexing use case, you are setting up the main partition to push changes to the indexing partition -- something that makes perfect sense to do between independent stores, but can be optimized behind the scenes in the partition case.  In fact, I could even see there being cases where you actually want the indexer to use a completely separate store, e.g. in a distributed environment where the indexer does not live near the live servers.  It would be neat if the design makes this trivial to implement.

[Brad Fitzpatrick]

On Mon, Jan 31, 2011 at 12:30 PM, Kenton Varda <temp...@gmail.com> wrote:

Good features solve multiple problems.  :)

One tweak I'd make to what you said:  Instead of specifying ?partition=<name> as a URL parameter, why not map the partition to an entirely separate URL?

Couple downsides, since you asked:  :-)

-- now we have to specify rules for allow characters in partition names.  at least URL escaping in URL parameters is well-defined.

-- now intermediate caches might not treat /camli/main/sha1-deadbeef and /camli/alt/sha1-deadbeef as the same object, even though they are.    (Brett and I didn't want to allow partitions for gets, at least for now, since we only need it in enumerate & delete, which aren't really cachable anyway)

Not sure either are good enough reasons, though.  I don't think proxies will be present or useful in practice.  (Clients will know the caching rules, or smart proxies can see blobrefs and "/camli/" in URLs.

 

 I like this approach because it means that different partitions can be treated like completely separate stores, which I think simplifies the tools that work with them.  This way, apps written against camlistore can support partitions even if they haven't been explicitly designed to do so.

Yeah, true.  I'd like to be able to define a camli server as simply a URL prefix (currently we describe it as a host:port, which is basically a prefix), so partitions perhaps could be:

http://host:port/partition-name/camli/sha1-xxx

http://host:port/partition-name/camli/enumerate?...

etc

Where partitions could also be different users on a hosted store.

Another property of a partition could be visibility of blobs between other partitions.  (so a pre-upload doesn't reveal to user B that user A has a certain blob, forcing them to re-upload it)

 

I think that every high-level operation that can be performed between two partitions should work between completely separate stores as well -- just not always as efficiently.  In your indexing use case, you are setting up the main partition to push changes to the indexing partition -- something that makes perfect sense to do between independent stores, but can be optimized behind the scenes in the partition case.  In fact, I could even see there being cases where you actually want the indexer to use a completely separate store, e.g. in a distributed environment where the indexer does not live near the live servers.  It would be neat if the design makes this trivial to implement.

agreed

[Kenton Varda]

Cool, sounds like we can agree on something.  I'm not in any rush to implement this, since I have quite a few other projects on my plate.  :)  But if you start down this road, let me know so that we can make sure the details work well for the sharing use case.  And if I do get my other stuff out of the way first, I'll take a crack at it.

[Brad Fitzpatrick]

On Mon, Jan 31, 2011 at 1:43 PM, Kenton Varda <temp...@gmail.com> wrote:

Cool, sounds like we can agree on something.

hey, I'm reasonable and logical.  I'm just also quick to say I don't like stuff without much regard to other's feelings or coming across as polite.  :-)

 

 I'm not in any rush to implement this, since I have quite a few other projects on my plate.  :)  But if you start down this road, let me know so that we can make sure the details work well for the sharing use case.  And if I do get my other stuff out of the way first, I'll take a crack at it.

Cool.  I'll implement this (probably today) for the queue-to-{mirror,indexer}.  But that's not much.  I imagine the sharing stuff will be more work.