Protocol documentation and questions
119 views
Skip to first unread message
Nick Gerakines
Mar 7, 2013, 2:25:35 PM3/7/13
to camli...@googlegroups.com
Hey everyone,
I spent some time consolidating the protocol documentation as I implemented things.
http://ngerakines.github.com/camlispp/
The source of that generated output can be found here: https://github.com/ngerakines/camlispp/blob/master/docs/protocol.markdown
With that, I've got a few questions.
Q: Is there a list of hashing algorithms that all blob server implementations are expected to support. I'm currently using sha1 ("sha1"), sha256 ("sha256"), md5 ("md5"), MurmurHash3 ("murmur3") and CityHash ("ch32" and "ch64"), but if there are others then I'd like to add them to the docs.
Q: The camput tool attempts to hit the base url with a special "accepts" HTTP header value to get back a chunk of json that describes the camlistore services exposed by that host. Is that configuration specific to the camlistored and camput implementations or are all camlistore protocol implementations expected to provide that as well?
Q: In the docs I've read blob servers are logically private to a single user:
"The simplest and lowest layer of the Camlistore servers. A blob server, while potentially shared between users, is logically private to a single user and holds that user's blobs (whatever they may represent)."
So with that, is it assume that a single instance of a blob server represents a single user's stuff?
Q: Is the following behavior OK? If the blob server has multiple blob refs for the same blob then enumeration will include all possible refs that may lead to duplicate blob transfers. This may occur if the blob server supports multiple digests (sha1 and sha256, etc). Currently, there is no way to determine if a given blob ref has sibling blob refs.
Q: Is the following statement accurate? When enumerating, if the blob server supports multiple digest types (ie sha1 and sha256, etc), there is no way to iterate over blob refs of a specific digest type.
Q: Is the follow behavior OK? When enumerating, the "last" query string parameter must contain a blob ref of a blob that is known to exist. Partial blob refs (ie "sha1-77F5242") may cause unexpected behavior.
Q: What is the reasoning for randomly generated upload URLs?
Q: The upload and upload resume functionality seems to be in conflict. As per the documentation for the upload feature, if payload of a blob hashes to something different than what is provided by the name field, the server rejects the blob. Won't this happen with a partial upload making upload resume impossible?
Q: Do blob servers have any garbage collection duties?
I spent some time consolidating the protocol documentation as I implemented things.
http://ngerakines.github.com/camlispp/
The source of that generated output can be found here: https://github.com/ngerakines/camlispp/blob/master/docs/protocol.markdown
With that, I've got a few questions.
Q: Is there a list of hashing algorithms that all blob server implementations are expected to support. I'm currently using sha1 ("sha1"), sha256 ("sha256"), md5 ("md5"), MurmurHash3 ("murmur3") and CityHash ("ch32" and "ch64"), but if there are others then I'd like to add them to the docs.
Q: The camput tool attempts to hit the base url with a special "accepts" HTTP header value to get back a chunk of json that describes the camlistore services exposed by that host. Is that configuration specific to the camlistored and camput implementations or are all camlistore protocol implementations expected to provide that as well?
Q: In the docs I've read blob servers are logically private to a single user:
"The simplest and lowest layer of the Camlistore servers. A blob server, while potentially shared between users, is logically private to a single user and holds that user's blobs (whatever they may represent)."
So with that, is it assume that a single instance of a blob server represents a single user's stuff?
Q: Is the following behavior OK? If the blob server has multiple blob refs for the same blob then enumeration will include all possible refs that may lead to duplicate blob transfers. This may occur if the blob server supports multiple digests (sha1 and sha256, etc). Currently, there is no way to determine if a given blob ref has sibling blob refs.
Q: Is the following statement accurate? When enumerating, if the blob server supports multiple digest types (ie sha1 and sha256, etc), there is no way to iterate over blob refs of a specific digest type.
Q: Is the follow behavior OK? When enumerating, the "last" query string parameter must contain a blob ref of a blob that is known to exist. Partial blob refs (ie "sha1-77F5242") may cause unexpected behavior.
Q: What is the reasoning for randomly generated upload URLs?
Q: The upload and upload resume functionality seems to be in conflict. As per the documentation for the upload feature, if payload of a blob hashes to something different than what is provided by the name field, the server rejects the blob. Won't this happen with a partial upload making upload resume impossible?
Q: Do blob servers have any garbage collection duties?
Brad Fitzpatrick
Mar 7, 2013, 2:53:24 PM3/7/13
to camli...@googlegroups.com
On Thu, Mar 7, 2013 at 11:25 AM, Nick Gerakines <ni...@gerakines.net> wrote:
Hey everyone,
I spent some time consolidating the protocol documentation as I implemented things.
http://ngerakines.github.com/camlispp/
The source of that generated output can be found here: https://github.com/ngerakines/camlispp/blob/master/docs/protocol.markdown
With that, I've got a few questions.
Q: Is there a list of hashing algorithms that all blob server implementations are expected to support. I'm currently using sha1 ("sha1"), sha256 ("sha256"), md5 ("md5"), MurmurHash3 ("murmur3") and CityHash ("ch32" and "ch64"), but if there are others then I'd like to add them to the docs.
md5 should be banned. And murmur and city are out too, as they're not cryptographic hash functions.
sha1 and sha256 are fine.
No required ones, but I'd say minimum sha1 for today. There's no way (yet) to ask a server which it supports. That should be added to discovery. I've filed https://code.google.com/p/camlistore/issues/detail?id=122
Q: The camput tool attempts to hit the base url with a special "accepts" HTTP header value to get back a chunk of json that describes the camlistore services exposed by that host. Is that configuration specific to the camlistored and camput implementations or are all camlistore protocol implementations expected to provide that as well?
It's not required, but things work easier (much less configuration) if implemented. It should probably be documented, but it's also kinda obvious, so it hasn't been a priority.
Q: In the docs I've read blob servers are logically private to a single user:
"The simplest and lowest layer of the Camlistore servers. A blob server, while potentially shared between users, is logically private to a single user and holds that user's blobs (whatever they may represent)."
So with that, is it assume that a single instance of a blob server represents a single user's stuff?
The same base URL and same credentials == a single user's private stuff.
Q: Is the following behavior OK? If the blob server has multiple blob refs for the same blob then enumeration will include all possible refs that may lead to duplicate blob transfers. This may occur if the blob server supports multiple digests (sha1 and sha256, etc). Currently, there is no way to determine if a given blob ref has sibling blob refs.
Putting the same blob "xyz" in as sha1-xxxxxxxxxx and sha256-xxxxxxx is okay. Both should be returned. Whether the server dups some bytes behind the scenes is an implementation detail and not to be exposed in the enumerate response.
Q: Is the following statement accurate? When enumerating, if the blob server supports multiple digest types (ie sha1 and sha256, etc), there is no way to iterate over blob refs of a specific digest type.
You can start enumeration at a point, though.
Q: Is the follow behavior OK? When enumerating, the "last" query string parameter must contain a blob ref of a blob that is known to exist. Partial blob refs (ie "sha1-77F5242") may cause unexpected behavior.
I'm not sure that's true. It's just a string that's lexically compared.
Q: What is the reasoning for randomly generated upload URLs?
Legacy. App Engine's old limitations. We should fix that. Filed https://code.google.com/p/camlistore/issues/detail?id=123
Q: The upload and upload resume functionality seems to be in conflict. As per the documentation for the upload feature, if payload of a blob hashes to something different than what is provided by the name field, the server rejects the blob. Won't this happen with a partial upload making upload resume impossible?
I forget how this works. I'm not sure it was ever finished. It might pre-date the rolling checksum good file schema, so blobs shouldn't be so large they need resuming anyway. The data should be modeled differently.
Q: Do blob servers have any garbage collection duties?
No. A higher-level handles that. They shouldn't interpret their contents.
Nick Gerakines
Mar 7, 2013, 3:14:56 PM3/7/13
to camli...@googlegroups.com
Comments below.
On Thu, Mar 7, 2013 at 11:53 AM, Brad Fitzpatrick <br...@danga.com> wrote:
> On Thu, Mar 7, 2013 at 11:25 AM, Nick Gerakines <ni...@gerakines.net> wrote:
>>
>> Hey everyone,
>>
>> I spent some time consolidating the protocol documentation as I
>> implemented things.
>>
>> http://ngerakines.github.com/camlispp/
>>
>> The source of that generated output can be found here:
>> https://github.com/ngerakines/camlispp/blob/master/docs/protocol.markdown
>>
>> With that, I've got a few questions.
>>
>> Q: Is there a list of hashing algorithms that all blob server
>> implementations are expected to support. I'm currently using sha1 ("sha1"),
>> sha256 ("sha256"), md5 ("md5"), MurmurHash3 ("murmur3") and CityHash ("ch32"
>> and "ch64"), but if there are others then I'd like to add them to the docs.
>
>
> md5 should be banned. And murmur and city are out too, as they're not
> cryptographic hash functions.
>
> sha1 and sha256 are fine.
>
> No required ones, but I'd say minimum sha1 for today. There's no way (yet)
> to ask a server which it supports. That should be added to discovery. I've
> filed https://code.google.com/p/camlistore/issues/detail?id=122
I'll remove everything but sha1 and sha256 then.
>
>>
>> Q: The camput tool attempts to hit the base url with a special "accepts"
>> HTTP header value to get back a chunk of json that describes the camlistore
>> services exposed by that host. Is that configuration specific to the
>> camlistored and camput implementations or are all camlistore protocol
>> implementations expected to provide that as well?
>
>
> It's not required, but things work easier (much less configuration) if
> implemented. It should probably be documented, but it's also kinda obvious,
> so it hasn't been a priority.
>
I think the service discovery url and part of the stat url should be
consolidated. I can write up an issue in gh/gc if you'd like, but
doing something along the lines of this would be nice:
# Service Discovery
HTTP GET requests to the base url return a json data structure
(content-type: text/javascript) that returns the following elements:
blobRoot
: String; The url you just hit, can be used as a fully qualified
identifier of the blob server incase you were redirected there.
ownerName
: String; Contact details for the server operator or data owner
canLongPoll
: Boolean; Does the server support long-polling.
maxUploadSize
: Integer; An integer of max byte size for whole upload request
payload, which may be one or more blobs.
uploadUrl
: String; The full url to send upload requests to. Should usually be
some form of standard blobRoot + "/"? + "upload" string.
Granted, in my mind this would be part of each blob server. So if
you've got something like camlistored that has blobserver handlers at
different URLS (/bs1, /bs2, etc), the above data structure would be
return when doing a GET against each of them (ie GET /bs1/, etc)
>>
>> Q: In the docs I've read blob servers are logically private to a single
>> user:
>>
>> "The simplest and lowest layer of the Camlistore servers. A blob
>> server, while potentially shared between users, is logically private to a
>> single user and holds that user's blobs (whatever they may represent)."
>>
>> So with that, is it assume that a single instance of a blob server
>> represents a single user's stuff?
>
>
> The same base URL and same credentials == a single user's private stuff.
I didn't realize that the blob server needed to support
authentication. I may have some more questions about how that is
supposed to work later on.
>
>>
>> Q: Is the following behavior OK? If the blob server has multiple blob refs
>> for the same blob then enumeration will include all possible refs that may
>> lead to duplicate blob transfers. This may occur if the blob server supports
>> multiple digests (sha1 and sha256, etc). Currently, there is no way to
>> determine if a given blob ref has sibling blob refs.
>
>
> Putting the same blob "xyz" in as sha1-xxxxxxxxxx and sha256-xxxxxxx is
> okay. Both should be returned. Whether the server dups some bytes behind
> the scenes is an implementation detail and not to be exposed in the
> enumerate response.
Cool.
>
>>
>> Q: Is the following statement accurate? When enumerating, if the blob
>> server supports multiple digest types (ie sha1 and sha256, etc), there is no
>> way to iterate over blob refs of a specific digest type.
>
>
> You can start enumeration at a point, though.
>
>>
>> Q: Is the follow behavior OK? When enumerating, the "last" query string
>> parameter must contain a blob ref of a blob that is known to exist. Partial
>> blob refs (ie "sha1-77F5242") may cause unexpected behavior.
>
>
> I'm not sure that's true. It's just a string that's lexically compared.
Interesting. I'll make note and will write some unit tests to make
sure I'm treating enumeration the right way then.
>
>>
>> Q: What is the reasoning for randomly generated upload URLs?
>
>
> Legacy. App Engine's old limitations. We should fix that. Filed
> https://code.google.com/p/camlistore/issues/detail?id=123
>>
Great.
>>
>> Q: The upload and upload resume functionality seems to be in conflict. As
>> per the documentation for the upload feature, if payload of a blob hashes to
>> something different than what is provided by the name field, the server
>> rejects the blob. Won't this happen with a partial upload making upload
>> resume impossible?
>
>
> I forget how this works. I'm not sure it was ever finished. It might
> pre-date the rolling checksum good file schema, so blobs shouldn't be so
> large they need resuming anyway. The data should be modeled differently.
>
Ok. Given that, I'm probably going to avoid implementing upload resume
for the time being.
>>
>> Q: Do blob servers have any garbage collection duties?
>
>
> No. A higher-level handles that. They shouldn't interpret their contents.
>
Great.
> --
> You received this message because you are subscribed to the Google Groups
> "Camlistore" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to camlistore+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
On Thu, Mar 7, 2013 at 11:53 AM, Brad Fitzpatrick <br...@danga.com> wrote:
> On Thu, Mar 7, 2013 at 11:25 AM, Nick Gerakines <ni...@gerakines.net> wrote:
>>
>> Hey everyone,
>>
>> I spent some time consolidating the protocol documentation as I
>> implemented things.
>>
>> http://ngerakines.github.com/camlispp/
>>
>> The source of that generated output can be found here:
>> https://github.com/ngerakines/camlispp/blob/master/docs/protocol.markdown
>>
>> With that, I've got a few questions.
>>
>> Q: Is there a list of hashing algorithms that all blob server
>> implementations are expected to support. I'm currently using sha1 ("sha1"),
>> sha256 ("sha256"), md5 ("md5"), MurmurHash3 ("murmur3") and CityHash ("ch32"
>> and "ch64"), but if there are others then I'd like to add them to the docs.
>
>
> md5 should be banned. And murmur and city are out too, as they're not
> cryptographic hash functions.
>
> sha1 and sha256 are fine.
>
> No required ones, but I'd say minimum sha1 for today. There's no way (yet)
> to ask a server which it supports. That should be added to discovery. I've
> filed https://code.google.com/p/camlistore/issues/detail?id=122
>
>>
>> Q: The camput tool attempts to hit the base url with a special "accepts"
>> HTTP header value to get back a chunk of json that describes the camlistore
>> services exposed by that host. Is that configuration specific to the
>> camlistored and camput implementations or are all camlistore protocol
>> implementations expected to provide that as well?
>
>
> It's not required, but things work easier (much less configuration) if
> implemented. It should probably be documented, but it's also kinda obvious,
> so it hasn't been a priority.
>
consolidated. I can write up an issue in gh/gc if you'd like, but
doing something along the lines of this would be nice:
# Service Discovery
HTTP GET requests to the base url return a json data structure
(content-type: text/javascript) that returns the following elements:
blobRoot
: String; The url you just hit, can be used as a fully qualified
identifier of the blob server incase you were redirected there.
ownerName
: String; Contact details for the server operator or data owner
canLongPoll
: Boolean; Does the server support long-polling.
maxUploadSize
: Integer; An integer of max byte size for whole upload request
payload, which may be one or more blobs.
uploadUrl
: String; The full url to send upload requests to. Should usually be
some form of standard blobRoot + "/"? + "upload" string.
Granted, in my mind this would be part of each blob server. So if
you've got something like camlistored that has blobserver handlers at
different URLS (/bs1, /bs2, etc), the above data structure would be
return when doing a GET against each of them (ie GET /bs1/, etc)
>>
>> Q: In the docs I've read blob servers are logically private to a single
>> user:
>>
>> "The simplest and lowest layer of the Camlistore servers. A blob
>> server, while potentially shared between users, is logically private to a
>> single user and holds that user's blobs (whatever they may represent)."
>>
>> So with that, is it assume that a single instance of a blob server
>> represents a single user's stuff?
>
>
> The same base URL and same credentials == a single user's private stuff.
authentication. I may have some more questions about how that is
supposed to work later on.
>
>>
>> Q: Is the following behavior OK? If the blob server has multiple blob refs
>> for the same blob then enumeration will include all possible refs that may
>> lead to duplicate blob transfers. This may occur if the blob server supports
>> multiple digests (sha1 and sha256, etc). Currently, there is no way to
>> determine if a given blob ref has sibling blob refs.
>
>
> Putting the same blob "xyz" in as sha1-xxxxxxxxxx and sha256-xxxxxxx is
> okay. Both should be returned. Whether the server dups some bytes behind
> the scenes is an implementation detail and not to be exposed in the
> enumerate response.
>
>>
>> Q: Is the following statement accurate? When enumerating, if the blob
>> server supports multiple digest types (ie sha1 and sha256, etc), there is no
>> way to iterate over blob refs of a specific digest type.
>
>
> You can start enumeration at a point, though.
>
>>
>> Q: Is the follow behavior OK? When enumerating, the "last" query string
>> parameter must contain a blob ref of a blob that is known to exist. Partial
>> blob refs (ie "sha1-77F5242") may cause unexpected behavior.
>
>
> I'm not sure that's true. It's just a string that's lexically compared.
sure I'm treating enumeration the right way then.
>
>>
>> Q: What is the reasoning for randomly generated upload URLs?
>
>
> Legacy. App Engine's old limitations. We should fix that. Filed
> https://code.google.com/p/camlistore/issues/detail?id=123
>>
>>
>> Q: The upload and upload resume functionality seems to be in conflict. As
>> per the documentation for the upload feature, if payload of a blob hashes to
>> something different than what is provided by the name field, the server
>> rejects the blob. Won't this happen with a partial upload making upload
>> resume impossible?
>
>
> I forget how this works. I'm not sure it was ever finished. It might
> pre-date the rolling checksum good file schema, so blobs shouldn't be so
> large they need resuming anyway. The data should be modeled differently.
>
for the time being.
>>
>> Q: Do blob servers have any garbage collection duties?
>
>
> No. A higher-level handles that. They shouldn't interpret their contents.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Camlistore" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to camlistore+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
Reply all
Reply to author
Forward
0 new messages