Enumerating over blobs with multiple hashes

[Nick Gerakines]

Has anyone implemented a way to iterate over a list of blobs that may have multiple hashes associated with them?

I've been thinking about the case where a blob server may have a series of bytes {72, 101, 108, 108, 111} with a default hashing function of sha1 (f572d396fae9206628714fb2ce00f72e94f2258f). Another client comes along and puts the same byte array in (hooray) but with sha256 ("5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03"). Now, the blob server has a single blob with two references for it, so internally it should just be a single payload with multiple refs.

When enumerating over all of the blobs that the server has, I expect the list of refs to contain possible duplicates. As in both the sha256 and sha1 refs will be provided and it is up to the consumer of that API to fetch both and determine that they are duplicates. The downside is that duplicate data will be transmitted and both ends have to validate blob hashes when they are given them and when sync operations occur.

Is there a better way to handle this?

[Brad Fitzpatrick]

The reference counting like you alluded to seems straight-forward.  I haven't cared yet to implement it, though.  There have always been bigger, more important/interesting problems.

--
You received this message because you are subscribed to the Google Groups "Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camlistore+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nick Gerakines]

Ok, just wanted to make sure the direction that I'm going in isn't completely off.