Blobserver replication and sync implementations

[Nick Gerakines]

I'm starting on replication and sync, but wanted to clarify the behavior first.

Assuming you've got two blobservers,  each are configured to have the location of the other. On some sort of timer, each will attempt to enumerate the other's blobrefs and for each blobref that it doesn't have, attempt to get it and store it locally.

An alternative would be to have a "replicator" mode for one of the two. That replicator would perform the operations in the normal sync, but instead of having the other contact it, it would iterate over its own blobrefs and perform a stat on the remote server. If it finds that the remote doesn't have one of its own, it would attempt to submit the blob to it. This mode would be ideal for cases where one blobserver is unreachable (behind a firewall or the like).

Thoughts?

[Brad Fitzpatrick]

See how camtool sync and the built-in server sync work.  They share code.

--
You received this message because you are subscribed to the Google Groups "Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camlistore+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nick Gerakines]

Thanks.

[Brad Fitzpatrick]

The files are:

./cmd/camtool/sync.go

./pkg/server/sync.go

./pkg/client/sync.go

Basically it just enumerates both blobservers' blobs (in sorted order) to identify missing ones, and keeps a pool of blob copies running.

Thanks.

--

[Nick Gerakines]

Yeah, looks pretty straightforward.