[ANN] rclone - rsync for object storage

[Nick Craig-Wood]

I'm looking for testers, users and developers for rclone the somewhat
ambitiously subtitled "rsync for object storage".

It is written in Go so may be of interest here!

-----

Rclone is a command line program to sync files and directories to and from

* Google Drive
* Amazon S3
* Openstack Swift / Rackspace cloud files / Memset Memstore
* The local filesystem

Features

* MD5SUMs checked at all times for file integrity
* Timestamps preserved on files
* Partial syncs supported on a whole file basis
* Copy mode to just copy new/changed files
* Sync mode to make a directory identical
* Check mode to check all MD5SUMs
* Can sync to and from network, eg two different Drive accounts

Links

* http://rclone.org/ - main website
* http://github.com/ncw/rclone - code/bugs
* https://google.com/+RcloneOrg - G+

--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

[Gustavo Niemeyer]

That looks very interesting, and I'll certainly have a closer look next week.

Thanks for publishing, Nick.


(trivial note: s/go install/go get/ on docs)

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

gustavo @ http://niemeyer.net

[Aaron Cannon]

Sounds awesome. Question: any reason you chose md5, over a non deprecated hash algorithm?

Aaron

--
This message was sent from a mobile device

[Daniel Theophanes]

rsync uses a hash to identify uniqueness in a non-security setting. In this case you want the fastest hash that does the job.

The only other hash I'd suggest using is blake2 (https://blake2.net/) but it probably doesn't have a practical difference for most people.

[Gustavo Niemeyer]

On Sat, May 31, 2014 at 5:46 AM, Daniel Theophanes <kard...@gmail.com> wrote:
> rsync uses a hash to identify uniqueness in a non-security setting. In this
> case you want the fastest hash that does the job.

The stronger digest is only ever applied after the rolling checksum
has matched, and the rolling checksum is only ever applied on a buffer
window with a given probability, so I doubt the speed difference is
relevant in this case. Much more likely, he's using MD5 because that's
already in use by some of those services. For example:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
https://developers.google.com/drive/v1/reference/files


gustavo @ http://niemeyer.net

[Daniel Theophanes]

I'm aware of how the rsync algo works [1], when I tested on my package it made enough difference that I kept it as md5 by default (though you can replace it with whatever Hasher you want). But at the time I was using it for rdiff for a file over 2 GiB, so the hash time added up.

1. https://bitbucket.org/kardianos/rsync/src/

[Gustavo Niemeyer]

On Sat, May 31, 2014 at 6:02 AM, Daniel Theophanes <kard...@gmail.com> wrote:
> I'm aware of how the rsync algo works [1], when I tested on my package it
> made enough difference that I kept it as md5 by default (though you can
> replace it with whatever Hasher you want). But at the time I was using it
> for rdiff for a file over 2 GiB, so the hash time added up.

Even for large files, the difference should be similar to proportional
to the difference of applying both algorithms to the file once.

> 1. https://bitbucket.org/kardianos/rsync/src/

That doesn't look right. The signature is being created based on fixed
block sizes. If you prepend 1 byte to the whole file, it'll resend the
whole file as the delta, because all of the blocks have been shifted.
You need to apply the same rolling checksum logic when creating the
signature to split the blocks at appropriate places and allow for
reuse.


gustavo @ http://niemeyer.net

[Aaron Cannon]

Have you considered CRC64 then?  I haven't tested it, but I suspect it would be an order of magnitude faster than any hash designed for security. 

Aaron

--

This message was sent from a mobile device

[Gustavo Niemeyer]

On Sat, May 31, 2014 at 6:23 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
>> 1. https://bitbucket.org/kardianos/rsync/src/
>
> That doesn't look right. The signature is being created based on fixed
> block sizes. If you prepend 1 byte to the whole file, it'll resend the

Sorry, I see your apply implementation also diverges from rsync:
you're computing the weak hash and looking for a weak match on every
single byte rather than doing what rsync does. Would get a significant
boost in speed by making the match more probabilistic, but the blocks
will vary in size.

gustavo @ http://niemeyer.net

[John Souvestre]

I had the same thought.  Also, if you aren’t comfortable with the 96 bits provided by the combination of the rolling checksum (32-bits) and a 64-bit CRC, then you could use a 128-bit CRC.  I understand it would take about 50% longer than a 64-bit CRC.

 

John

    John Souvestre - New Orleans LA

[Nick Craig-Wood]

On 31/05/14 02:35, Aaron Cannon wrote:
> Sounds awesome. Question: any reason you chose md5, over a non deprecated hash algorithm?

md5 is because it seems to have become a standard in object storage
systems for integrity checking.

All of s3, swift and google drive can report md5sums of the remote
objects, so you can check against the md5sum of the local file without
having to download the object.

[Nick Craig-Wood]

That is exactly the reason - you are right!

The fact that the remote object storage system can report the MD5
without you having to download the object is what makes rclone work.