CRC and file transfer

[Gueguen, Gretchen (gmg2n)]

Does anyone have any thoughts about using a tool that uses CRC to verify transfer instead of a MD5 or SHA checksum. I was looking at this tool TeraCopy (http://codesector.com/teracopy) to use to transfer some data from my local machine to network storage because it aids with the speed of transfer. It uses CRC to check that the transfer is complete and the data is fixed and I’m wondering if that would be sufficient without doing a separate check (MD5 checksums are being created for each file as part of accessioning, this would just cut out the step of doing them again and checking post-transfer).

 

Also, has anyone used this tool or can you suggest alternatives that are more archivally sound?

 

Gretchen Gueguen

Digital Archivist

Albert and Shirley Small Special Collections Library

University of Virginia

PO Box 400110

Charlottesville, VA 22904

(434) 924-4073

 

[Chris Prom]

Gretchen,

We have used that tool successfully here at Illinois.  I am not aware of a tool that uses MD5 or SHA checking on both sides.  Teracopy been useful to us, since it has caught several failed file transfers and while, while cutting the step of a manual check process, which would complicate the workflow quite a bit unless we were to develop some kind of automated checking tool. (if there is one out there, we'd consider using it.)

Chris Prom

University of Illinois Archives

-- 
You received this message because you are subscribed to the Google Groups "Digital Curation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-curati...@googlegroups.com.
To post to this group, send email to digital-...@googlegroups.com.
Visit this group at http://groups.google.com/group/digital-curation?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nathan Tallman]

I'd be interested in responses too. I use TeraCopy personally and somewhat professionally. Haven't fully adopted professionally because it uses CRC checksums instead of MD5 or better. Unfortunately, the paid version does not allow you to use MD5 either. It's a great utility. If there are alternatives that use MD5, would love to hear about it!

Nathan

--

[Seth Shaw]

I use the free TeraCopying all the time when loading materials into dark storage. Like you we generate/verify checksums (SHA256) at other points in the processes (before and after) and TeraCopy is great for a quick and easy copy/verify transfer when you don't need to retain the checksums from the action. 

For purely copy/verify activity I don't think MD5 (or SHA256 for that matter) is sufficiently "better" to discount using a CRC verification tool.  Yes, MD5 is "broken" but no one is going to be able to implement a man-in-the-middle attack on the fly when I am simply copying to another server and the likelihood of a random error collision is still astronomically low for CRC (how much tinier of a collision probability do you need to be "archivally sound"?).

     - Seth

[Chris Adams]

MD5/SHA-1 are basically free on any modern system compared to the overhead of the actual transfer (e.g. even my 3-year-old laptop can run SHA256 at gigabit ethernet wire speed using a single CPU core; md5/sha1 are considerably faster) so there isn't a strong pressure to do this for performance grounds unless you have both extremely fast network/storage equipment combined with extremely limited local CPU capacity. By the mid-2000s something like rsync was capable of running faster than the underlying network & disk arrays while performing block-level MD5s.

The downside to a CRC is that it produces a smaller size value, which directly increases the likelihood of a collision: to a first approximation, in the absence of malice you could expect an error roughly once every checksum-size bits transfered. For CRC-32 you're looking at once every 4GB or roughly 2^96 times more frequently than using MD5. Some research papers looking at error rates (e.g. http://academic.research.microsoft.com/Paper/22436.aspx) have found that errors occur more frequently than expected in the wild and, if I remember correctly, there have been a few reported cases where the errors produced were more likely to break a CRC than anticipated.

I'm not sure if I'm correctly interpreting your note that "MD5 checksums are being created for each file as part of accessioning, this would just cut out the step of doing them again and checking post-transfer" but I would advise against skipping a post-transfer check as network errors are not the only source of problems and it's not uncommon to encounter bit-errors on storage unless you're using a filesystem like ZFS which actively checksums data to prevent this. Running a post-transfer check against your original MD5s makes the choice of transfer tool far less important.

Chris