Re: Digest for collexion@googlegroups.com - 2 Messages in 1 Topic
Open Data
Chad
Group: http://groups.google.com/group/collexion/topics
- Data DeDup [2 Updates]
Todd Willey <to...@rubidine.com> Feb 04 10:34PM -0500
Use find + md5sum to generate a file that lists all files with their
sums. Then run an awk script like the one at
http://unstableme.blogspot.com/2008/05/count-number-of-occurrences-using-awk.html
so find which sums appear multiple times. Then grep the file of md5s
for each recurring one to get the file names.
-todd[1]
--
CONFIDENTIALITY NOTICE: This email is for sole use of intended
recipient(s). Unauthorized use is prohibited.
Mike Andrews <mand...@bit0.com> Feb 05 01:22AM -0500
There's always ZFS on a file server, which is still server stuff but is
at least open source :)
The dedup feature was pretty slow when I tried it, but it may have
improved in FreeBSD 9. Haven't tried it on Solaris/Illuminos yet.
On 2/4/12 5:46 PM, Open Data wrote:
You received this message because you are subscribed to the Google Group collexion.
--
You can post via email.
To unsubscribe from this group, send an empty message.
For more options, visit this group.
You received this message because you are subscribed to the Google Groups "Collexion" group.
To post to this group, send email to coll...@googlegroups.com.
To unsubscribe from this group, send email to collexion+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/collexion?hl=en.
Dave
> From: Open Data <opend...@gmail.com>
> Subject: Re: Digest for coll...@googlegroups.com - 2 Messages in 1 Topic
> To: coll...@googlegroups.com
> Date: Monday, February 6, 2012, 11:36 AM
>
> Todd -- that looks like a winner. Thx.
Use a little caution in using the MD5 (or any other hash) to find matching
files. Remember that, just because the hashes from two files are the same,
that doesn't necessarily mean that the content of the files are identical;
it's possible for two differing files to hash to the same value (Ok, so it's
not very likely, but there is still a VERY small probability that it will
happen.). Still, it's a good technique for identifying candidates that may
be the same, but it's still a good idea to then verify that the contents
really are the same.
> Chad
Dave
Mike Andrews
paranoia check. I haven't seen a hash collision doing that yet. :)
Open Data
--
You received this message because you are subscribed to the Google Groups "Collexion" group.
To post to this group, send email to coll...@googlegroups.com.
To unsubscribe from this group, send email to collexion+unsubscribe@googlegroups.com.