Data DeDup

[Open Data]

Any good data dedup tools (gui or command line) for Mac OSX? My
google searches have been fruitless.

Thx, Chad

[Aaron Arnett]

What data format are you trying to dedup? For some formats (like line data) it's often easier to just use a text editor with good sort and regex support...

Aaron

On Feb 4, 2012 10:29 AM, "Open Data" <opend...@gmail.com> wrote:

[Noah Adler]

Data format would be good to know.  If it's really simple, there's always uniq on the commandline.  Google Refine also looks like a really good tool, but I don't have much hands-on experience with it.

--
You received this message because you are subscribed to the Google Groups "Collexion" group.
To post to this group, send email to coll...@googlegroups.com.
To unsubscribe from this group, send email to collexion+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/collexion?hl=en.

[Open Data]

I mean data dedup at the root level as in the entire file system
outside of necessary system and apps files. I know for a fact that I
have many many files that are duplicates. It's more of a housekeeping
task if you know what I mean. There are several "tools" available to
do this at the hardware level on servers, blade centers, enterprise
storage chassis, etc. but I haven't found anything useful for home
use.

> On Feb 4, 2012 10:29 AM, "Open Data" <opendata...@gmail.com> wrote:

[Todd Willey]

Use find + md5sum to generate a file that lists all files with their
sums. Then run an awk script like the one at
http://unstableme.blogspot.com/2008/05/count-number-of-occurrences-using-awk.html
so find which sums appear multiple times. Then grep the file of md5s
for each recurring one to get the file names.

-todd[1]

> --
> You received this message because you are subscribed to the Google Groups "Collexion" group.
> To post to this group, send email to coll...@googlegroups.com.
> To unsubscribe from this group, send email to collexion+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/collexion?hl=en.
>

--
CONFIDENTIALITY NOTICE:  This email is for sole use of intended
recipient(s).  Unauthorized use is prohibited.

[Mike Andrews]

There's always ZFS on a file server, which is still server stuff but is
at least open source :)

The dedup feature was pretty slow when I tried it, but it may have
improved in FreeBSD 9. Haven't tried it on Solaris/Illuminos yet.

[nx]

This will return a list of duplicate filenames from a list of md5sums and their filenames:

find /dir/of/files -type f -exec md5sum {} >> ~/sumlist \;
awk '{ print $1 }' ~/sumlist | sort | uniq -d | grep - ~/sumlist | awk '{ print $2 }'

--
nx <n...@nu-ex.com>

[nx]

Should have tested that. I can't pipe data to grep like that, instead I have to use xargs to run grep for each line of input. Here is the corrected script:

find /dir/of/files -type f -exec md5sum {} >> ~/sumlist \;

awk '{ print $1 }' ~/sumlist | sort | uniq -d | xargs -i grep {} ~/sumlist | awk '{ print $2 }'

--
nx <n...@nu-ex.com>