I'm looking for a program that will go through a number of files...

[jules Gilbert]

Fully decompressed, I have (this is an estimate only!,) about 20-30 TB
of files, some, .gz, some .bz2, some .tar. some .arj, you get the
idea. I think the largest file is, say, around 250GB.

And these files are, in some cases, duplicates of others. Or almost
duplicates, but for a few diff's.

Also, I want to apply certain rules, such as always deleting a file
that is "*.core" (which is a UN*X style core dump.)

I want a program that will rummage through my files, organizing, etc.
I intend to use the dar program. That's "D-ar", a French built and
managed open-source tool built to replace tar but with much greater
functionality. I have the source and am attempting to extend it, at
least for internal use.

I've purchased one of those new WD external drives, called a "My Book"
Essential. (I think that's the name, don't have the box handy.) In
fact I got that drive expecting to use it with my laptop but
discovered that USB 3.0 is pretty new and had to purchase a PCI card
converter. No big deal, just be careful if you're thinking of getting
one.

I've been backing up for years and years and now have thousands
of ,tar files. And want help from a program in reorganizing these
bunches of files. Does such a program exist??

[stan]

jules Gilbert wrote:
> Fully decompressed, I have (this is an estimate only!,) about 20-30 TB
> of files, some, .gz, some .bz2, some .tar. some .arj, you get the
> idea. I think the largest file is, say, around 250GB.

The first step would seem t clearly define the problem.

> I want a program that will rummage through my files, organizing, etc.
> I intend to use the dar program. That's "D-ar", a French built and
> managed open-source tool built to replace tar but with much greater
> functionality. I have the source and am attempting to extend it, at
> least for internal use.

The tar utility has been used for backups for many years, if this dar
is better why not ask people who use dar? IOW why would anyone here
know anything about it?

> I've been backing up for years and years and now have thousands
> of ,tar files. And want help from a program in reorganizing these
> bunches of files. Does such a program exist??

It's very difficult to imagine any program could read your mind to
determine the changes you want. Your vague description would seem to
call for a simple shell script but I can't imagine why you would think
this is the place to discuss shell scripts or backups.

For the record, why not just apply your secret sauce and shrink all
your stuff down to fit in on a floppy? Your system works right?

[Alex Mizrahi]

> I've been backing up for years and years and now have thousands
> of ,tar files. And want help from a program in reorganizing these
> bunches of files. Does such a program exist??

UNIX shell such as bash or zsh might help, I guess, it won't think for you.

> Also, I want to apply certain rules, such as always deleting a file
> that is "*.core" (which is a UN*X style core dump.)

In zsh it is

rm **/*.core

Simple, eh?

[jules Gilbert]

Oh no, apparently you don't know that several large firms who's
business involves IT storage solutions have developed or purchased
(for their customer's,) programs that go through their clients files
doing things like, seeing two nearly identical files, represent the
older, but the newer is replaced by a link to the older plus the
necessary diff's.

You're not up to date. What I was looking for (I agree, I didn't say
this specifically,) is an open-source program that does this. Or is
it too soon?

[jules Gilbert]

that's really cool! I'm going to load that on my laptop today. Thank
you.

[Thomas Richter]

And in a standard POSIX environment, it is

find . -name "*.core" -exec rm \{\} \;

but what is the relation to compression? This is just a simple trivial
exercise in the POSIX toolchain.