How to clone a directory structure?
Martin Gross
image you have a (rather large) directory structure beneath
/somedir/parent/. This structure contains some directories containing
only directories, some only files and some both. Now, imagine you
want to clone the directory structure to /somedir/parent-clone/. This
clone should hold *only* the directory structure of the parent and not
any of the files contained in parent.
The clone structure will be filled with files as well, but these files
are different from parent's files. Whenever a tree in parent is being
added or deleted, this action should be performed on the clone as well
(on demand, not via daemon). If the action is a deletion in the
clone's tree, then for sure the files should also be deleted which are
in the clone's part which is to be deleted.
Does anyone know about a tool which is capable of doing so?
Thanks and cheers
Martin Gross
David
# Makes a directory tree with all directory permissions
#
# find . -type d | cpio -pdum TARGET_DIRECTORY
--
Confucius: He who play in root, eventually kill tree.
Registered with the Linux Counter. http://counter.li.org
Martin Gross
> #
> # find . -type d | cpio -pdum TARGET_DIRECTORY
Hi David,
Need tool to create the clone structure, thanks for the idea.
However, that's not the main problem. Main issue is about maintaining
the clone, i.e. in case I erase a directory inside the source/parent
tree, it should be deleted with the next "clone update" in the clone
tree as well.
If no other solution come up, I could still use find|cpio and would
only need to recursively traverse the clone for directories which need
to be deleted since they are gone in the parent; but it would be much
more need to have all that in one tool or command line expression.
Any more ideas?
Anonymous
> Need tool to create the clone structure, thanks for the idea.
> However, that's not the main problem. Main issue is about maintaining
> the clone, i.e. in case I erase a directory inside the source/parent
> tree, it should be deleted with the next "clone update" in the clone
> tree as well.
>
> If no other solution come up, I could still use find|cpio and would
> only need to recursively traverse the clone for directories which need
> to be deleted since they are gone in the parent; but it would be much
> more need to have all that in one tool or command line expression.
'rsync' is the tool for this job.
#!/bin/sh
# clone a directory stucture (no files copied)
f=/your/source/directory
t=/your/destination/directory
filelist=/tmp/files
mkdir -p $t
cd $f
# get the names of all files/sockets/etc. and remove the initial ./ part
find . -not -type d|sed 's@^\./@@' > $filelist
rsync -v -rlpogDt --delete --force --exclude-from=$filelist . $t
rm $filelist
Crd
(http://www.cis.upenn.edu/~bcpierce/unison/)
It's a file/dir sync tool, with many options.
Corrado
Martin Gross
> Martin Gross <ma...@writeme.com>:
> > Need tool to create the clone structure, thanks for the idea.
> > However, that's not the main problem. Main issue is about maintaining
> > the clone, i.e. in case I erase a directory inside the source/parent
> > tree, it should be deleted with the next "clone update" in the clone
> > tree as well.
>
>
Cool, that's really what I needed. I will first need to look into how
performance is - for the first steps the $filelist is a 6MB file with
some 60.000 file listings ;-) In the final application it should end
up with somewhere around 1/3 to 1/2 of that and I am not really sure
about the performance yet ;-) A simple argument such as --dirs-only
would probably be better, but seems not available.
But it'll work, and that's the point ;-)
Thanks, and a Happy New Year!
John-Paul Stewart
What about something like:
#!/bin/sh
# clone a directory stucture (no files copied)
# derived from a script by another poster
f=/your/source/directory
t=/your/destination/directory
dirlist=/tmp/dirs
mkdir -p $t
cd $f
# get the names of all directories and remove the initial ./
part
find . -type f | sed 's@^\./@@' > $dirlist
rsync -v -rlpogDt --delete --force --include-from=$dirlist
. $t
rm $dirlist
The script is untested; use it at your own risk. All I've
done is change the 'find' parameters to find and list the
directories _only_. Then I've changed the 'rsync'
parameters to include only those (rather than excluding 6MB
of filenames).
Again, this script is untested and is intended only to
demonstrate the concept.
Anonymous
> Cool, that's really what I needed. I will first need to look into how
> performance is
> A simple argument such as --dirs-only would probably be better, but
> seems not available.
Send your feature request to the 'rsync' developers. That's how
programs get better/smarter (and bulkier...)
> But it'll work, and that's the point ;-)
Good to hear it helped.
>
> Thanks, and a Happy New Year!
You are welcome.
Martin Gross
> What about something like:
...
> The script is untested; use it at your own risk. All I've
> done is change the 'find' parameters to find and list the
> directories _only_. Then I've changed the 'rsync'
> parameters to include only those (rather than excluding 6MB
> of filenames).
Thanks, John-Paul! Really stupid, I am. RTFM could have helped a
great deal here. Your version works when a --exclude '*' is added at
being the --include-from. However, the script can be even shorter and
become a one-line rsync command, basically:
#!/bin/sh
# clone a directory stucture (no files copied)
# Input from comp.os.linux.misc
f=/tmp/source
t=/tmp/target
cd $f
rsync -v -rlpogDt --delete --force --include '*/' --exclude '*' . $t
Really everything works - besides the only point is that the
combination "--delete --force" somehow doesn't work. Directories
deleted in source will be deleted in target as well, but only if
target does not contain any *files*. If the directory in target
contains nothing or directories only, it will be deleted. But not, if
it does contain any file... As soon as the files are deleted, next
execution of rsync successfully deletes the directories.
I'll do some more checks on this, but I guess this is a bug. Besides
this, no need to send an email to the developers, a flag
"--directories-only" is not needed since the line above does the job.
Even more funny, in the first script, the combination "--delete
--force" *does* work. Does anyone have any more hints on what I think
is a bug?
Cheers
Martin
Martin Gross
Please forget about the posting I submitted a couple of minutes ago as
reply to John-Paul's posting. I found the "bug" - it really is not
one, but there is another option which is necessary for this rsync.
The final, working code is:
#!/bin/sh
f=/tmp/source
t=/tmp/target
cd $f
rsync -v -rlpogDt --delete --delete-excluded --force --include '*/'
--exclude '*' . $t
The important part is "--delete-excluded", which according to the man
page is necessary to "also delete excluded files on the receiving
side". That's it ;-)
I also did a quick check on my real data structure (15k files, 414
directories) and an rsync without changes runs in the range <1 second
(the first script was somewhere around 3 min). Just the initial rsync
(to generate the clone for the first time) took by far too long, I
stopped it after some minutes when I saw that the clone was already
there (I don't know what rsync was waiting for at that time).
Some more checks will follow, but now I have the solution I needed,
thanks to you all!
Cheers
Martin