Need help with shutils.copytree

[Steven W. Orr]

I have a 'master' directory and a collection of 'slave' dirs. I want the
master to collect all of the stuff in the slave dirs.

The slaves all look like this,

.
|-- slaveX
| `-- archI
| | `-- distJ
| | | ` -- FILE

Where the different slaveX dirs may contain multiple occurrences of archI and
distJ, but across all slaveX dirs, there will only be one *unique* instance of
FILE in archI and distJ.

Here's an example: Given slave[1234], arch1 and arch2, and dist1 and dist2, I
want master to end up looking like this:

.
|-- master
| `-- arch1
| | ` -- dist1
| | | ` -- FILE
| `-- arch1
| | ` -- dist2
| | | ` -- FILE
| `-- arch2
| | ` -- dist1
| | | ` -- FILE
| `-- arch2
| | ` -- dist2
| | | ` -- FILE

etc...


In bash, I might use cpio passthrough mode and say something like:

master=$path_to_master
for slave in ${slaves}
do
pushd $slave
find . -print | cpio -pdum $master
popd
done

but I'm having a hard time trying to get this functionality in python. (I'm
trying to avoid writing a subprocess.)

I tried using shutil.copytree with a try / except that does a pass on OSError
(which is what gets raised when trying to create a dir that already exists).
No joy there. I also tried an ignore function that always returns ().

Someone must have done this before. Any suggestions / pointers are much
appreciated.

(I hope this was clear to read.)

TIA

--
Time flies like the wind. Fruit flies like a banana. Stranger things have .0.
happened but none stranger than this. Does your driver's license say Organ ..0
Donor?Black holes are where God divided by zero. Listen to me! We are all- 000
individuals! What if this weren't a hypothetical question?
steveo at syslang.net

[Chris Rebert]

You have multiple directories at the same level in the hierarchy with
identical names (e.g. two "arch1"s), which is invalid. I assume you
meant for them to be combined?

> In bash, I might use cpio passthrough mode and say something like:
>
> master=$path_to_master
> for slave in ${slaves}
> do
>    pushd $slave
>    find . -print | cpio -pdum $master
>    popd
> done
>
> but I'm having a hard time trying to get this functionality in python. (I'm
> trying to avoid writing a subprocess.)
>
> I tried using shutil.copytree with a try / except that does a pass on
> OSError (which is what gets raised when trying to create a dir that already
> exists). No joy there.

Right; the stack has already been unwound by the time your `except`
clause is reached.
You just need to recover by instead copying the children of the
subtree individually yourself when their parent already exists.
For example: master/arch1 already exists? Then copy the
slave/arch1/distN-s individually.

Or alternately, abandon copytree() entirely: you just LYBL and check
if the parent directories already exist; if not, you try to create the
directories yourself; and finally, you copy the individual files.
[Useful funcs: os.listdir(), os.path.exists(), os.mkdir() /
os.mkdirs()]

> I also tried an ignore function that always returns ().

That's an effective no-op which doesn't alter copytree()'s behavior whatsoever.

Cheers,
Chris
--
http://rebertia.com