Pipelining tar create and tar extract the "Python" way...

[Ray Van Dolson]

Hi all;

In the land'o'shell, I can do something like the following:

tar cvf - SrcDir | (cd /dest ; tar xvf -)

I'd like to learn the "Python" way to reproduce the above. Obviously I
could use the subprocess module and just call that exact command above,
but is there a way to do this with the tarfile module?

I realize I can use a StringIO object as the fileobj param then do the
extraction with the same StringIO object... but that's obviously not
the same as doing this all via a pipe as in the shell command.

I could see doing it with a fifo of some sort and having a consumer
process running that reads from the fifo and extracts with the tarfile
command as well... but that sounds overly complex.

I'd like to learn how to achieve this purely for curiosity's sake. I
realize that while the shell tar command is one of the faster ways to
recursively copy file trees around, there might be better ways to do
this from within python.. :)

Thanks,
Ray

[Ray Van Dolson]

On Wed, Sep 23, 2009 at 03:52:11PM -0700, Ray Van Dolson wrote:
> Hi all;
>
> In the land'o'shell, I can do something like the following:
>
> tar cvf - SrcDir | (cd /dest ; tar xvf -)
>

Bad form replying to my own post... while I'd still like to know if
this is possible to do with the tarfile class, it seems like using
subprocess.Popen() and calling tar from there with stdout set to PIPE
is probably the way to go.

I think this will result in the fastest way to copy files around.
Sounds like shutil.copytree() may not be all that robust (and probably
not very fast) in my version of Python (2.4.3 on RHEL5).

Still open to creative suggestions... :)

Ray

[Ray Van Dolson]

Never found a way to do this with the tarfile class directly, but used
Popen() to call tar:

# Time for some fancy shmancy calls to tar. First create the process that
# will generate our tar file and set it to output to a PIPE.
p1 = Popen(["tar", "cf", "-", "."], cwd=u['homedir'], stdout=PIPE)

# Next, set up our consumer tar process. This one should extract its data
# in the destination directory.
p2 = Popen(["tar", "xf", "-"], cwd=new_homedir, stdin=p1.stdout)

# Go!
err = p2.communicate()[1]

Nothing groundbreaking as this is from the examples in the
documentation, but just in case anyone else stumbles across this..

Ray

[Ishwor Gurung]

Ray.
Hi

>> >   tar cvf - SrcDir | (cd /dest ; tar xvf -)

Check this out Ray if you haven't done it already
http://docs.python.org/library/tarfile.html
"
The tarfile module makes it possible to __read__ and write tar
archives, including those using gzip or bz2 compression.
"
Try breaking it up the functionality.
That's how shell achieves this functionality as whole, why wouldn't
you apply same technique in Python :-)

Some modules that could help for your specific situations are tarfile and os.

Popen is one way(It's what your spec says) but you'd want to do purely
in Python. Aren't you curious? :-) It's late and TGIF! Need sleep.
Goodluck.

--
Regards,
Ishwor Gurung