Export to compressed files?

Tim Kearsley

unread,

Nov 11, 2002, 10:11:29 AM11/11/02

to

Hi all,

Environment here is:

Oracle 8i (mostly 8.1.7)
AIX 4.3.3
RS6000 SP hardware

For a long time I've used the technique of exporting databases
directly to a compressed file by using a named pipe as the FILE
parameter in the export and running a compress command in the
background to read whatever comes into the pipe and write out a
compressed file. This works very well.

The question is this:

Will a similar technique work for multiple files? That is, if I want
to generate a number of compressed files from the export (in my case
to avoid hitting a 2 Gbyte file size limit in a non large file-enabled
AIX filesystem) can I specify several named pipes in the FILE
parameter for the export and fire off a number of background processes
to read from the pipes? I'm not clear how the FILESIZE parameter for
the export would be used. Would the export start writing to another
pipe once it had written FILESIZE bytes to the previous pipe?

I'll experiemnt in the meantime, but would be very interested to know
whether anyone has tried this and if they were successful.

Many thanks.

Tim Kearsley
Database Manager
Milton Keynes Council

Tim Kearsley

unread,

Nov 12, 2002, 3:33:49 AM11/12/02

to

tim.ke...@milton-keynes.gov.uk (Tim Kearsley) wrote in message news:<725736ef.02111...@posting.google.com>...

Well I *have* experimented and have a bit more information now. It
seems you can use multiple named pipes in the FILE parameter for the
export and if you kick off a similar number of background processes to
read those pipes and compress what comes in to them then you get a set
of compressed export dumps.

For example, in the export parameter file:

FILE=/tmp/pipe1,/tmp/pipe2,/tmp/pipe3

Create the pipes with:

mknod /tmp/pipe1 p
mknod /tmp/pipe2 p
mknod /tmp/pipe3 p

And create background processes to compress what is read into the
pipe:

compress < /tmp/pipe1 > exp1.dmp.Z &
compress < /tmp/pipe2 > exp2.dmp.Z &
compress < /tmp/pipe3 > exp3.dmp.Z &

As I'm no Unix expert I wouldn't be at all surprised if there's a more
elegant way of doing this. If so, I'd love to know!

Cheers.

grjohnson

unread,

Nov 12, 2002, 6:27:55 PM11/12/02

to

I use a named pipe to for export large amount of data into a
compressed file. It's compresses the data as it goes and doesn't leave
it till the end.

First I create the named pipe.

mknod gzip_pipe p

nohup gzip < gzip_pipe > export_data.dmp.Z &

exp parfile=param.par

Parfile INCLUDES...

file='gzip_pipe'

Indicating that the export will go to the pipe and then compress into
export_data.dmp.Z

Now to import the data do the following

nohup gunzip -c export_data.dmp.Z > gizip_pipe &

imp parfile=impparam.par

Parfile INCLUDES...

file='gzip_pipe'

Cheers,

Greg Johnson
Oracle Certified Professional
OCP 8i DBA / OCP 9i DBA
Brisbane, Queensland, Australia

Bass Chorng

unread,

Nov 14, 2002, 4:25:55 PM11/14/02

to

Ever thought about using split ? We have done it before,
it goes something like this:

exp userid=/ full=y file=NamePipe compress=y &
cat NamePipe | compress -v > | split -b 1024M - ./full.dmp.Z

This should generate several compressed files with 1 GB each.

When you import you can do something like this:

zcat full.dmp[a-z] > full.dmp.p &
$ORACLE_HOME/bin/imp "USERID=SYSTEM/MANAGER BUFFER=256000 FULL=Y
FILE=full.dmp.p LOG=full.imp.log