bzip2 + large files ?
Michael Kraemer
I just came across a bz2-file that couldn't be decompressed
because of "file too large".
In compressed form it's just below 2GB, decompressed just above 2GB.
(ulimit is OK, large file enabled JFS, AIX 5.2).
I fetched the latest bzip2 version (1.0.5), compiled, but needed
cc -q64 and ar -X64 to obtain a binary which solved my problem.
This does, however, not run on 32bit hardware (of course).
Are those 64-bit flags mandatory to get large file support for
user apps ?
ISTR that large files were supported under AIX before the hardware and
the OS went 64-bit ?
Thomas Braunbeck
But they need to use open64, stat64...
PS: I just had inutoc failing on a bff file >2GB.
All needed to fix was adding 3 64 to enable it
to work with large files.
Michael Kraemer
> 32-bit programs can handle large files too.
> But they need to use open64, stat64...
that's what I expected somehow,
but it probably means I have to wade through bzip2's
sources to look what's going on :-(
Thomas Braunbeck
> Thomas Braunbeck schrieb:
>
> that's what I expected somehow,
> but it probably means I have to wade through bzip2's
> sources to look what's going on :-(
>
yes. However in the bzip2 case I think it fails on
a write() when the file gets >2GB. So you need to
find the open() and change it to open64(). A grep
over all source file to find all open() calls may
do it. Changing them all into open64() should be
no problem.
Thomas Braunbeck
> that's what I expected somehow,
> but it probably means I have to wade through bzip2's
> sources to look what's going on :-(
>
There is another option in case the problem is only
open() which needs to be changed to open64().
Build the 32bit version. Then dump -nv bzip2, look
for the open (coming from libc.a) and note the function
names listed before and after.
Use a binary editor on the executable, find this open
and change it to open64. You may want to work on a copy,
then dump -nv and see if the open changed to open64.
I once got a binary failing on files >2GB. The source
was no longer available to correct the open() calls.
I used od to locate the 'open' and dd to change it into
open64. The fixed binary is still working, now can handle
files >2GB.
The includes files defining the a.out format are shipped
with AIX and may give you an idea where in the binary the
functions from the shared libs are located.
Michael Kraemer
> yes. However in the bzip2 case I think it fails on
> a write() when the file gets >2GB. So you need to
> find the open() and change it to open64(). A grep
> over all source file to find all open() calls may
> do it. Changing them all into open64() should be
> no problem.
That's the kind of work I tried to avoid in the first place.
But if it's necessary to get a clean version ...
Michael Kraemer
> Michael Kraemer schrieb:
>
>> that's what I expected somehow,
>> but it probably means I have to wade through bzip2's
>> sources to look what's going on :-(
>>
>
> There is another option in case the problem is only
> open() which needs to be changed to open64().
(snip)
Appears like a hack to me.
I think I would prefer a clean solution,
probably other Unices (especially older ones)
will give similar 32/64 bit issues.
Darren Tucker
> I just came across a bz2-file that couldn't be decompressed
> because of "file too large".
Have you tried bzip2 -dc <file.bz2 >file ?
--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
Joachim Gann
The system header files define system calls and data structures
according to the directive _LARGE_FILES.
Defining this at compile time should allow your C programs to handle
large files.
Regards
bakan...@googlemail.com
If you just want to get it to work, the cat the file through bunzip,
i.e
cat largefile.bz2 | bunzip2 > uncompressed_file
Regards
Michael Kraemer
>
> If you just want to get it to work, the cat the file through bunzip,
> i.e
>
> cat largefile.bz2 | bunzip2 > uncompressed_file
Just for the records:
all of the workarounds were functional,
and I could also produce a large file enabled native version
(http://bio.gsi.de/DOCS/SOFTWARE/DOWNLOAD/AIX/POWER/52/).
thanks to all.