File size error in os.stat with large files.

[Joe Woodward]

Hi,
Does anyone know the file size limit that os.stat will handle
correctly?

I have a 4.2GB file that os.stat is reporting to be 164MB. Smaller
files seem to work just fine, but I have not played with it to find
out where the cut-off point is.

I am using Python 2.1 on Windows 2000.

Thank you,

Joe Woodward
Phoenix Analysis and Design Technologies
Tempe, AZ

[John Hunter]

>>>>> "Joe" == Joe Woodward <joe.wo...@padtinc.com> writes:

Joe> I have a 4.2GB file that os.stat is reporting to be
Joe> 164MB. Smaller files seem to work just fine, but I have not
Joe> played with it to find out where the cut-off point is.

You have to compile python with large file support to handle files
over 2GB. To test whether you have this enabled:

>>> fd = open('somefile.dat', 'rw')
>>> fd.tell()
0L

If it returns '0', LFS is not enabled. if it returns '0L', LFS is enabled.

If not, you'll need to rebuild python for LFS (or try and find a
precompiled binary). I've done the build for linux and not widows, so
you'll need to adapt them to your environment/compiler. On linux, to
compile python with LFS, unpack the source distribution and run
./configure. Then edit the Makefile and set the OPTs to be:

OPT= -ggdb -O2 -Wall -Wstrict-prototypes -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

and 'make install'. Try the 'tell' test again and you should get '0L'.

Good luck,
John Hunter

[Skip Montanaro]

Joe> I have a 4.2GB file that os.stat is reporting to be 164MB. Smaller
Joe> files seem to work just fine, but I have not played with it to find
Joe> out where the cut-off point is.

Was Python compiled with large file support?

--
Skip Montanaro
sk...@pobox.com
consulting: http://manatee.mojam.com/~skip/resume.html

[Tim Peters]

[Joe Woodward]

> Does anyone know the file size limit that os.stat will handle
> correctly?

It depends on OS, Python version, and whether Python was compiled with
"large file" support.

> I have a 4.2GB file that os.stat is reporting to be 164MB. Smaller
> files seem to work just fine, but I have not played with it to find
> out where the cut-off point is.
>
> I am using Python 2.1 on Windows 2000.

Microsoft's implementations of the standard C functions for doing this stuff
are limited to 32-bit results, and there's no indication of overflow when
they're applied to files that are "too large" for that; information is just
lost. Python 2.1 uses these MS C functions in 2.1.

Substantial work was done for Python 2.2 to use better large-file functions
on Windows.

C:\Python21>dir \junk\big.file2
07/09/2002 03:57p 5,847,842,817 big.file2

C:\Python21>python
Python 2.1.3 (#35, Apr 8 2002, 17:47:50) [MSC 32 bit (Intel)] on win32
Type "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path.getsize('/junk/big.file2') # Nonsense result
1552875521
>>>

C:\Python22>python
Python 2.2.1 (#34, Apr 9 2002, 19:34:33) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path.getsize('/junk/big.file2') # Correct result, as unbounded long
5847842817L
>>>

(Note that getsize() calls a stat()-like routine under the covers; the same
kinds of concerns apply to other calls, like stat() and tell()). If you
need to do real work with large files on Windows, upgrade to 2.2.1.

[Matt Gerrans]

> You have to compile python with large file support to handle files
> over 2GB. To test whether you have this enabled:

Do you know why the binaries are not compiled with large file support as
default? Does it cause other problems or does it slow down os.stat()?

Also, if this is on Windows, short of recompiling Python, you could try
these two things (I don't have time to create a 2+ GB file to test whether
they actually work! If you are running Windows, you can try it out and let
us know):

from win32com.client import Dispatch
fs = Dispatch('Scripting.FileSystemObject')
fileob = fs.GetFile('c:\\temp\\tkinter.txt' )
print fileob.size

Also, another trick that is even more likely to yield correct results on the
Windows platform (sorry) is to use win32api.FindFiles() (which I think is
implemented with the FindFirstFile() and FindNextFile() APIs to build a
list). For each matched file, the tuple contains information from
WIN32_FIND_DATA, which includes two 32-bit values: nFileSizeHigh and
nFileSizeLow. So this should do the trick:

import win32api
fileinfo = win32api.FindFiles( 'c:\\temp\\tkinter.txt' )[0]
size = (long(fileinfo[4]) <<32) + fileinfo[5]

- Matt

[Skip Montanaro]

>> You have to compile python with large file support to handle files
>> over 2GB. To test whether you have this enabled:

Matt> Do you know why the binaries are not compiled with large file
Matt> support as default?

I suspect it allows the same binaries to run on a larger number of Windows
versions.

[François Pinard]

[John Hunter]

> >>> fd = open('somefile.dat', 'rw')
> >>> fd.tell()
> 0L

> If it returns '0', LFS is not enabled. if it returns '0L', LFS is enabled.

Clever. Yet it seems that, in the long run, Python evoluation plan is to
abolish user-visible distinction between usual integers and long integers.
So I wonder if, despite the trick above might work for some while, it is
really safe relying on it for programs meant to live a looong life :-).

I do not know, but am wondering.

--
François Pinard http://www.iro.umontreal.ca/~pinard

[Skip Montanaro]

>> If it returns '0', LFS is not enabled. if it returns '0L', LFS is
>> enabled.

François> Clever. Yet it seems that, in the long run, Python evoluation
François> plan is to abolish user-visible distinction between usual
François> integers and long integers. So I wonder if, despite the trick
François> above might work for some while, it is really safe relying on
François> it for programs meant to live a looong life :-).

It seems to me that exposing interpreter build options via the sys module
would be a Good Thing. You can mess around with
distutils.sysconfig.get_config_var("CPPFLAGS"), but that is a bit fragile
(the format can change over time or across compilers, making it difficult to
write robust code that sniffs out the useful bits). Then again, maybe
that's by design. Still, I think it ought to be easy to discover at runtime
if large file support, threading, or other optional capabilities were
compiled into the interpreter.

[Tim Peters]

[someone]

> You have to compile python with large file support to handle files
> over 2GB. To test whether you have this enabled:

[Matt Gerrans]

> Do you know why the binaries are not compiled with large file

> support as default?

[Skip Montanaro]

> I suspect it allows the same binaries to run on a larger number of Windows
> versions.

"Even Win95" supports 64-bit file sizes and offsets via the Win32 API, which
has long supported them (albeit that Win95 didn't support any filesystem
capable of storing such large files)! That's not the problem, and there's
no compile trick you can use in Python 2.1 or earlier to get large files to
"work right" under any flavor of Windows. Those versions of Python used
"the usual" assortment of C functions under Windows, and Microsoft's
implementations of those just plain don't work for large files. All the
relevant Windows file code in Python was rewritten for 2.2, to use crankier
functions that do work with large files (and on any Win32 box).

The only cures are to move to Python 2.2.1 (then it just works out of the
box), or eschew portable Python file operations and code to the Win32 API by
hand.

"large-file-support"-means-something-different-on-every-platform-
and-counting-unix-as-a-dozen-distinct-platforms-ly y'rs - tim