MemoryError, can I use more?
Echavarria Gregory, Maria Angelica
I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.
Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use...
Thanks,
Angelica.
sste...@gmail.com
How are you determining that it has occupied "exactly only 2.2GB?"
S
Mark Lawrence
> Dear group:
>
> I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.
>
> Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use...
>
> Angelica.
Please check the archives for the thread Please help with MemoryError,
it was only posted a day or two back. It's a problem specific to Windows.
Kindest regards.
Mark Lawrence.
Benjamin Kaplan
<m.echavar...@umiami.edu> wrote:
> Dear group:
>
> I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.
>
> Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use...
>
> Thanks,
> Angelica.
There's nothing you can do in Python to fix it. A MemoryError means
that Python asked the OS to malloc something and it refused to give it
any more memory. In this case, it's because Windows will only allocate
2GB of RAM to a single process. There is a way to extend that to 3GB,
but I don't quite understand how to do it- something about setting
linker flags. Easiest way to handle this would be to switch to a
64-bit Python on a 64-bit platform. That way, you can use up to 8TB of
memory.
http://msdn.microsoft.com/en-us/library/aa366778%28VS.85%29.aspx#memory_limits
Christian Heimes
> Dear group:
>
> I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.
The amount of RAM in your boxes is unrelated to your issue. A 32bit
process is able to address only 4GB of virtual memory because it has no
way to access memory addresses beyond the 32bit border. Large amount of
the address space are reserved for multiple purposes like error checking
(e.g. the NULL pointer), the entry point of the program, its stack and
address space for reallocating dynamic linked libraries. On modern Linux
systems the space for the heap is about 2.6 to 2.7 GB large. The heap is
the memory segment used by malloc() and friends. It looks like the
memory segment for the heap is slightly smaller on Windows.
Virtual memory isn't the same as resident memory. A program may only
occupy 100 MB in your RAM (resident memory) but it may use 1.5 GB of
virtual memory. Virtual memory takes memory fragmentation and unused
pages into account. If you malloc() 1 GB memory but just file up the
first byte your program requires 1 GB of virtual memory but only a few
KB resident memory. At least that's the story on Linux and several other
Unix-like OSes. I'm not sure how Windows deals with memory.
If your program needs more than 2 GB of RAM you should switch to a 64bit
OS and a 64bit version of Python. Windows AMD64 builds for Python are
available on python.
Christian
Nobody
wrote:
> I am developing a program using Python 2.5.4 in windows 32 OS. The amount
A single process can't use much more than 2GiB of RAM without a 64-bit CPU
and OS.
If you're stuck with a 32-bit system, you really need to figure out how to
limit the amount of data which is kept in memory at any one time.
Anssi Saari
> A single process can't use much more than 2GiB of RAM without a 64-bit CPU
> and OS.
That's not really true. Even Windows XP has the /3GB boot option to
allow 3 GiB per process. On PCs, free operating systems and server
Windows can use PAE to give access to full 4 GB per process.
Diez B. Roggisch
No, PAE can be used to access much more memory than 4GB - albeit through
paging. AFAIK up to 2^36 Bytes.
Diez
Anssi Saari
That too. I admit, after checking, that you can't go above 3 GiB per
process even in server Windows. But for Linux there exists (or
existed, since it seems it hasn't been updated since 2004) a kernel
patch which provides a "4GB/4GB" address split. Kernel is in one
segment, userland in another and hence a process can access full 4GB.
Laszlo Nagy
kernel + PAE. But the ~3G per 32bit process limit still applies. It is
because the PAE extension allows the kernel to distribute the same
virtual address ranges between different processes, but map them to
different physical memory addresses. However, the process that was
compiled and is running in 32 bit mode, cannot access more than ~3GB
simply because it is not able to address more memory locations with 32
bit addresses. (minus ~1G is because it needs virtual addresses for I/O
devices as well, and those addresses cannot be mapped to physical memory).
So with any Python that is running in 32 bit mode, you cannot use more
than ~3G memory. But you can start many instances of those programs and
use 2G for each process.
L
Ross Ridge
> No, PAE can be used to access much more memory than 4GB - albeit
> through paging. AFAIK up to 2^36 Bytes.
Anssi Saari <a...@sci.fi> wrote:
>That too. I admit, after checking, that you can't go above 3 GiB per
>process even in server Windows. But for Linux there exists (or
>existed, since it seems it hasn't been updated since 2004) a kernel
>patch which provides a "4GB/4GB" address split. Kernel is in one
>segment, userland in another and hence a process can access full 4GB.
Windows has a similar feature that allows 32-bit applications running on
64-bit versions of Windows to have a nearly 4Gb virtual address space.
Windows also allows 32-bit applications to use more than 4GB of physical
memory through a paging mechanism called "Address Windowing Extensions".
Also 32-bit applications can effectively use more than 4GB of RAM through
indirect means like multiple processes, the disk cache or video card RAM.
Ross Ridge
--
l/ // Ross Ridge -- The Great HTMU
[oo][oo] rri...@csclub.uwaterloo.ca
-()-/()/ http://www.csclub.uwaterloo.ca/~rridge/
db //
Diez B. Roggisch
> 2010.02.13. 17:40 keltez�ssel, Diez B. Roggisch �rta:
>> Am 13.02.10 17:18, schrieb Anssi Saari:
>>> Nobody<nob...@nowhere.com> writes:
>>>
>>>> A single process can't use much more than 2GiB of RAM without a
>>>> 64-bit CPU
>>>> and OS.
>>>
>>> That's not really true. Even Windows XP has the /3GB boot option to
>>> allow 3 GiB per process. On PCs, free operating systems and server
>>> Windows can use PAE to give access to full 4 GB per process.
>>
>> No, PAE can be used to access much more memory than 4GB - albeit
>> through paging. AFAIK up to 2^36 Bytes.
> PAE is for the kernel. Yes, you can use much more memory with 32 bit
> kernel + PAE. But the ~3G per 32bit process limit still applies. It is
> because the PAE extension allows the kernel to distribute the same
> virtual address ranges between different processes, but map them to
> different physical memory addresses. However, the process that was
> compiled and is running in 32 bit mode, cannot access more than ~3GB
> simply because it is not able to address more memory locations with 32
> bit addresses. (minus ~1G is because it needs virtual addresses for I/O
> devices as well, and those addresses cannot be mapped to physical memory).
No. It can access more, through paging, similar like mmap. Specialized
software like databases do that to keep large data in memory. And this
has nothing to do with compilation.
Of course this doesn't help for python, at least not for python-objects,
as these need a flat memory model. But it's not true that only the
kernels benefit from the PAE.
Diez
Echavarria Gregory, Maria Angelica
I use the physical and kernel memory boxes in the windows task manager under the performance tab... in that way I can see the exact RAM that just OS and idle processes occupy before I run my app, and then also the limit at which my app pushes the memory...
M. Angelica Echavarria-Gregory, M.Sc., E.I.
Ph.D Candidate
University of Miami
Phone 305 284-3611
________________________________________
From: sste...@gmail.com [sste...@gmail.com]
Sent: Friday, February 12, 2010 7:58 PM
To: Echavarria Gregory, Maria Angelica
Cc: pytho...@python.org
Subject: Re: MemoryError, can I use more?
On Feb 12, 2010, at 7:21 PM, Echavarria Gregory, Maria Angelica wrote:
> Dear group:
>
> I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.
>
> Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use...
How are you determining that it has occupied "exactly only 2.2GB?"
S
Echavarria Gregory, Maria Angelica
Dear Chris,
One of the machines I tested my app in is 64 bit and happened the same. The RAM consumed by the OS and other processes is already included in the 2.2 I'm telling... my app enters to work when the RAM is already consumed in ~600 MB in the 3- 32 bit machines ... in the 64 bit machine was exactly the same only that it started a little bit higher because it has windows 7... so should I understand that there is nothing I can do for my app to use up more RAM?
Thanks for your time and answer,
Angelica.
M. Angelica Echavarria-Gregory, M.Sc., E.I.
Ph.D Candidate
University of Miami
Phone 305 284-3611
________________________________
From: Chris Kaynor [cka...@zindagigames.com]
Sent: Friday, February 12, 2010 7:44 PM
To: Echavarria Gregory, Maria Angelica
Cc: pytho...@python.org
Subject: Re: MemoryError, can I use more?
A 32 bit app can only use 4 GB of memory itself (regardless of the amount of system ram), the OS claims some of this for the system, dlls occupy some of it, etc. As such, the app can only really use a smaller subset (generally between 2 to 3 GB, depending upon the app and the OS).
Chris
On Fri, Feb 12, 2010 at 4:21 PM, Echavarria Gregory, Maria Angelica <m.echavar...@umiami.edu<mailto:m.echavar...@umiami.edu>> wrote:
Dear group:
I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.
Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use...
Thanks,
Angelica.
--
http://mail.python.org/mailman/listinfo/python-list
Dave Angel
saying how you're "measuring" usage, we've all been guessing just which
one you're hitting. There's physical RAM, virtual address space, and
swappable space (swapfile on disk). Each reaches some limit in
different ways.
And there are probably a dozen different ways to measure "memory use,"
that get a dozen different answers. If you say which one you're using,
that gives "exactly 2.2 GB," maybe someone will be familiar with that
particular approach, and its meaning.
DaveA
Tommy Grav
On Feb 14, 2010, at 10:16 PM, Dave Angel wrote:
> There are three different limits at play here. Since you're still not saying how you're "measuring" usage, we've all been guessing just which one you're hitting. There's physical RAM, virtual address space, and swappable space (swapfile on disk). Each reaches some limit in different ways.
>
> And there are probably a dozen different ways to measure "memory use," that get a dozen different answers. If you say which one you're using, that gives "exactly 2.2 GB," maybe someone will be familiar with that particular approach, and its meaning.
I ran into the same problem on Mac OS X and investigated it. My problem was that the program reached a memory error
after using 2.2GB of memory. Now what caused it specifically I am less sure of, but switching to a 64bit python on
a 64bit OS (snow leopard) means that I now frequently move past this boundary (when I have enough memory of course :)
Tommy