Memory error only in Spyder IDE

[Patrick Leinenkugel]

Hi,

I am using Spyder within python (x,y) (Version 2.7.6.1) on Windows. Doing the following causes a MemoryError in my Spyder Python IDE:

>>> from numpy  import *
>>> a_flt = ones((7000,7000), dtype=float64)+4
>>> b_flt = ones((7000,7000), dtype=float64)+1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>    
MemoryError
>>> 

This is weird, since the memory usage in the statusbar of Spyder shows that only approx. 25% of my memory is used. Furthermore, when generating even a higher number of these 7000*7000 arrays in the standard Python IDE GUI, everything works fine.

>>> from numpy  import *
>>> a_flt = ones((7000,7000), dtype=float64)+4
>>> b_flt = ones((7000,7000), dtype=float64)+1
>>> c_flt = ones((7000,7000), dtype=float64)+1
>>> d_flt = ones((7000,7000), dtype=float64)+1
>>> e_flt = ones((7000,7000), dtype=float64)+1 

Even with 5 floating point arrays created, memory requirements only amount to about a third of my total memory of 16GB. So this cannot be a real memory problem. Then updated to Spyder 2.3.1 using Python 2.7.6 32bits, Qt 4.8.4, PyQt4 (APIv2) 4.9.6 on Windows and the same Memory Error results alreay when trying to create the first array with 7000*7000 float elements (a_flt)! Furthermore, the same error is reproducable in the Spyder IDE on the computer of my colleague, so it is not an error related to my specific computer settings. 

I also found a similiar question on this issue in this spyder google group https://groups.google.com/forum/#!msg/spyderlib/qE9tiriT91s/0x3s2Aw-efMJ, however no answer was provided. It would really be nice if someone could help me with this paradoxical situation. Thanks in advance,
Patrick

[Adrian Klaver]

On 09/23/2014 04:04 AM, Patrick Leinenkugel wrote:
> Hi,
>
> I am using Spyder within python (x,y) (Version 2.7.6.1) on Windows.
> Doing the following causes a MemoryError in my Spyder Python IDE:
>
> |>>> from numpyimport *
>>>> a_flt= ones((7000,7000), dtype=float64)+4

>>>> b_flt= ones((7000,7000), dtype=float64)+1

> Traceback (most recent call last):

> File "<stdin>", line1, in <module>

> MemoryError
>>>>
>
> |
>
> This is weird, since the memory usage in the statusbar of Spyder shows
> that only approx. 25% of my memory is used. Furthermore, when generating
> even a higher number of these 7000*7000 arrays in the standard Python
> IDE GUI, everything works fine.
>
> |>>> from numpyimport *
>>>> a_flt= ones((7000,7000), dtype=float64)+4
>>>> b_flt= ones((7000,7000), dtype=float64)+1
>>>> c_flt= ones((7000,7000), dtype=float64)+1
>>>> d_flt= ones((7000,7000), dtype=float64)+1

>>>> e_flt= ones((7000,7000), dtype=float64)+1

>
> |
>
> Even with 5 floating point arrays created, memory requirements only
> amount to about a third of my total memory of 16GB. So this cannot be a

> real memory problem. Thenupdated to Spyder 2.3.1 using Python 2.7.6

> 32bits, Qt 4.8.4, PyQt4 (APIv2) 4.9.6 on Windows and the same Memory
> Error results alreay when trying to create the first array with
> 7000*7000 float elements (a_flt)! Furthermore, the same error is
> reproducable in the Spyder IDE on the computer of my colleague, so it is
> not an error related to my specific computer settings.
>
> I also found a similiar question on this issue in this spyder google
> group
> https://groups.google.com/forum/#!msg/spyderlib/qE9tiriT91s/0x3s2Aw-efMJ

> <https://groups.google.com/forum/#%21msg/spyderlib/qE9tiriT91s/0x3s2Aw-efMJ>,

> however no answer was provided. It would really be nice if someone could
> help me with this paradoxical situation. Thanks in advance,

Some more information is probably in order.

So:

1) Where are you running the above code, the Python or IPython console?

2) If you are using IPython, what version?
2a) What happens if you run in a stand alone IPython console?

3) What is the 'standard' Python IDE GUI you refer to?

> Patrick
>
> --

--
Adrian Klaver
adrian...@aklaver.com

[Patrick Leinenkugel]

Hi Adrian,

thanks for helping me out. To answer your questions:

1) On the Python console

2) I am not familiar with IPython, yet.

3) The Python IDLE that is included within every pure Python distribution, without installing other applications.

Cheers,

Patrick

[Adrian Klaver]

On 09/23/2014 11:25 AM, Patrick Leinenkugel wrote:
> Hi Adrian,
>
> thanks for helping me out. To answer your questions:
>
> 1) On the Python console

So are you using the same Python when you are in Spyder as when you are
in the IDLE console?

Also are the numpy versions the same?

The reason I ask is that it is fairly common on Windows to get multiple
installs of Python interfering with each other. In that vein how did you
do this from your first post?:

"Then updated to Spyder 2.3.1 using Python 2.7.6 32bits, Qt 4.8.4, PyQt4
(APIv2) 4.9.6"

> 2) I am not familiar with IPython, yet.
> 3) The Python IDLE that is included within every pure Python
> distribution, without installing other applications.

Actually no, only in the Windows and OS X installers.

>
> Cheers,
> Patrick
>

--
Adrian Klaver
adrian...@aklaver.com

[Patrick Leinenkugel]

Hi Adrian,

I use Python(x,y)  Version 2.7.6.1 that includes Spyder 2.5.5 and Python 2.7.6 32bits. I then realized that there was a newer version of Spyder and updated to Spyder 2.3.1 using the executable offered on the Spyder website. But this could not be the reason, since the error also appeared when using the Spyder Version that came together with Python(y,x). I did not update Python and I only have Python 2.7.6 installed on my computer. Thus it is definitely the same Python version and numpy module that I use in IDLE and Spyder. 

Cheers,

Patrick

[stone...@gmail.com]

16Gb of memory and a 32bit python distribution may be an not well tested combination.
Shouldn't you try with a 64bit distribution ?

[Adrian Klaver]

On 09/23/2014 01:35 PM, Patrick Leinenkugel wrote:
> Hi Adrian,
>

> I use Python(x,y) Version 2.7.6.1 that includes Spyder 2.5.5 and Python
> 2.7.6 32bits. I then realized that there was a newer version of Spyder
> and updated to Spyder 2.3.1 using the executable offered on the Spyder
> website. But this could not be the reason, since the error also appeared
> when using the Spyder Version that came together with Python(y,x). I did
> not update Python and I only have Python 2.7.6 installed on my computer.
> Thus it is definitely the same Python version and numpy module that I
> use in IDLE and Spyder.

Alright, trying to get a handle on the moving parts. I tried your code
here on Linux using Spyder 2.3.1 and did not get a MemoryError.

What I did see:

1) Using the IPython console I ran your code as is with no issues.

2) In the Python console, when I ran your code as is I found that it did
not actually run. So when I did:

from numpy import *
a_flt = ones((7000,7000), dtype=float64)+4

and then

a_flt <Enter> I got:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>

NameError: name 'a_flt' is not defined

If I did

import numpy
a_flt = numpy.ones((7000, 7000),dtype=numpy.float64)+4

then everything worked. I could create multiple *_flt objects with no
problems.

I suspect there is an import conflict going on in the Python console
versus the IPython console. To get a handle on this I would suggest:

1) Try running the code in the IPython console. It should be there, as
PythonXY installs IPython

2) Look in Preferences --> Console --> Advanced Settings and see what
the console is using in the PYTHONSTARTUP replacement section.

[Joseph Martinot-Lagarde]

On the contrary, this is a well tested combination and it is know to work badly.

When using 32bit python, you're limited to around 2GB of memory, whatever your total amount of RAM. If you want to use more memory, you have to install a 64bit distribution of python. Python(x,y) is 32bits only for now, but you can use anaconda instead (or winpython but it seems that it's less up to date).

[Adrian Klaver]

On 09/24/2014 02:25 AM, Joseph Martinot-Lagarde wrote:
> On the contrary, this is a well tested combination and it is know to
> work badly.
>
> When using 32bit python, you're limited to around 2GB of memory,
> whatever your total amount of RAM. If you want to use more memory, you
> have to install a 64bit distribution of python. Python(x,y) is 32bits
> only for now, but you can use anaconda instead (or winpython but it
> seems that it's less up to date).

But this does not answer the question of why the same code on the same
machine works with IDLE?

>
> Le mercredi 24 septembre 2014 00:10:34 UTC+2, stone...@gmail.com a écrit :

--
Adrian Klaver
adrian...@aklaver.com

[Patrick Leinenkugel]

Hi Adrian,

I meanwhile figured out that it was not a problem of Spyder. I did not expext that the 32-bit version was limited to so little processing memory.  I now installed a 64-bit version of WinPython including a 64-bit version of Spyder and everything works fine! Thanks a lot!

[Carlos Córdoba]

Hi,

That was what I was about to say. This is not a Spyder problem because I got the same MemoryError in all programs I tried (IDLE, IPython terminal and Qt console).

Also if you changed float64 for float32 in your dtypes, everything worked fine, proving Joseph point. In any case, I'm glad that you solved it.

Cheers,
Carlos

El 25/09/14 a las #4, Patrick Leinenkugel escribió:

--
You received this message because you are subscribed to the Google Groups "spyder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spyderlib+...@googlegroups.com.
To post to this group, send email to spyd...@googlegroups.com.
Visit this group at http://groups.google.com/group/spyderlib.
For more options, visit https://groups.google.com/d/optout.

[Adrian Klaver]

On 09/25/2014 01:01 AM, Patrick Leinenkugel wrote:
> Hi Adrian,
>
> I meanwhile figured out that it was not a problem of Spyder. I did not
> expext that the 32-bit version was limited to so little processing
> memory. I now installed a 64-bit version of WinPython including a 64-bit
> version of Spyder and everything works fine! Thanks a lot!
>

Glad it is working. However one thing still confuses me from your
original post:

"Furthermore, when generating even a higher number of these 7000*7000
arrays in the standard Python IDE GUI, everything works fine."

If it was not a Spyder problem but a 32/64bit issue how could that be?


--
Adrian Klaver
adrian...@aklaver.com

[Joseph Martinot-Lagarde]

I don't know exactly why, but here are some hints:

- A 7000*7000 array of float64 takes 7000 * 7000 * 8 / 1024 / 1024 = 373 Mo of memory (headers are negligible)
- Each line where the arrays are created creates an additional temporary array (because of the addition), so in spyder the problem arises at 373 * 3 = 1119 Mo of memory. This has to be added to spyder's own memory which I guess is higher than IDLE. Does the variable explorer make a copy of the data, since it pickles then unpickles the arrays ? That might be considered a bug.
- In IDLE, the 5 arrays + 1 temporary represent 2238Mo. I don't understand why it didn't break. Is is certain that it is from the same python install ?

[Adrian Klaver]

On 09/26/2014 01:31 AM, Joseph Martinot-Lagarde wrote:
> I don't know exactly why, but here are some hints:
>
> - A 7000*7000 array of float64 takes 7000 * 7000 * 8 / 1024 / 1024 = 373
> Mo of memory (headers are negligible)
> - Each line where the arrays are created creates an additional temporary
> array (because of the addition), so in spyder the problem arises at 373
> * 3 = 1119 Mo of memory. This has to be added to spyder's own memory
> which I guess is higher than IDLE. Does the variable explorer make a
> copy of the data, since it pickles then unpickles the arrays ? That
> might be considered a bug.
> - In IDLE, the 5 arrays + 1 temporary represent 2238Mo. I don't
> understand why it didn't break. Is is certain that it is from the same
> python install ?

That is what I am trying to get a handle on. If it is just a straight
memory access problem due to memory restrictions in 32 bit Windows then
the second case should fail if the first does. I can understand moving
up to 64bit, but the question is whether that is really a solution or a
way of hiding a problem? I do not have a Windows installation to play
with, so I am not able to investigate on this end, which is why I have
been asking the questions.

According to the OP there is only one version of Python installed:

" ...I only have Python 2.7.6 installed on my computer. Thus it is

definitely the same Python version and numpy module that I use in IDLE
and Spyder. "

>

> Le jeudi 25 septembre 2014 19:45:55 UTC+2, Adrian Klaver a écrit :

--
Adrian Klaver
adrian...@aklaver.com

[olli.s...@gmail.com]

> That is what I am trying to get a handle on. If it is just a straight
> memory access problem due to memory restrictions in 32 bit Windows then
> the second case should fail if the first does.

Just guessing, but maybe this is caused by fragmentation of the 32-bit address space. Perhaps memory allocation in some cases doesn't find a contiguous 373 M block even though adding up all the smaller free blocks would show more free memory.

As another data point, I have Python 3.4.1 |Anaconda 2.0.1 (32-bit)| (default, Jun 11 2014, 17:29:32) [MSC v.1600 32 bit (Intel)] running on 64-bit Windows 7, and it consistently gives the memory error on "b_flt" in Spyder, IPython, as well as starting python.exe from the Windows command prompt.

Cheers,
   Olli

[Patrick Leinenkugel]

Yes, I definitely used the same python distribution. 

What perhaps is of interest to you is that I experienced different tolerances in the Python IDLE of producing these arrays (3-5 arrays), while with Spyder, it was either one or a maximum of two arrays, that could be created. So in all cases, I was able to create a higher number of arrays with IDLE than with Spyder. Now, with the 64-bit version, I have no problems creating a large number of arrays also in Spyder.

Cheers,

Patrick

[Jonno]

I don't know if it's the same issue but I'm also seeing a memory error only in Spyder and I too have 32 bit Spyder & Python via Anaconda 2.0.1 on Win7 64 bit. I don't see the issue outside of Spyder but using the same Python interpreter.

Interestingly this issue doesn't exist for a different machine running: Spyder 2.2.0 / Python 2.7.5 (32 bit) / Win7 (64 bit) with 2GB RAM.

[Jonno]

I filed a ticket here: https://code.google.com/p/spyderlib/issues/detail?can=2&q=1999&colspec=ID%20MS%20Stars%20Priority%20Modified%20Cat%20Type%20Status%20Owner%20Summary&id=1999

[Adrian Klaver]

On 10/06/2014 08:51 AM, Patrick Leinenkugel wrote:
> Yes, I definitely used the same python distribution.
> What perhaps is of interest to you is that I experienced different
> tolerances in the Python IDLE of producing these arrays (3-5 arrays),
> while with Spyder, it was either one or a maximum of two arrays, that
> could be created. So in all cases, I was able to create a higher number
> of arrays with IDLE than with Spyder. Now, with the 64-bit version, I
> have no problems creating a large number of arrays also in Spyder.

My guess is that this is related to this thread:

https://groups.google.com/forum/#!topic/spyderlib/PoOig4rf43c

You might to file an issue:

https://code.google.com/p/spyderlib/issues/list

[Gonzalo A. PEÑA CASTELLANOS]

Maybe a  bit off topic but still something that could be potentially interesting and useful is that we could/should start compiling all these "bugs" and turn them into unit tests so that future proofing is handled.

Not sure if for a IDE like spyder Travis (now that the move to github seems inminent) could provide some sort of testing in this direction.

I will start scrapping all these "bugs" in the wiki for the moment.

Cheers

[olli.s...@gmail.com]

Hi,

A look at memory usage in the posted test cases indicates that address space fragmentation is a major factor at least in my setup
Python 3.4.1 |Anaconda 2.1.0 (32-bit)| (default, Jun 11 2014, 17:29:32) [MSC v.1600 32 bit (Intel)] running on 64 bit Windows 7.
Such fragmention is described in, e.g., http://www.mgroeber.de/misc/windows_heap.html

The statistics in the following listing are from VMMap.exe from Microsoft Sysinternals.

Spyder started
  - 1734 MB free address space in 91 blocks, largest block 883 MB

In [1]: from numpy  import *
  - 1741 MB free address space in 86 blocks, largest block 883 MB

In [2]: a_flt = ones((7000,7000), dtype=float64)+4
  - 1358 MB free address space in 87 blocks, largest block 382 MB

In [3]: b_flt = ones((7000,7000), dtype=float64)+1
  - Raises MemoryError
  - 1358 MB free address space in 87 blocks, largest block 382 MB
  - The next largest free blocks: 207 MB, 173 MB, 117 MB, 115 MB, ...

In [4]: b_flt = ones((7000,7000), dtype=float64)
  - 975 MB free address space in 86 blocks, largest block 207 MB

In [5]: b_flt += 1
  - 975 MB free address space in 86 blocks, largest block 207 MB

The numbers vary slightly in consecutive runs. The next case:

Spyder started
  - 1741 MB free address space in 81 blocks, largest block 880 MB

In [1]: import random
  - 1741 MB free address space in 81 blocks, largest block 880 MB

In [2]: import struct
  - 1741 MB free address space in 81 blocks, largest block 880 MB

In [3]: intlist = [int(100*random.random()) for _ in range(33000000)]
  - 1612 MB free address space in 81 blocks, largest block 880 MB

In [4]: buf = struct.pack('%sh' %len(intlist), *intlist)
  - 1450 MB free address space in 81 blocks, largest block 784 MB
  - The next largest free blocks: 173 MB, 117 MB, 78 MB, ...

VMMap.exe (View --> Fragmentation View) shows that DLL's are loaded in a pretty fragmented pattern. After running the second case there are 1171 separate blocks belonging to executable images. The most fragmented module is PyQt4\QtGui.pyd: 69 separate blocks.

The main difference I see between running in Spyder and otherwise is the number of DLL's loaded. I don't know if anything can be done about that, except for loading fewer modules (and of course, switching to 64 bits and/or to Linux). A similar issue has been discussed in the Python bug tracker http://bugs.python.org/issue19246. It was closed with a comment:

"So the issue comes the Windows heap allocator. I don't see any obvious improvment that Python can do to improve the memory usage. I close the issue."

Cheers,
   Olli