[spyder] Spyder hangs in both Windows and Linux during python program execution

[BDL]

OK, this is a weird one. I have some simulation code that works fine
in ipython.0.10 (Ubuntu9.10-64). It would be nice to be able to use
spyder (or pythonxy on windows) and execute the code. However, in
both cases spyder hangs.

Under win32 I get a fault message before the window becomes non-
responsive. (I'll get back to this.) I have to kill the non-
responsive window.

Under Ubuntu9.10-64, the window becomes non responsive (after running
a few minutes) and the CPU usage goes to 100% on one of my cores.
After waiting a few minutes, I have to kill this window also.

So, I'm looking for a few tips on diagnosis. The program in question
is rather large, so it would not be easy to post, nor easy to
"sanitize". I am using scipy and numpy in the simulations and doing a
lot of matrix manipulations, including svd, and lstsq.

While under WinXP32, I get the following message :

**On entry to ZGELSS parameter number 12 had an illegal value

Then ipython is exited. Sometimes the console2 is dead after that.

Now ZGELSS is in the LAPACK library, IIRC. I never get this problem
when running ipython from a terminal in Ubuntu. I was astonished to
see this error in pythonxy. And even more astonished to see it in
spyder on Ubuntu.

Have any ideas? I tried posting to scipy list, ahem, they answered
with a rather stony silence. Perhaps they were quick to blame spyder,
I don't know. I do know I have not got a single response to my post.
Maybe it was ill-framed and I have to try again...

I think spyder is tres cool. However, I'd like to be able to use it
for my code... And for others to "look and use my code"

Bruce
Ubuntu 9.10 x86-64
WinXP 32 plus what ever service pack...

--
You received this message because you are subscribed to the Google Groups "spyder" group.
To post to this group, send email to spyd...@googlegroups.com.
To unsubscribe from this group, send email to spyderlib+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/spyderlib?hl=en.

[Pierre Raybaut]

On Apr 30, 11:35 pm, BDL <bdlab...@gmail.com> wrote:
> OK, this is a weird one.  I have some simulation code that works fine
> in ipython.0.10 (Ubuntu9.10-64).  It would be nice to be able to use
> spyder (or pythonxy on windows) and execute the code.  However, in
> both cases spyder hangs.

To sum-up, your simulation code works fine only with IPython 0.10 on
Ubuntu.
And it does not work with the following configurations:
* IPython 0.10 on Windows with Console2
* Spyder on both Windows and Ubuntu
Have you tried the following configurations?
* IPython 0.10 on Windows with the system command window (cmd.exe)
* Python interpreter (just python.exe) on Windows with the system
command window
* Python interpreter on Ubuntu
* Spyder with the external console on Windows and Ubuntu (see
below)

> Under win32 I get a fault message before the window becomes non-
> responsive.  (I'll get back to this.)   I have to kill the non-
> responsive window.
>
> Under Ubuntu9.10-64, the window becomes non responsive (after running
> a few minutes) and the CPU usage goes to 100% on one of my cores.
> After waiting a few minutes, I have to kill this window also.
>
> So, I'm looking for a few tips on diagnosis.  The program in question
> is rather large, so it would not be easy to post, nor easy to
> "sanitize".  I am using scipy and numpy in the simulations and doing a
> lot of matrix manipulations, including svd, and lstsq.
>
> While under WinXP32, I get the following message :
>
> **On entry to ZGELSS parameter number 12 had an illegal value
>
> Then ipython is exited.  Sometimes the console2 is dead after that.
>
> Now ZGELSS is in the LAPACK library, IIRC.  I never get this problem
> when running ipython from a terminal in Ubuntu.  I was astonished to
> see this error in pythonxy.  And even more astonished to see it in
> spyder on Ubuntu.
>
> Have any ideas?  I tried posting to scipy list, ahem, they answered
> with a rather stony silence.  Perhaps they were quick to blame spyder,
> I don't know.  I do know I have not got a single response to my post.
> Maybe it was ill-framed and I have to try again...

I'm sure it was just a bad timing and that they would not blame
Spyder ;-)

> I think spyder is tres cool.  However, I'd like to be able to use it
> for my code...  And for others to "look and use my code"

Have you tried running your code in Spyder's external console
(separate process) instead of the interactive console? Behaviour
should be different as the interactive console is running in the same
thread/process as Spyder itself.

If you are not familiar with the external console features/
limitations, know that the only difference with the interactive
console is a limitation/bug of the PyQt Input Hook on win32 which
prevent it to be fully functionnal for interactive plotting with
matplotlib (or similar GUI interactive manipulations). Except this
PyQt bug, you may do exactly the same things as in the interactive
console -- try opening a new interpreter in the external console,
you'll see that you may even a variable explorer (similar to
interactive console's workspace).

Pierre

[BDL]

More results:

Win32
cmd window: ipython --> fails with ZGELSS error
python myfile.py --> fails with ZGELSS error
Spyder: External console --> fails with ZGELSS error, but Spyder is
functional
Internal/interactive --> fails with Spyder Window abruptly
closing!

Ubuntu 10.04 (upgraded from 9.10)
from xterm: ipython --> pass. python myfile.py --> pass
from Spyder: external console --> pass! but slow update on console
window. After figure delete, takes a long time for Spyder to respond,
on the order of minutes.
internal/interactive console --> pass, but after deleting
figure, the console takes minutes to "pay attention to the window"
internal console --> hangs for long times

At least for WinXP32, the fault appears to be (I think) within lstsq.
I believe I have tracked it down to there. The lstsq is inside of a
try block, but no exceptions seem to be caught.

S1,S2 = getS1S2( S, m, L )
print 'I got here...'
try:
phi = lstsq( S1, S2 )
except ValueError:
lstsqfailcount += 1
print 'lstsq failed to converge'
return array([-1000000000])
except LinAlgError:
lstsqfailcount += 1
print 'Linear Algebra Error'
return array([-1000000000])
except:
print 'Unexpected error:',sys.exc_info()[]
else:
""" continue processing ... """
print 'Finished the lstsq function'

For Ubuntu, I'm somewhat ready to believe that lstsq is also to
blame. I see it struggling to get a solution when the SNR is low. I
guess that is proper, but I am wondering if the function should have a
timeout, or some other escape hatch.


scipy.linalg.lstsq version 0.7.1 has a bug. I posted it to scipy-
users today. It faults rather ungracefully in my CME under Win32 XP
using the Python(x,y) tools. However, even if I use numpy.linalg.lstsq
instead, Spyder is taking a lot of resources and "dying". I'm testing
two versions of my original code running under SciTE on Windows. The
first version passed using numpy.linalg.lstsq. I was somewhat
surprised to see that SciTE running my python code was taking 15% of
one core, vs. Spyder taking 80%+ of a core. I guess that is the price
one pays for the nice GUI. Or am I doing something wrong?

I am running the second version (much more stressful case) under SciTE
as I am typing.

[Pierre Raybaut]

For a pure computing algorithm, I don't see any reason why Spyder
would take much CPU usage than SciTE. Besides, if you are using the
external console, this is exactly the same as typing "python
yourscript.py" in a system command window. So this result is very
surprising.

And regarding the price to pay for the nice GUI, it should be relevant
only if you are plotting a new figure every ~10ms because of the
matplotlib/Spyder integration interface and still I'm really not sure
it's consuming more CPU than the stand-alone matplotlib figures.

However if your script is printing a lot of text to screen, Spyder
should consume a lot more CPU than SciTE because of some high-level
features of the console widget (note: performances should be better
with the latest v1.1.0 beta release since console widgets are now
based on QPlainTextEdit instead of QTextEdit -- and, in the near
future, it should be even better because I will introduce a limitation
for the console buffer size which is currently unlimited, i.e. one has
to clear the console to start from scratch).

[BDL]

Ah, that makes some sense. I am writing a bunch o'text to the
screen. It is mostly diagnostic stuff, I have not stripped out the
gratuitous screen writes yet.

Under SciTE, the "stress" case runs fine. However, using Spyder, it
chokes. I can try editing out more of the print statements to see if
it matters.
Without getting rid of the screen writes it chokes after running ~ 15
minutes. I then 'lock' the core and Spyder becomes unresponsive.

FWIW, on my laptop, this routine takes ~1200 sec to run (under
SciTE). (On my server, it takes 240 sec. Guess where I do most of my
work:) ) However, if I want others to be able to run my code, it
would be nice to be able to use the capabilities of Spyder.

Later on, I'll try getting rid of 90% of the screen writes. I still
need to keep 10% to tell where I am in the code...

Bruce

[Pierre Raybaut]

Two things may slow down console widget performances:
* the text print rate (1 line per second is ok, 1 line per
millisecond should be hard to follow for the widget)
* the total amount of text displayed by the widget (this should be
solved very soon with the next v1.1.0 beta release)

For the first point (the rate), you may try doing 'dir /s' in a
terminal opened in Spyder on win32 in your system root and watch how
much CPU it's consuming.

Pierre

> You received this message because...
>
> read more »