python crash on windows but not on linux
hjebbers
I am running an EDI translator, and doing stress tests.
When processing a test with a (relatively) big EDI file(s) on
windowsXP I get a crash:
'sorry for the inconvenience' etc (so no clues about what is
causing the problem)
This happens with python 2.4, 2.5, 2.6
It does not happen in the same setup&tests on linux. (while the linux
machine has only half of the RAM of the windows machine).
I am quite sure it is not some external module; there's no GUI (apart
from 'print'), no network connections.
Actually, the crash is not predictable....the crash occurs most of the
time...but at different points.
So basically I am wondering how to proceed.
Is it related to a memory problem? (which does not occur on Linux)
any pointer is very welcome,
henk-jan
M3RT
The problem may be related to how you treat the EDI file or lets say
DATA. Also your coding style is important. Can you provide more info?
hjebbers
Yes, a whole lot more; but I do not want to bother you with that now.
I was just wondering if it is possible that the same setup gives a
CRASH! on windows and just works on linux.
(where is the bug?)
kind regards, henk-jan
Peter Otten
In the platform-specific code ;)
Even on alt.haruspicy they cannot do much without a liver now and then...
hjebbers
if it is in the platform-specific code should I make this into a
bugreport?
(because the crash does not appear at the same place in my code (when
doing the same test-run) it will be quite hard to point down....)
henk-jan
hjebbers
if it is in the platform-specific code should I make this into a
hjebbers
if it is in the platform-specific code should I make this into a
Jerry Hill
> To all,
> I am running an EDI translator, and doing stress tests.
> When processing a test with a (relatively) big EDI file(s) on
> windowsXP I get a crash:
> 'sorry for the inconvenience' etc (so no clues about what is
> causing the problem)
>
You need to give us more information if we're going to be able to
help. At the very least, you'll need to copy & paste the actual error
message. It would be even better if you could show us some of your
code, and better yet if you could give us a small bit of code that is
self contained and reproduces the problem you're experiencing.
--
Jerry
hjebbers
the error is a windows thing, I can make a screenshot of it, but I can
not copy/paste text.
how do I upload a png-file?
problem is that the same error happens over and over (I can reproduce
it), but not at the same place (the is a logging in the application so
that is quite easy to see.)
but ....I can show you my code. it's an open source EDI application.
but it is not a small bit of code....
I am more than will to show you how to reproduce the error.
kind regards,
henk-jan
hjebbers
the error is a windows thing, I can make a screenshot of it, but I can
Carl Banks
People coming here ask for help will vary in the amount of detail
given, but I've rarely seen anyone as reluctant as you. It's like
walking into an auto repair shop and asking the mechanic what's wrong
with your car by trying to imitate the noise it makes.
If your code is very largre, you're not going to get out of this
without doing some of your own legwork. I'm sorry.
Do this: Take your code, make a copy of it. Start removing code from
the copy (in a controlled way) until the problem disappears. When you
remove code don't worry about whether it produces anything useful, the
object is to try to identify what's causing the error. Whatever code
you removed when the error disappears should give you a clue what's
causing it. Then, if need be, you can come back and post details. If
you manage to get the program to a small size without eliminating the
problem, you can post it here.
Carl Banks
Terry Reedy
Are you using a 3rd-party extension (compiled-C) module? If so, I would
suspect something platform specific in that.
Emile van Sebille
> To all,
> I am running an EDI translator,
... let's say bots :)
> and doing stress tests.
> When processing a test with a (relatively) big EDI file(s) on
> windowsXP I get a crash:
> 'sorry for the inconvenience' etc (so no clues about what is
> causing the problem)
... and it's running directly under python...
I use textpad. It allows me to launch a script from a command window
under its control, which when python subsequently crashes, commonly
leaves the traceback in the textpad launched command window.
Perhaps that'll help?
Emile
--also on the bots list and will be digging in seriously over the next
several weeks
hjebbers
well, there is a connection to a database.
but I tested this with connections to SQLite, MySQL, PostGreSQL, and
the crash keeps coming.
So I think it is not a 3rd party module....
And all else is is pure python.
kind regards, Henk-Jan
hjebbers
yes, we are talking bots.
I run the bots engine from the commandline.
But... there is no python traceback. Python crashed 'hard'.
or have you something else in mind?
kind regards,henk-jan
hjebbers
hjebbers
hjebbers
occurs during stress tests.
I am more than willing to show how to reproduce the crash.
> If your code is very largre, you're not going to get out of this
> without doing some of your own legwork. I'm sorry.
I am more than willing to do this, of course.
>
> Do this: Take your code, make a copy of it. Start removing code from
> the copy (in a controlled way) until the problem disappears. When you
> remove code don't worry about whether it produces anything useful, the
> object is to try to identify what's causing the error. Whatever code
> you removed when the error disappears should give you a clue what's
> causing it. Then, if need be, you can come back and post details. If
> you manage to get the program to a small size without eliminating the
> problem, you can post it here.
OK, I know how to trace a bug.
As I said, these are stress tests, in this case the input files are
MUCH larger than usualy.
Other stress tests run thousands of input file of regular to big
size.....
This all runs.
It does run OK on Linux (so am I looking for a bug in my program??).
The crash pops up at different places...this makes the procedure you
suggest quite awkward.
It is a hard crash of python. There is no traceback or something:
Python dies. Python is no more. It's stone dead. It has ceased to be.
It's a stiff. It's kicked the bucket etc
kind regards,
henk-jan
Aahz
hjebbers <hjeb...@gmail.com> wrote:
>
>the error is a windows thing, I can make a screenshot of it, but I can
>not copy/paste text.
In that case, you need to -- very carefully -- make sure you transcribe
exactly the message that you see on the screen.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
"At Resolver we've found it useful to short-circuit any doubt and just
refer to comments in code as 'lies'. :-)"
Terry Reedy
> In article<34fcf680-1aa4-4835...@q16g2000yqq.googlegroups.com>,
> hjebbers<hjeb...@gmail.com> wrote:
>>
>> the error is a windows thing, I can make a screenshot of it, but I can
>> not copy/paste text.
I think I know what box you mean. I believe this happened about 18
months ago with IDLE before 3.0 was released. Something to do with MS
VC, which is different from gcc used on *nix.
hjebbers
> In article <34fcf680-1aa4-4835-9eba-3db3249f3...@q16g2000yqq.googlegroups.com>,
>
> hjebbers <hjebb...@gmail.com> wrote:
>
> >the error is a windows thing, I can make a screenshot of it, but I can
> >not copy/paste text.
>
> In that case, you need to -- very carefully -- make sure you transcribe
> exactly the message that you see on the screen.
> --
>
> "At Resolver we've found it useful to short-circuit any doubt and just
> refer to comments in code as 'lies'. :-)"
The message on the screen is (I typed it over):
**************************************************************************************
python.exe
python.exe has encountered a problem and needs to close.
We are sorry for the inconvenience.
If you were in the middle of something, the information you were
working on might be
lost
For more information about the error, click here.
<close-
button>
***************************************************************************************
the information about the error is a windows dump.
kind regards,
henk-jan
Dave Angel
> On Feb 12, 3:41 am, a...@pythoncraft.com (Aahz) wrote:
>
>> In article <34fcf680-1aa4-4835-9eba-3db3249f3...@q16g2000yqq.googlegroups.com>,
>>
>> hjebbers <hjebb...@gmail.com> wrote:
>>
>>
>>> the error is a windows thing, I can make a screenshot of it, but I can
>>> not copy/paste text.
>>>
>> In that case, you need to -- very carefully -- make sure you transcribe
>> exactly the message that you see on the screen.
>> --
>> Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/
>>
>> "At Resolver we've found it useful to short-circuit any doubt and just
>> refer to comments in code as 'lies'. :-)"
>>
>
> The message on the screen is (I typed it over):
>
You shouldn't ever have to retype the message. Just paste from the DOS
box (or "Command Prompt"). It's easy using QuickEdit mode.
You select a rectangle of the text by selecting with the mouse, then
using right-click to put it in the clipboard.
If that doesn't work, you haven't yet enabled QuickEdit mode. Use
right-click on the DOS box's title bar. Choose Properties. Select the
first tab, which is Options.
You should then see a selection box called "Quick Edit Mode". Turn it
on, and you won't have this problem again.
DaveA
hjebbers
Hi Dave,
thanks.
but the message is not in the DOS box (or "Command Prompt").
I do run the program in a DOS box (or "Command Prompt"), but the error
is a windows dialog box.
I can not select in this dialog box.
kind regards,
henk-jan
Steve Holden
> hjebbers wrote:
>> On Feb 12, 3:41 am, a...@pythoncraft.com (Aahz) wrote:
>>
>>> In article
>>> <34fcf680-1aa4-4835-9eba-3db3249f3...@q16g2000yqq.googlegroups.com>,
>>>
>>> hjebbers <hjebb...@gmail.com> wrote:
>>>
>>>
>>>> the error is a windows thing, I can make a screenshot of it, but I can
>>>> not copy/paste text.
>>>>
>>> In that case, you need to -- very carefully -- make sure you transcribe
>>> exactly the message that you see on the screen.
>>> --
>>> Aahz (a...@pythoncraft.com) <*>
>>> http://www.pythoncraft.com/
>>>
>>> "At Resolver we've found it useful to short-circuit any doubt and
>>> just refer to comments in code as 'lies'. :-)"
>>>
>>
>> The message on the screen is (I typed it over):
>>
>
> You shouldn't ever have to retype the message. Just paste from the DOS
> box (or "Command Prompt"). It's easy using QuickEdit mode.
>
> You select a rectangle of the text by selecting with the mouse, then
> using right-click to put it in the clipboard.
>
> If that doesn't work, you haven't yet enabled QuickEdit mode. Use
> right-click on the DOS box's title bar. Choose Properties. Select the
> first tab, which is Options.
>
> You should then see a selection box called "Quick Edit Mode". Turn it
> on, and you won't have this problem again.
>
> DaveA
>
I think you miss the point: this message is being displayed in a Windows
error message dialog box, not in the command window.
regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/
Steve Holden
> hjebbers wrote:
>>
>>> In article
>>> <34fcf680-1aa4-4835-9eba-3db3249f3...@q16g2000yqq.googlegroups.com>,
>>>
>>> hjebbers <hjebb...@gmail.com> wrote:
>>>
>>>
>>>> the error is a windows thing, I can make a screenshot of it, but I can
>>>> not copy/paste text.
>>>>
>>> In that case, you need to -- very carefully -- make sure you transcribe
>>> exactly the message that you see on the screen.
>>> --
>>> Aahz (a...@pythoncraft.com) <*>
>>> http://www.pythoncraft.com/
>>>
>>> "At Resolver we've found it useful to short-circuit any doubt and
>>> just refer to comments in code as 'lies'. :-)"
>>>
>>
>> The message on the screen is (I typed it over):
>>
Christian Heimes
> The message on the screen is (I typed it over):
>
> **************************************************************************************
> python.exe
>
> python.exe has encountered a problem and needs to close.
> We are sorry for the inconvenience.
>
> If you were in the middle of something, the information you were
> working on might be
> lost
>
> For more information about the error, click here.
>
> <close-
> button>
> ***************************************************************************************
> the information about the error is a windows dump.
I suggest you install some sort of debugger on your system (e.g. Visual
Studio) and do a post mortem analysis of your stack trace.
Christian
Peter Otten
> On Feb 11, 7:01 pm, Peter Otten <__pete...@web.de> wrote:
>> hjebbers wrote:
>> > On Feb 11, 5:45 pm, M3RT <mgul...@gmail.com> wrote:
>> >> The problem may be related to how you treat the EDI file or lets say
>> >> DATA. Also your coding style is important. Can you provide more info?
>>
>> > Yes, a whole lot more; but I do not want to bother you with that now.
>> > I was just wondering if it is possible that the same setup gives a
>> > CRASH! on windows and just works on linux.
>> > (where is the bug?)
>>
>> In the platform-specific code ;)
>>
>> Even on alt.haruspicy they cannot do much without a liver now and then...
>
> if it is in the platform-specific code should I make this into a
> bugreport?
A this stage a bug report would be useless. The details you have provided so
far don't give anyone without physical access to your machine a chance to
reproduce the error.
> (because the crash does not appear at the same place in my code (when
> doing the same test-run) it will be quite hard to point down....)
Indeed. The culprit may be an extension written in C that is overwriting
memory still in use by other parts of your program.
Peter
Bruno Desthuilliers
(snip)
> Even on alt.haruspicy they cannot do much without a liver now and then...
Muhahahahaha !-)
+1 QOTW
Dave Angel
a pain. Sorry my response wasn't applicable.
DaveA
hjebbers
No C extensions used. Definitely not.
As these crashes occur during stress tests, I go to the limits of the
memory available.
What strikes me is:
1. the crash on windows, but linux works OK (same test sets)
2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
much).
kind regards, Henk-jan
Rob Williscroft
3348b7...@u9g2000yqb.googlegroups.com in comp.lang.python:
> the information about the error is a windows dump.
This may help:
# http://msdn.microsoft.com/en-us/library/ms680621(VS.85).aspx
SEM_FAILCRITICALERRORS = 1
SEM_NOALIGNMENTFAULTEXCEPT = 4
SEM_NOGPFAULTERRORBOX = 4
SEM_NOOPENFILEERRORBOX = 8
import ctypes
from ctypes.wintypes import UINT
SetErrorMode = ctypes.windll.kernel32.SetErrorMode
SetErrorMode.restype = UINT
SetErrorMode.argtypes = ( UINT,)
Then putting:
SetErrorMode( SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX )
at the start of you programme, should stop the "Critical Error" dialog
box you are seeing and you may get a chance to see a traceback.
Rob.
Peter Otten
Random thoughts:
If there were not enough memory Python should give a MemoryError rather than
segfault.
Do both systems use the same memory layout (i. e. 32 or 64bit OS, 16 or 32
bit unicode)?
Try an alpha of Python 2.7. If you're lucky "your" bug was found 'n' fixed
independently.
Peter
hjebbers
> hjebbers wrote in news:2864756a-292b-4138-abfd-
>
> > the information about the error is a windows dump.
>
> This may help:
>
>
> SEM_FAILCRITICALERRORS = 1
> SEM_NOALIGNMENTFAULTEXCEPT = 4
> SEM_NOGPFAULTERRORBOX = 4
> SEM_NOOPENFILEERRORBOX = 8
>
> import ctypes
> from ctypes.wintypes import UINT
>
> SetErrorMode = ctypes.windll.kernel32.SetErrorMode
> SetErrorMode.restype = UINT
> SetErrorMode.argtypes = ( UINT,)
>
> Then putting:
>
> SetErrorMode( SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX )
>
> at the start of you programme, should stop the "Critical Error" dialog
> box you are seeing and you may get a chance to see a traceback.
>
> Rob.
Hi Rob,
thanks.
I tried this, but did not work for me.
henk-jan
hjebbers
but the windows box uses 16 bits unicode, on linux 32bits unicode is
used.
the memory used by my program is for the larger part unicode strings.
....well, but that makes the better performance of linux even
stranger...
>
> Try an alpha of Python 2.7. If you're lucky "your" bug was found 'n' fixed
> independently.
(I run into this crash a longer time ago)
but seriously, I'll try this.
kind regards,
henk-jan
>
> Peter
hjebbers
> On Fri, 12 Feb 2010 09:21:07 -0800 (PST), hjebbers <hjebb...@gmail.com>
> declaimed the following in gmane.comp.python.general:
>
> > What strikes me is:
> > 1. the crash on windows, but linux works OK (same test sets)
> > 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
> > much).
>
>
> Windows in a normal installation only grants 2GB address space to
> user code, reserving the other 2GB space for the OS shared libraries. If
> your program attempts to allocate over that, it will fail. That the
> Windows box has twice the physical memory only means it doesn't resort
> to page swapping as soon.
>
> There is a boot parameter switch that toggles Windows into a 3GB
> user/1GB OS mode --
hey, that would be great!! on my 1,5G mahcine ;-)
> it is mainly meant for server machines where there
> won't be many disjoint OS libraries loaded, but the server applications
> need lots of data space.
>
> What split does the Linux OS use? If it give 3GB to user space,
> while you'd start to page swap much soon, you'd also have 50% more
> virtual memory that can be allocated than under Windows.
> --
> Wulfraed Dennis Lee Bieber KD6MOG
> wlfr...@ix.netcom.com HTTP://wlfraed.home.netcom.com/
I will check this.....any advice on how to check this?
henk-jan
hjebbers
thanks for the pointers so far.
if you feel the need to reproduce the crash, it's not that hard,
(downloading and installing my edi translator, install configuration
(button-click), and run.
I have a modified version (replace some *.py files) that eliminate a
lot of stuff (simpler setup, no database used, no external C-modules).
contact me for recipe and files
about me being reluctant to 'show the code':
- the crash that happens during stress tests.
- the crash occurs not always, and at different points.
- the code does runs OK on linux (which does not mean there's no bugt
in it ;-)
kind regards,
henk-jan
Diez B. Roggisch
> On Feb 13, 10:25 am, Dennis Lee Bieber<wlfr...@ix.netcom.com> wrote:
>> On Fri, 12 Feb 2010 09:21:07 -0800 (PST), hjebbers<hjebb...@gmail.com>
>> declaimed the following in gmane.comp.python.general:
>>
>>> What strikes me is:
>>> 1. the crash on windows, but linux works OK (same test sets)
>>> 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
>>> much).
>>
>> Which on its own does not mean much.
>>
>> Windows in a normal installation only grants 2GB address space to
>> user code, reserving the other 2GB space for the OS shared libraries. If
>> your program attempts to allocate over that, it will fail. That the
>> Windows box has twice the physical memory only means it doesn't resort
>> to page swapping as soon.
>>
>> There is a boot parameter switch that toggles Windows into a 3GB
>> user/1GB OS mode --
>
> hey, that would be great!! on my 1,5G mahcine ;-)
You don't get it - it's about the virtual memory address space
available. This has nothing to do with physical memory. Obviously if the
latter runs out, the swapping will make the machine painfully slow. But
that's another issue.
Diez
hjebbers
Diez,
you are right; I misunderstood.
henk-jan
hjebbers
The crash still happens.
btw, the crash does not happen at a peak memory usage.
According to windows task manager, at the moment of crash mem usage of
my program is 669kb, peak memory usage is 1.136kb
henk-jan
Alf P. Steinbach
Probably you meant to write "M", not "k" or "K"?
I've never managed to make much sense of the displays in Windows' Task Manager,
if that's what you're using, but I think the mem usage it's displaying by
default is the process' working set, or something very similar to that measure.
You can display additional columns in Task Manager, and one useful one is how
much virtual memory is allocated,.
And perhaps then (if that's not what you're looking it already) it'll be closer
to 2 GiB?
Cheers & hth.,
- Alf
hjebbers
> On Fri, 12 Feb 2010 09:21:07 -0800 (PST), hjebbers <hjebb...@gmail.com>
> declaimed the following in gmane.comp.python.general:
>
> > 1. the crash on windows, but linux works OK (same test sets)
> > 2. the linux box has 750Mb RAM, the windows box has 1.5Gb (twice as
> > much).
>
>
> Windows in a normal installation only grants 2GB address space to
> user code, reserving the other 2GB space for the OS shared libraries. If
> your program attempts to allocate over that, it will fail. That the
> Windows box has twice the physical memory only means it doesn't resort
> to page swapping as soon.
>
> There is a boot parameter switch that toggles Windows into a 3GB
> won't be many disjoint OS libraries loaded, but the server applications
> need lots of data space.
>
> What split does the Linux OS use? If it give 3GB to user space,
> while you'd start to page swap much soon, you'd also have 50% more
> virtual memory that can be allocated than under Windows.
> --
> Wulfraed Dennis Lee Bieber KD6MOG
> wlfr...@ix.netcom.com HTTP://wlfraed.home.netcom.com/
Hi Wulfraed,
I tried this (the 3GB switch).
But python still crashes.
kind regards,
henk-jan
Dave Angel
But the one you might want is a boot.ini option that tells the OS to
only reserve 1gb for itself, and leave 3gb for user space. But there
are tradeoffs, including the need to modify an application's executable
to take advantage of it. And the whole system may run substantially
slower, even when your're extended app isn't running. See links:
http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx
http://blogs.technet.com/askperf/archive/2007/03/23/memory-management-demystifying-3gb.aspx
DaveA
Roel Schroeven
Note that the memory measurements in Task Manager are pretty limited
and. Comparing Task Manager and Process Explorer:
Task Manager - Process Explorer
Mem Usage - Working Set
VM Size - Private Bytes
n/a - Virtual Size
I tend to trust Process Explorer a lot more than Task Manager. Note that
what Task Manager calls VM Size is not the size of the virtual memory as
might be expected (if Process Explorer is to be trusted), and that Task
Manager doesn't show the virtual memory size (at least on Windows XP).
--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
-- Isaac Asimov
Roel Schroeven
hjebbers
>
> DaveA
Yes, you are right, i also would need to modify the executable.
the reason why I posted on this list was the hard crash of python -
which python really should not do I think. AFAICS there is no 'bug' in
my edi translator (runs OK on linux) - but it uses far to much
memory.....
For me, I am going to bring back the memory footprint of my edi
translator. Which should be fairly easy to do.
kind regards,
henk-jan