Strange crash while running a script with a embedded python interpreter

[Rickard Englund]

First, some system info
* Windows 7 (also tested on 8 and 10)
* Python 3.5.1 64bit (previously also tested using several 3.x versions) (also tested with 32 bit, but with 3.4.2)
* Microsoft Visual Studio 2015 (earlier version of python also tested with Visual Studio 2013)

We have tested with both the debug and release libs for 3.5 and 3.5.1 (those available from python installer) with the same problem.

The following problem has also been tested on Unix and on MacOS where it works as expected, the problem is only in Windows.


Problem description:
We have an application for scientific visualization in which we have enabled scripting using embedded python. We have exposed quite some features of our application to python using modules, this works perfectly fine.
The problem we have is that the application crashes as soon as a script is trying to import some of python's default libraries, such as os or random. For example, running the single line code "import os" produces a crash with the following stacktrace:
http://pastebin.com/QpfGrMY0

We initialize the python interpreter using the following:
http://pastebin.com/uyx6gdMb

We have tried with and without the Py_NoSiteFlag = 1. When removing that flag it works for some modules, for example "import os" works, but other modules, for example "import random" crashes.


In the pastebin link above for the python initialization there are three lines commented out, "importModule("os")","importModule("glog")" and "importModule("random")", if I uncomment those lines the application will not crash when importing those modules. But that is not a viable solution, we can't import all possible modules that way.

The following piece of code is how we execute python code:
http://pastebin.com/HDUR2LKT


It only crashes for modules that exists , if I try to import a module that does not at all exist (eg import unexisting_module) , it fails as expected with a nice error message saying it does not exist.


I don't know if we are doing anything wrong in the initialization of the interpreter or when running the script. We can run scripts that are not using any modules or are only using the python modules we create in c++ to expose functionality of our application.
Everything is working fine on MacOSX and Linux (tested with Ubuntu) which are the platforms we support. We had a similar setup before using python 2.x instead of 3.x which also worked without any problems.


The software i am talking about is using QT for gui and and OpenGL for hardware accelerated graphics. But I have tested it in a minimal application where all QT stuff were omitted and the same issue is still there.

The software is open source and can be downloaded/clones from github.com/inviwo/inviwo.

The python initialization is done in
https://github.com/inviwo/inviwo-dev/blob/master/modules/python3/pyinviwo.cpp
and the script compiling and running is handled in
https://github.com/inviwo/inviwo-dev/blob/master/modules/python3/pythonscript.cpp


I have searched a lot on google and in the issue tracker for python but found nothing that can answer my questions. So I hope that someone here can us solve this.

[Rickard Englund]

I notice now that I have pasted 2 links to our private repository, which will not work.
The correct links are:
https://github.com/inviwo/inviwo/blob/master/modules/python3/pyinviwo.cpp
https://github.com/inviwo/inviwo/blob/master/modules/python3/pythonscript.cpp

[Michael Torrie]

On 01/08/2016 09:18 AM, Rickard Englund wrote:
> First, some system info
> * Windows 7 (also tested on 8 and 10)
> * Python 3.5.1 64bit (previously also tested using several 3.x versions) (also tested with 32 bit, but with 3.4.2)
> * Microsoft Visual Studio 2015 (earlier version of python also tested with Visual Studio 2013)

Are you using the same version of Visual Studio that Python itself was
compiled with? If not there can be subtle problems with incompatible
msvcrt dlls. No idea if this would be contributing to the problem or
not, though.

[Rickard Englund]

I've just downloaded the python source code and build it myself, the compiler settings in our project is the same as in the python projects.


Though, your comment led me in the correct direction. When using the libs/binary I built my self it let me see the values of the PyObjects around the crash and it seems like it is related to our modules and/or how we expose them to the interpreter. When disabling our own modules everything seems to be working as it should.

I think I got it wokring now, used some ugly hacks I need to clean up before I certain it works.

Thanks for the help.

[Chris Angelico]

Interesting. So somewhere along the way, you have native code (C
code?) that's creating a module, and that's where the trouble starts?
I would first look at all your refcount management; if that goes
wrong, all sorts of craziness could happen (if you still have a
pointer to a block of memory that gets released and reused, hilarity
will ensue - or, something will). Have you considered using Cython?
That might let you do what you need without worrying about all those
annoying internal API details.

ChrisA

[Travis Griggs]

This may not be a great list for this question (which would be?); it’s a big group, and I’m hoping there’s some people here that cross into these same areas.

I’m new to dbus, it seems it’s a sort of CORBA for the Linux world. :) Python seems to be a popular way to interact with it. I’m trying to interact with the BlueZ services for Bluetooth LE stuff, and the scant hints I can find seem to indicate dbus is the preferred way going forward. The BlueZ distribution even provides example code. That said, my question should be independent of whether this was BLE or a dbus interface for a Calculator program.

There is a class defined as such:

class Characteristic(dbus.service.Object):
def __init__(self, bus, index, uuid, flags, service):
# … set a bunch of attributes
dbus.service.Object.__init__(self, bus, self.path)

@dbus.service.method(GATT_CHRC_IFACE, in_signature='ay')
def WriteValue(self, value):
print('Default WriteValue called, returning error’)
raise NotSupportedException()

Then I have a subclass of my own:

class MyCharacteristic(Characteristic):
def __init__(self, bus, index, uuid, flags, service):
Characteristic.__init__(self, bus, index, uuid, flags, service)
# … set some of my own attributes

def WriteValue(self, value):
print(‘Got write value:’, value)
self.anotherMethod(value)
print(‘back from anotherMethod’)

def anotherMethod(self, value):
print(‘pondering this value:’, value)

My program does not call WriteValue directly. It seems to be called by the bluetooth machinery. The mainloop, or something like that. I don’t honestly know. I just know I use some other boilerplate code involving registration and the mainloop, to get it all running. And so the MyCharacteristic.WriteValue() method DOES get fired. I see the output. But when it gets to the anotherMethod call, it just seems to... not. More callbacks may fire later. But that’s the end of that one. I’ve tried this under python2 AND python3.

So my basic python-dbus question is: Is this some nuance where a callback method, inheriting from what looks like a proxy of some sort (dbus.service.Object) should/can not send messages to itself?

Help me python-obis, help me. You’re my only hope.