Embeded Multi-Threaded Python: PyEval_InitThread(), PyEval_SaveThread(),...
Nicolas Duchastel
into a multi-threaded C++ application. We are using Pthreads on Solaris 7.
My application works fine, but only with 1 thread; and then, when I stop it,
it core dumps.
I am using Python 2.1.1 built on Solaris 7 with the SUN's C++ 5.2 compiler.
Here is a simplified architecture of the system:
-----------------------------------------------
I have an C++ application which creates a bunch of threads and then
accepts messages. Each messages is a request to execute some Python code.
Here's some pseudo-code of a 4 threaded system:
thread A: // INIT thread
do init stuff;
init Python interpreter:
Py_Initializ();
PyImport_ImportModule("MyModule");
...
start thread B;
start thread C;
end this thread;
thread B and C: // WORKER thread
wait and receive a new message;
if (msg is exit_message)
start Thread D;
stop other worker thread (B or C);
end this thread;
else
call python code:
PyRun_SimpleString(...);
OR
PyObbject_CallMethod(...);
thread D: // CLEAN-UP thread
wait for threads B and C to be stopped;
clean-up stuff
Py_Finalize();
QUESTIONS:
---------
1) what are the calls that I should be making !? in what order ?
any examples ? (i.e. not example of Python code executing some
multi-threaded stuff using the thread module, but rather some C/C++
code using the API C functions to implement a multi-threaded Interpreter).
2) what is the init sequence ? i.e. should PyEval_InitThreads be called ?
if so, before or after Py_Initialized() ? What objects need to be created ?
How ? (e.g. how do you create the locks ? what about the Thread State ?)
3) does the Python interpreter remember which OS thread is running what ?
i.e. one of my hypothesis for explaining the core dump is that the
interpreter is saying something like "Hey! You called Py_Initialize()
on thread t@1, you called PyObject_CallMethod() on thread t@4 and
now you are calling Py_Finalize() on thread t@7; what the #@$ are
you trying to do!?"
---> do I need to create a ThreadState object for every OS thread which
could possibly use the Python Interpreter ? or is it, on per
concurent thread ?
4) when executing my message (i.e. logic in threads B or C), what do I
have to do to lock ? When do I need to lock ? or unlock ?
Which methods: PyEval_SavedThread(), PyEval_AcquireThread(),
PyEval_AcquireLock(), Py_BEGIN_ALLOW_THREADS,.... ?
Any help would be greatly appreciated.
Thanks,
Nicolas Duchastel
Ignacio Vazquez-Abrams
> I am trying to use the Python/C API function to get an embeded Interpreter
> into a multi-threaded C++ application. We are using Pthreads on Solaris 7.
> My application works fine, but only with 1 thread; and then, when I stop it,
> it core dumps.
> I am using Python 2.1.1 built on Solaris 7 with the SUN's C++ 5.2 compiler.
http://www.python.org/doc/current/api/initialization.html
http://www.python.org/doc/current/api/threads.html
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Titus Brown
Nicolas Duchastel <nic...@otelnet.com> wrote:
>I am trying to use the Python/C API function to get an embeded Interpreter
>into a multi-threaded C++ application. We are using Pthreads on Solaris 7.
>My application works fine, but only with 1 thread; and then, when I stop it,
>it core dumps.
>I am using Python 2.1.1 built on Solaris 7 with the SUN's C++ 5.2 compiler.
Hi, Nicolas,
you might be interested in the PyWX source code: pywx.idyll.org/. It's
an LGPL'd project that put an embedded Python interpreter into AOLserver,
and it (correctly ;) deals with all of the issues you are facing.
Keeping the intricacies of the Python threading model in my head doesn't
seem to have worked too well, but here goes:
>I have an C++ application which creates a bunch of threads and then
>accepts messages. Each messages is a request to execute some Python code.
>Here's some pseudo-code of a 4 threaded system:
> thread A: // INIT thread
> do init stuff;
> init Python interpreter:
> Py_Initializ();
> PyImport_ImportModule("MyModule");
> ...
> start thread B;
> start thread C;
> end this thread;
I would do things in this order:
--
Py_Initialize();
PyEval_InitThreads(); // grabs global lock
/* start pthreads stuff -- actual OS threading -- here */
--
Before you do things with threads B and C, you'll want to have a separate
Interpreter or ThreadState.
> thread B and C: // WORKER thread
> wait and receive a new message;
> if (msg is exit_message)
> start Thread D;
> stop other worker thread (B or C);
> end this thread;
> else
> call python code:
> PyRun_SimpleString(...);
> OR
> PyObbject_CallMethod(...);
--
// grab Python lock
// if exit message, tell all other threads to exit and alert D;
// else call Python code
// release Python lock
--
> thread D: // CLEAN-UP thread
> wait for threads B and C to be stopped;
> clean-up stuff
> Py_Finalize();
what you have above, roughly.
>QUESTIONS:
>---------
> 1) what are the calls that I should be making !? in what order ?
> any examples ? (i.e. not example of Python code executing some
> multi-threaded stuff using the thread module, but rather some C/C++
> code using the API C functions to implement a multi-threaded Interpreter).
See PyWX for actual examples...
> 2) what is the init sequence ? i.e. should PyEval_InitThreads be called ?
> if so, before or after Py_Initialized() ? What objects need to be created ?
> How ? (e.g. how do you create the locks ? what about the Thread State ?)
Py_Initialize should be followed by InitThreads.
> 3) does the Python interpreter remember which OS thread is running what ?
No.
> i.e. one of my hypothesis for explaining the core dump is that the
> interpreter is saying something like "Hey! You called Py_Initialize()
> on thread t@1, you called PyObject_CallMethod() on thread t@4 and
> now you are calling Py_Finalize() on thread t@7; what the #@$ are
> you trying to do!?"
That sounds pretty reasonable to me ;).
> ---> do I need to create a ThreadState object for every OS thread which
> could possibly use the Python Interpreter ? or is it, on per
> concurent thread ?
I believe, technically, that you'll want to create a ThreadState object
for every Python execution process whose execution you want interleaved.
ThreadStates execute Python code in sequence...
> 4) when executing my message (i.e. logic in threads B or C), what do I
> have to do to lock ? When do I need to lock ? or unlock ?
> Which methods: PyEval_SavedThread(), PyEval_AcquireThread(),
> PyEval_AcquireLock(), Py_BEGIN_ALLOW_THREADS,.... ?
You'll want to allow threads *after* your C++ code is finished doing
C API calls into Python, and grab the lock *before* your C++ code starts
doing C API calls into Python. Python code itself is sprinkled with the
statements so you don't need to worry about executing bodies of Python code.
Note also that PyEval_InitThreads grabs the lock, so you'll need to do a
lock release before the end of your A function.
It took me a few weeks to get PyWX working well with all of this; I'd be happy
to correspond with you after you take a look at that, to help explain stuff.
I find that the C API in Python for this stuff is confusing unless you have
a handle on what functions exist ;), so that's why I'd suggest starting to play.
Once it all works, it's great, though ;).
cheers,
--titus
Nicolas Duchastel
I actually RTFM-ed it; that's why my question is so detailed.
The doc seem to be conflicting in some areas and lacks some important
information.
For example, Chapter 8 states that Py_Initialized() can be called
before or after PyEval_InitThreads(); so which is best ? what are the
implications ? any nuance between these two ways ? any examples ?
Also, chapter 8.1 does indicated 4 different ways to do locking:
C_Function_ToCallPythonCode()
{
PyThreadState* save = PyEval_SaveThread(); // LOCK
PyImport_ImportModule( ...); // load another module
PyRun_SimpleString(...); // execute a 1 line of Python code
PyPObject_CallMethod(....); // execute a full Python method
PyEval_RestoreThread(save); // UNLOCK
}
OR
C_Function_ToCallPythonCode()
{
PyEval_AcquireLock(); // LOCK
PyImport_ImportModule( ...); // load another module
PyRun_SimpleString(...); // execute a 1 line of Python code
PyPObject_CallMethod(....); // execute a full Python method
PyEval_ReleaseLock(); // UNLOCK
}
OR
C_Function_ToCallPythonCode()
{
PyEval_AcquireThread(??); // LOCK
PyImport_ImportModule( ...); // load another module
PyRun_SimpleString(...); // execute a 1 line of Python code
PyPObject_CallMethod(....); // execute a full Python method
PyEval_ReleaseThread(??); // UNLOCK
}
OR
C_Function_ToCallPythonCode()
{
Py_BEGIN_ALLOW_THREADS
PyImport_ImportModule( ...); // load another module
PyRun_SimpleString(...); // execute a 1 line of Python code
PyPObject_CallMethod(....); // execute a full Python method
Py_END_ALLOW_THREADS
}
So which one should I use ?
Also, image my 2 distinct thread B and C executing the 1st snipplet of
code above, if the call to PyObject_CallMethod() is VERY long and
takes ages,...
chapter 8 says
"In order to support multi-threaded Python programs, the
interpreter regularly releases and reacquires the lock --
by default, every ten bytecode instructions."
Does that thus mean that a 2nd thread running this 1st snipplet
above will be interupted in the middle of its work in
PyObject_CallMethod() and another thread will have a go at it.
If so, what happens to the ThreadState object when the 2nd thread
calls PyEval_SaveThread() ? if it doesn't swap it, it will run with
the 1st thread's ThreadState object. If it does swap it, when this
2nd thread will also get interupted, won't the 1st thread get back
to run and thus it will run with the 2nd thread's object !?
I mean, I must be missing someting !?
Also, what about the things I was asking with regards to the actual OS
thread's ID ? i.e. does it matter that my code starts in one thread
and then works from another thread ? do I need to create ThreadState
objects for each threads ?
Thus,.. please RTFP (where P is for Posting rather than Manual)
Thanks,
Nicolas
Ignacio Vazquez-Abrams <ign...@openservices.net> wrote in message news:<mailman.1000924701...@python.org>...
Ignacio Vazquez-Abrams
> I actually RTFM-ed it; that's why my question is so detailed.
> The doc seem to be conflicting in some areas and lacks some important
> information.
>
> For example, Chapter 8 states that Py_Initialized() can be called
> before or after PyEval_InitThreads(); so which is best ? what are the
> implications ? any nuance between these two ways ? any examples ?
It doesn't seem as though it matters, just so long as you haven't created any
threads before calling PyEval_InitThreads().
> Also, chapter 8.1 does indicated 4 different ways to do locking:
>
> [snip]
>
> So which one should I use ?
Probably the one that is clearest, i.e., Py_(BEGIN|END)_ALLOW_THREADS.
> Also, image my 2 distinct thread B and C executing the 1st snipplet of
> code above, if the call to PyObject_CallMethod() is VERY long and
> takes ages,...
> chapter 8 says
>
> "In order to support multi-threaded Python programs, the
> interpreter regularly releases and reacquires the lock --
> by default, every ten bytecode instructions."
>
> Does that thus mean that a 2nd thread running this 1st snipplet
> above will be interupted in the middle of its work in
> PyObject_CallMethod() and another thread will have a go at it.
AFAIK, when you're running a C program that doesn't use PyRun_*() you aren't
actually using any bytecode so there's no reason for there to be a problem.
> Also, what about the things I was asking with regards to the actual OS
> thread's ID ? i.e. does it matter that my code starts in one thread
> and then works from another thread ? do I need to create ThreadState
> objects for each threads ?
The easiest thing to do is probably just to have your main thread both
initialize and finalize the Python environment.
In my experience the most frequent cause of core dumps is that you have under-
or overestimated the need to increment or decrement the reference count of a
Python object. If you're willing, I can take a look at your code and tell you
if that's the case.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Frederic Giacometti
"Nicolas Duchastel" <nic...@otelnet.com> wrote in message
news:8e0ac4fb.01091...@posting.google.com...
> Thanks for the quick response.... but.....
>
> I actually RTFM-ed it; that's why my question is so detailed.
What is this funky RTFM verb ???
> The doc seem to be conflicting in some areas and lacks some important
> information.
>
> For example, Chapter 8 states that Py_Initialized() can be called
> before or after PyEval_InitThreads(); so which is best ? what are the
> implications ? any nuance between these two ways ? any examples ?
This is not conflicting, it just states the order dones not matter; which is
true.
What's missing in the documentation is the indication that
PyEval_InitThreads() creates and initialize the global lock; and thus must
have been called before any real thread operation is engaged.
The code of PyEval_InitThreads is only a couple of lines long. Looking at it
might help you, too.
> Also, chapter 8.1 does indicated 4 different ways to do locking:
> C_Function_ToCallPythonCode()
> {
> PyThreadState* save = PyEval_SaveThread(); // LOCK
> PyImport_ImportModule( ...); // load another module
> PyRun_SimpleString(...); // execute a 1 line of Python code
> PyPObject_CallMethod(....); // execute a full Python method
> PyEval_RestoreThread(save); // UNLOCK
But they don't refer to the same locks.... For instance, there is a global
interpreter lock, and an import lock; these are two different locks.
So, yes, you're locking, but you're not locking the same thing. Not all
locks are the same, you know :))
> }
> OR
> C_Function_ToCallPythonCode()
> {
> PyEval_AcquireLock(); // LOCK
> PyImport_ImportModule( ...); // load another module
> PyRun_SimpleString(...); // execute a 1 line of Python code
> PyPObject_CallMethod(....); // execute a full Python method
> PyEval_ReleaseLock(); // UNLOCK
> }
> OR
> C_Function_ToCallPythonCode()
> {
> PyEval_AcquireThread(??); // LOCK
> PyImport_ImportModule( ...); // load another module
> PyRun_SimpleString(...); // execute a 1 line of Python code
> PyPObject_CallMethod(....); // execute a full Python method
> PyEval_ReleaseThread(??); // UNLOCK
There are two global interpreter lock/thread function/macro sets: one which
is source-compatible with single-threaded Python builds, and another one
which is not.
In addition to this, PyEval_InitThread() does some acrobaties to dynamically
enable/disable thread on python extension modules, so that the same module
can work on mutli-threaded and single-threaded python engine; non of this
being actually documented, of course :))
As resultat des courses, when you invoke binary extensions from the 'python'
command, the PyEval_InitThread() is only executed upon loading a module
requiring threads (e.g. the threads module).
If you create your own binary module which might work on its own thread,
you'll also want to insert a PyEval_InitThread() call in the module
initialisation function.
Voila...
> }
> OR
> C_Function_ToCallPythonCode()
> {
> Py_BEGIN_ALLOW_THREADS
> PyImport_ImportModule( ...); // load another module
> PyRun_SimpleString(...); // execute a 1 line of Python code
> PyPObject_CallMethod(....); // execute a full Python method
> Py_END_ALLOW_THREADS
> }
>
> So which one should I use ?
You should use the macros, preferably to explicit function calls, whenever
you can...
> Also, image my 2 distinct thread B and C executing the 1st snipplet of
> code above, if the call to PyObject_CallMethod() is VERY long and
> takes ages,...
Python takes care of it, don't worry :)))
> chapter 8 says
>
> "In order to support multi-threaded Python programs, the
> interpreter regularly releases and reacquires the lock --
> by default, every ten bytecode instructions."
>
> Does that thus mean that a 2nd thread running this 1st snipplet
> above will be interupted in the middle of its work in
> PyObject_CallMethod() and another thread will have a go at it.
That's what thread switching is about, doesn't it ?
> If so, what happens to the ThreadState object when the 2nd thread
> calls PyEval_SaveThread() ? if it doesn't swap it, it will run with
> the 1st thread's ThreadState object. If it does swap it, when this
> 2nd thread will also get interupted, won't the 1st thread get back
> to run and thus it will run with the 2nd thread's object !?
> I mean, I must be missing someting !?
No (or maybe a little imagination :)): Just think of the Python virtual
machine as a single process ressource shared between the native threads.
> Also, what about the things I was asking with regards to the actual OS
> thread's ID ? i.e. does it matter that my code starts in one thread
> and then works from another thread ? do I need to create ThreadState
> objects for each threads ?
?????
> Thus,.. please RTFP (where P is for Posting rather than Manual)
Qu'es aqueo ????
>
> Thanks,
De rien.
> Nicolas
FG