How to use Python from C/C++ threads (new!)

[Guido van Rossum]

At the Python workshop the problem of invoking Python from a thread
created by C or C++ came up again. I did some thinking, and I believe
that the following will work. Note that the style is a bit rambling
-- I discovered more details while writing this up... Also note that
I haven't tested it, nor will I have the time to test it any time soon
-- but I'd love to hear if people get this to work, since it's more]
portable than Greg Stein's "free threading" patches and doesn't have a
problem with unprotected concurrent access to lists/dictionaries.

Here's the deal. Think about locking preconditions and postconditions
for the Python global interpreter lock (or, short, "the lock").

All Python API functions require that the lock is held on entry and
will hold it on exit. Some will temporarily release the lock and
reacquire it. In fact, anything that can invoke the interpreter will
do this -- the interpreter releases and reacquires the lock every N
bytecode instructions (settable through sys.setcheckinterval(N);
default N=10). (And anything that invokes DECREF can invoke the
interpreter. :-)

Threads created by the Python "thread" module acquire the lock when they
run (see t_bootstrap() in threadmodule.c) and release it when they exit.
Of course, the interpreter will still regularly release it while such
threads run -- but when it's released the lock, it won't run any Python
bytecode instructions itself.

Now, if you want to call the interpreter from a thread that *wasn't*
created from Python, you don't have the lock. Here's the proper procedure
to manipulate the lock, assuming Python is already initialized!!fmt
(glanced from t_bootstrap, but with new names):

PyEval_RestoreThread((void *)NULL);
...invoke the interpretyer to your heart's content...
(void) PyEval_SaveThread();

The call PyEval_RestoreThread() acquires the lock; PyEval_SaveThread()
releases it.

You must invoke PyEval_InitThreads() before you even think about
invoking Python from a thread, or calling PyEval_RestoreThread() -- it
makes sure the lock exists (if it doesn't exist, nothing is thread-safe
in Python). Do this right after calling Py_Initialize(). Oops --
here's a complication. This should only be done once, even though the
source looks like you can call Py_Initialize() and PyEval_InitThreads()
as often as you like. The reason is that if you are the first one to
call PyEval_InitThreads(), your thread ends up having acquired the lock
-- but if you are not the first, you don't have the lock! Since I
presume you will want to initialize Python from a thread (e.g. upon the
first reqest to a server that needs Python), here's what you have to
do:

static Python_Initialized = 0;

<begin critical section>
if (!Python_Initialized) {
Python_Initialized = 1;
Py_Initialize();
PyEval_InitThreads(); /* Create and acquire the lock */
}
else {
PyEval_RestoreThread(NULL); /* Acquire the lock */
}
<end critical section>

When this section completes, you have acquired the Python lock.

Alternatively, you can call PyEval_SaveThread() after
PyEval_InitThreads(), and not call PyEval_RestoreThread(), so you
always end up without the lock at this point. You can then acquire it
when you need it as shown above.

I have a strong feeling that this should enable multiple threads
started from C to use Python. You have to be careful that you only
invoke this when you *don't* have the lock -- functions invoked from
Python, e.g. the wrapper functions in extension modules, *do* have the
lock on entry (and should make sure they have it, still or again, on
any form of exit). Such functions, when they delve into code that
doesn't need the lock and may take arbitrarily long (e.g. by blocking
for I/O) should release and reacquire the lock by surrounding such code
with the macros Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS (look
at any Python extension that does I/O for an example, e.g. getline() in
fileobject.c. (Search for the old-style names BGN_SAVE and END_SAVE)

--Guido van Rossum (home page: http://www.python.org/~guido/)