python callback called from a C thread
krja...@lbl.gov
from python. I was hoping someone might be able to help me with some
threading issues.
I'm trying to set a python callback function that will get called from a
thread created in the C library. After a little reading/trial & error,
I have something that sort of works, but isn't right.
In the function that sets the python callback I do the following:
thread_state = PyEval_SaveThread();
interp_state = thread_state->interp;
PyEval_RestoreThread(thread_state);
where interp_state is a static variable. This gives me a pointer to the
interpreter state that I can use later.
Then in the code that calls the python code I do:
my_state = PyThreadState_New(interp_state);
PyEval_RestoreThread(my_state);
call to PyEval_CallObject to call the python code
PyEval_SaveThread();
This leaks a PyThreadState structure everytime through, but has bigger
problems then that. If I don't push it to hard, everything will sort of
work, but I'll see the following message; "currentThread(): no current
thread for 2051". If I try to push it a bit, I'll get the following
also; "Fatal Python error: PyThreadState_Get: no current thread". The
fatal error appears to be when it's coming back to the main thread in
the interpreter.
Any examples, hints, suggestions on the right way to implement a python
callback that gets called from a thread created in a C library would be
greatly appreciated!
thx,
--keith
----------------------------------------------------------------
Keith R. Jackson KRJa...@lbl.gov
Distributed Security Research Group (510) 486-4401
Lawrence Berkeley National Laboratory
http://www-itg.lbl.gov/~kjackson/
David Bolen
> I'm trying to set a python callback function that will get called from a
> thread created in the C library. After a little reading/trial & error,
> I have something that sort of works, but isn't right.
Well, you need a thread state, but I don't think you should have to
create a new one for each callback.
Here's what I've done in one of my applications. It's not a perfect
match to your example, but perhaps it will help point in the right
direction.
In my case, Python is embedded in the C application, so the C code
controls startup and initial script execution (which runs in a
dedicated thread). The Python script can set an event object that the
C code (in another thread) will tickle when new data arrives. To do
this, that other C thread calls the "set" method on the event object,
but I have to call into the interpreter to call the method, so it
should work just as well for any other operation such as what you are
trying to do.
During my initialization, from within the dedicated Python thread, I
initialize the interpreter with:
Py_Initialize();
PyEval_InitThreads();
ecnaPy_RcvThreadState = PyThreadState_New(PyThreadState_Get()->interp);
where "ecnaPy_RcvThreadState" is a global PyThreadState pointer. I
then set up my own module (AddModule/InitModule and so on) and then
hand off control to the script (in my case with PyRun_SimpleFile).
Now, assuming that the Python script has at some point in the past
called into my "module" to provide me with the event object, in my
separate C thread that is receiving data, I set that event object with
the following:
PyEval_AcquireThread(ecnaPy_RcvThreadState);
if (!PyObject_CallMethod(ecnaPy_FrameEvent,"set",NULL)) {
/* Error handling */
}
PyEval_ReleaseThread(ecnaPy_RcvThreadState);
This acquires the global interpreter lock, using the thread state
created for use by the "extra" internal C code thread, then performs
the operation in the Python interpreter, and then releases the lock.
From what I recall, the SaveThread/RestoreThread operations are more
geared towards _releasing_ the global interpreter lock so you can
perform non-Python operations as opposed to the reverse, which is what
you want here (to obtain the lock so you can call into Python).
I've been able to run this way with hundreds of thousands of very
rapid callbacks into the interpreter from the C code during an
application execution without any problems.
> This leaks a PyThreadState structure everytime through, but has bigger
> problems then that. If I don't push it to hard, everything will sort of
> work, but I'll see the following message; "currentThread(): no current
> thread for 2051".
BTW, I don't think you can get rid of this message at least once, if
your script is using the threading module, at least not without
modifying that module.
The threading module keeps track of all of the threads that it
creates, but in this case, your internal C thread was not created
through the threading module. So when the first threading module
function gets used from within the context of your thread (e.g.,
during the callback), you'll get this message and the threading module
will allocate a dummy thread instance for its use. So you'll get the
message once, but it's really just a warning (although I found it sort
of annoying, since I didn't see a clean way to disable it even if I
knew what was going on with my internal thread).
--
-- David
--
/-----------------------------------------------------------------------\
\ David Bolen \ E-mail: db...@fitlinxx.com /
| FitLinxx, Inc. \ Phone: (203) 708-5192 |
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\-----------------------------------------------------------------------/
krja...@lbl.gov
I tried the AcquireThread function and everytime I do I get a
segmentation violation down in sigsuspend. From the core file it looks
like this is happening from the code that initiated the callback. I'm
not sure what's going on here.
>
> From what I recall, the SaveThread/RestoreThread operations are more
> geared towards _releasing_ the global interpreter lock so you can
> perform non-Python operations as opposed to the reverse, which is what
> you want here (to obtain the lock so you can call into Python).
Yes I agree. They don't cause seg faults for me though. ;-)
>
> I've been able to run this way with hundreds of thousands of very
> rapid callbacks into the interpreter from the C code during an
> application execution without any problems.
>
> > This leaks a PyThreadState structure everytime through, but has bigger
> > problems then that. If I don't push it to hard, everything will sort of
> > work, but I'll see the following message; "currentThread(): no current
> > thread for 2051".
>
> BTW, I don't think you can get rid of this message at least once, if
> your script is using the threading module, at least not without
> modifying that module.
>
> The threading module keeps track of all of the threads that it
> creates, but in this case, your internal C thread was not created
> through the threading module. So when the first threading module
> function gets used from within the context of your thread (e.g.,
> during the callback), you'll get this message and the threading module
> will allocate a dummy thread instance for its use. So you'll get the
> message once, but it's really just a warning (although I found it sort
> of annoying, since I didn't see a clean way to disable it even if I
> knew what was going on with my internal thread).
Ah, thanks for that info. It is kind of annoying warning, but I can
deal with that.
thanks for the help,
--keith
>
> --
> -- David
> --
> /-----------------------------------------------------------------------\
> \ David Bolen \ E-mail: db...@fitlinxx.com /
> | FitLinxx, Inc. \ Phone: (203) 708-5192 |
> / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
> \-----------------------------------------------------------------------/
--
David Bolen
> I tried the AcquireThread function and everytime I do I get a
> segmentation violation down in sigsuspend. From the core file it looks
> like this is happening from the code that initiated the callback. I'm
> not sure what's going on here.
Hmm - this was with a thread state pointer that you allocated solely
for the use of your callback thread?
From the reference to sigsuspend, I'm guessing this is a Unix platform
- my experience so far has been under NT. I suppose it might be an
issue with the specific threading support on your particular platform.