xmlrpc idea for getting around the GIL
Patrick Stinson
RPC mechanism in order to move the GIL out of the process? I have
multiple audio threads, each of which use the python interpreter but
don't have to interact with each other and can might as well use a
separate interpreter handled by a daemon with libpython linked in.
I've never used a C xmlrpc lib before and would probably need to set
up some shared memory to get it to work. Can anyone suggest a library
out there that would at do that sort of thing, maybe with some
cross-platform process management?
I imagine this is how the multiprocessing module works.
Daniel Fetchinson
Yes, it does seem you need multiprocessing. That would take advantage
of multiple cores and the GIL wouldn't bother you at all. Take a look
at
http://docs.python.org/library/multiprocessing.html
For what you describe (separate tasks running concurrently without
communicating) the multiprocessing module would be ideal.
Cheers,
Daniel
--
Psss, psss, put it down! - http://www.cafepress.com/putitdown
Diez B. Roggisch
The problem is that the OP has a embedded application running threads.
multiprocssing doesn't help there.
Diez
Patrick Stinson
Carl Banks
wrote:
> On Sun, Nov 22, 2009 at 3:15 PM, Diez B. Roggisch <de...@nospam.web.de> wrote:
icating) the multiprocessing module would be ideal.
> > The problem is that the OP has a embedded application running threads.
> > multiprocssing doesn't help there.
>
It's quite possible to do that. A thread started from C can make
calls to Python if it first calls PyGILState_Ensure, although you'd
have to make sure that the Python interpreter has been previously
initialized. See PEP 311 for details.
http://www.python.org/dev/peps/pep-0311/
I also suggest that if you want people to be more receptive to write
your replies after the quoted text. That is the custom in this
newsgroup/mailing list.
Carl Banks
Patrick Stinson
> http://mail.python.org/mailman/listinfo/python-list
>
thanks for the tip.
What I meant was that I am *not allowed* to make calls to the CPython
API from the threads I currently have because these threads are high
priority and are never allowed to make blocking calls. Fortunately,
each of these threads can have a completely separate interpreter, so
my idea was to create a daemon process for each thread. This daemon
would contain the libpython symbols and would make the CPython calls.
This would keep my current threads from having to contend over the
single GIL.
My question was whether or not anyone has done anything like this in
C/C++. This situation is slightly unique in that I am trying to
maintain separate interpreters within a single app (which is currently
kind of hacked out using a single interpreter, but it's ugly), but I
could see how this sort of thing would be useful for other C/C++ apps
that implement an embedded scripting engine.
My reference to multiprocessing was based on the idea that the library
hides the details fo the process management, shared memory, and rpc
mechanisms. Again, I can't use multiprocessing because it runs *in*
python I need this to be implemented *outside* of python to avoid
acquiring the GIL. complex, I know.
Naturally, the most intimidating part of perusing this kind of idea is
the overhead of the processess management and the RPC. Further,
libpython executes callable objects using C function pointers, and I
can't think of a way that this would be able to re-gain access to the
original app's respective functions if the interpreter was living in
another processes.
That's not to mention the impossibility of debugging.
Damn you, Gil.
Diez B. Roggisch
AFAIK, instantiating several interpreters is possible. There seem to be
problems with extension modules, but maybe you don't need them, or can
work around that problem.
http://mail.python.org/pipermail/python-dev/2006-December/070370.html
Diez
"Martin v. Löwis"
> API from the threads I currently have because these threads are high
> priority and are never allowed to make blocking calls. Fortunately,
> each of these threads can have a completely separate interpreter, so
> my idea was to create a daemon process for each thread. This daemon
> would contain the libpython symbols and would make the CPython calls.
> This would keep my current threads from having to contend over the
> single GIL.
If this ("never allowed to make blocking calls") is a strict
requirement, then this is no solution, either. Communicating to the
remote process would involve IO, and all IO operations may block.
Of course, you might use non-blocking IO, in which case I wonder how
the thread continues if it finds that the IO is blocking. I also wonder
how the thread will find out that the result of the computation is
available, when it is not allowed to wait for the result.
In any case, I don't think you'll need a multi-process solution;
a single-process multi-threading approach will do fine. Just create
*another* thread, that runs at a low priority and is allowed to block.
Run the Python interpreter in that thread (create multiple of these if
you need them).
Then, use some queuing producer-consumer communication between the
high-priority thread and the low priority thread. Have the high
priority threads put requests into the queue, and the low priority
thread take them from the queue, dispatching them to Python. Make
sure you use non-blocking synchronization on the queue, or else
priority inheritance if you can get permission to do so.
> Damn you, Gil.
It may sound to you like the GIL is responsible for that. However,
consider a world where the GIL was removed in Python. There would
*still* be synchronization for data structures be going on. As you
are not allowed to have any blocking operation in the high-priority
threads, you *still* couldn't directly call the Python interpreter
(or any other thread-safe library) from your high-priority threads.
Regards,
Martin
sturlamolden
wrote:
> Has anyone every tried wrapping the CPython lib into a daemon with an
> RPC mechanism in order to move the GIL out of the process?
> I imagine this is how the multiprocessing module works.
It does not.
sturlamolden
wrote:
> What I meant was that I am *not allowed* to make calls to the CPython
> API from the threads I currently have because these threads are high
> priority and are never allowed to make blocking calls. Fortunately,
> each of these threads can have a completely separate interpreter,
This seems confused. What would they do with the interpreter if they
cannot call the CPython API?
> My question was whether or not anyone has done anything like this in
> C/C++.
I have programmed parallel numerical software in Python, including on-
line signal processing. I have yet to find the GIL gets in my way.
Usually people complaining about the GIL does not know what they are
talking about.
Daniel Fetchinson
>>> > The problem is that the OP has a embedded application running threads.
>>> > multiprocssing doesn't help there.
>>>
>>> that's right. I cannot make CPython calls from my original C-based
>>> threads.
>>
>>
>> It's quite possible to do that. A thread started from C can make
>> calls to Python if it first calls PyGILState_Ensure, although you'd
>> have to make sure that the Python interpreter has been previously
>> initialized. See PEP 311 for details.
>>
>> http://www.python.org/dev/peps/pep-0311/
>>
>> I also suggest that if you want people to be more receptive to write
>> your replies after the quoted text. That is the custom in this
>> newsgroup/mailing list.
>>
>
> API from the threads I currently have because these threads are high
> priority and are never allowed to make blocking calls. Fortunately,
> my idea was to create a daemon process for each thread. This daemon
> would contain the libpython symbols and would make the CPython calls.
> This would keep my current threads from having to contend over the
> single GIL.
>
> maintain separate interpreters within a single app (which is currently
> kind of hacked out using a single interpreter, but it's ugly), but I
> could see how this sort of thing would be useful for other C/C++ apps
> that implement an embedded scripting engine.
>
> hides the details fo the process management, shared memory, and rpc
> mechanisms. Again, I can't use multiprocessing because it runs *in*
> python I need this to be implemented *outside* of python to avoid
> acquiring the GIL. complex, I know.
>
> Naturally, the most intimidating part of perusing this kind of idea is
> the overhead of the processess management and the RPC. Further,
> libpython executes callable objects using C function pointers, and I
> can't think of a way that this would be able to re-gain access to the
> original app's respective functions if the interpreter was living in
> another processes.
>
> That's not to mention the impossibility of debugging.
>
> Damn you, Gil.
By the way, you might be interested in this thread as well:
HTH,
sturlamolden
> By the way, you might be interested in this thread as well:
>
> http://groups.google.com/group/comp.lang.python/browse_thread/thread/...
It is Windows specific, and it does not work with extension modules.
Yes we can embed multiple interpreters just by making multiple copies
of Python26.dll. It's an ugly hack and not one I'd recommend for
production servers.
Antoine Pitrou
Actually, it is how multiprocessing works under Windows (for lack of the
fork() function), except that it uses pickle by default (but it does have
xmlrpc support).
Regards
Antoine.
sturlamolden
> >> Has anyone every tried wrapping the CPython lib into a daemon with an
> >> RPC mechanism in order to move the GIL out of the process?
>
> >> I imagine this is how the multiprocessing module works.
>
> > It does not.
>
> Actually, it is how multiprocessing works under Windows (for lack of the
> fork() function), except that it uses pickle by default (but it does have
> xmlrpc support).
Windows does not have daemons, so this is obviously incorrect. (There
are something called Windows Services, but multiprocessing does not
use them.)
Multiprocessing on Windows uses the subprocess module.
Antoine Pitrou
>
> Windows does not have daemons, so this is obviously incorrect. (There
> are something called Windows Services, but multiprocessing does not use
> them.)
This is nitpicking. Technically it might not be a daemon but it's used as
such.
The important point is that it does use a (custom) form of RPC through
marshalling, which is what the original question was about.
Aahz
=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <mar...@v.loewis.de> wrote:
>
>In any case, I don't think you'll need a multi-process solution; a
>single-process multi-threading approach will do fine. Just create
>*another* thread, that runs at a low priority and is allowed to block.
>Run the Python interpreter in that thread (create multiple of these if
>you need them). Then, use some queuing producer-consumer communication
>between the high-priority thread and the low priority thread. Have the
>high priority threads put requests into the queue, and the low priority
>thread take them from the queue, dispatching them to Python. Make sure
>you use non-blocking synchronization on the queue, or else priority
>inheritance if you can get permission to do so.
This is close to what I would recommend, but I would suggest that the
low-priority thread make calls out to an external Python process; that
way, you can have a Python worker pool of processes. Based on what I've
seen posted over the years, I generally recommend against instantiating
multiple interpreter instances (although I've seen some success stories,
I've seen more people butting their heads against the brick wall).
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
The best way to get information on Usenet is not to ask a question, but
to post the wrong information.
Patrick Stinson
thread in which I am "not allowed to make a blocking call," but actual
turn around times would be far better than forcing all threads to wait
on the Gil. As it stands, blocking on a single thread lock *almost*
works, and while we can claim this or that theoretical deficiency, the
proof is in the pudding and the pudding almost works. In fact, it does
work and has made us money, we just need one more once of performance
at low latencies and I can put this way.
Most machines that are running an audio sequencing host app are not
running other apps, and we can generally assume that the host app is
the only one using up the majority of resources on the local machine.
So, as it turns out, we can assume quite a predictable and stable
operating environment for our code.
We don't need extension modules, and all we need to do is run some
fairly basic scripts that make callbacks and use some sip-wrapped
types. Py_CallObject is our only entry point into python execution
from within the critical thread.
SIDE NOTE: There are a lot of use cases that the educated python
community is not familiar with that are really spectacularly
interesting, and I think some of the myths surrounding python are
incorrect.
- Python is not suitable for real-time work.
Not true. We have been running python callback code using
PyObject_CallObject from within our audio thread for some time without
a problem, and it's *extremely* fast.
- Especially in a production environment.
Not true. We sell the industry leading sampler engine, and it has been
paying my salary for three years. It's high performance - every cycle
counts. Our sampled instruments is loaded as a plugin from third-party
applications and has been used to make movies you have seen. Our
customers love it and have never complained about problems with the
scripting engine.
Audio work is interesting, because you can think and think and think
about what will work and what wont, and at the end of the day all that
matters is that the audio is smooth and the code doesn't crash. We
need just a liiiitle push to get our code to work at low latencies,
and the only thing that is standing in our way is that all threads
9usually around 20 have to block on the Gil, and this causes small
gaps in the sound at low latencies (around 5ms, or 64 sample period).
...almost perfect.
alex23
> Not true. We sell the industry leading sampler engine, and it has been
> paying my salary for three years. It's high performance - every cycle
> counts. Our sampled instruments is loaded as a plugin from third-party
> applications and has been used to make movies you have seen. Our
> customers love it and have never complained about problems with the
> scripting engine.
As an occasional bedroom producer I have to ask if you are talking
about EastWest's Play?
When was Python support added? I'm surprised I missed this at the
time...
sturlamolden
wrote:
> We don't need extension modules, and all we need to do is run some
> fairly basic scripts that make callbacks and use some sip-wrapped
> types.
Sure, you use SIP but not extension modules...
> - Python is not suitable for real-time work.
>
> Not true. We have been running python callback code using
> PyObject_CallObject from within our audio thread for some time without
> a problem, and it's *extremely* fast.
It seems you are confusing "real-time" with "real-fast". The fact that
something runs fast does not make it "real-time".
Python is not suitable for real-time applications, nor are the OSes
commonly used to run Python.
> We
> need just a liiiitle push to get our code to work at low latencies,
> and the only thing that is standing in our way is that all threads
> 9usually around 20 have to block on the Gil, and this causes small
> gaps in the sound at low latencies (around 5ms, or 64 sample period).
>
> ...almost perfect.
Python is not programmed with real-time applications in mind: You have
no guarrantees on maximum time-lag when a thread is blocked on the
GIL.
"Priority requests" (i.e. pre-emptive multitasking) was removed from
Antoine's "newgil" branch, but that is the kind of mechanism you would
need. Even with priority requests, Python would not be suitable for
real-time apps, as extension modules (e.g. C++ wrapped with SIP) can
hold the GIL forever.
You will also need an OS with a real-time scheduler and a real-time C
library, such as QNX or VxWorks.
I find thea idea of a true real-time Python very interesting, but it
would take a completely reworked interpreter.
Patrick Stinson
> On 2 Des, 02:47, Patrick Stinson <patrickstinson.li...@gmail.com>
> wrote:
>
>> We don't need extension modules, and all we need to do is run some
>> fairly basic scripts that make callbacks and use some sip-wrapped
>> types.
>
> Sure, you use SIP but not extension modules...
>
>
>> - Python is not suitable for real-time work.
>>
>> Not true. We have been running python callback code using
>> PyObject_CallObject from within our audio thread for some time without
>> a problem, and it's *extremely* fast.
>
> It seems you are confusing "real-time" with "real-fast". The fact that
> something runs fast does not make it "real-time".
>
> Python is not suitable for real-time applications, nor are the OSes
> commonly used to run Python.
>
Semantics aside, my point is that it runs well enough in our
environment. If the audio is smooth, it is considered "working".
>
>
>> We
>> need just a liiiitle push to get our code to work at low latencies,
>> and the only thing that is standing in our way is that all threads
>> 9usually around 20 have to block on the Gil, and this causes small
>> gaps in the sound at low latencies (around 5ms, or 64 sample period).
>>
>> ...almost perfect.
>
> Python is not programmed with real-time applications in mind: You have
> no guarrantees on maximum time-lag when a thread is blocked on the
> GIL.
We don't need theoretical guarantees, because we've tried it and it
works. That's the bottom line
>
> "Priority requests" (i.e. pre-emptive multitasking) was removed from
> Antoine's "newgil" branch, but that is the kind of mechanism you would
> need. Even with priority requests, Python would not be suitable for
> real-time apps, as extension modules (e.g. C++ wrapped with SIP) can
> hold the GIL forever.
see above.
>
> You will also need an OS with a real-time scheduler and a real-time C
> library, such as QNX or VxWorks.
>
> I find thea idea of a true real-time Python very interesting, but it
> would take a completely reworked interpreter.
>
>
>
>
>
>