python threads and the linuxthread pthread library
anton wilson
I've been attempting to understand how python threads will interact with my
Linux system (mainly the linuxthread library and the scheduler). I've
concluded that the python interpreter's context switching every 10 byte-codes
will rely on the Linux scheduler to somehow switch between threads after one
thread releases the global interpreter lock. However, I am a bit confused on
how threads interact with the python interpreter at a low level. Is the
Python Virtual Machine it's own thread? How do threads aquire the global
interpreter lock once they are awoken? How exactly do the byte-codes for a
particular thread get run by the Python interpreter? Does each thread run the
interpreter?
If anyone has any explanations, links, or pointers please let me know.
Thanks,
Anton Wilson
--
Camotion
Software Development
- c o v e n t r y -
applications do not perform as well as I expected, so an
explaination/pointers would help greatly.
-c
Tim Peters
> ...
> However, I am a bit confused on how threads interact with the python
> interpreter at a low level.
In general, if you have N threads, at any given time one of those threads is
holding the GIL (global interpreter lock), and the other N-1 threads are
blocked trying to acquire the GIL. No more than one thread can be running
in the PVM at a time. A C module that knows for certain it won't be running
the PVM can release the GIL to do some work in C (or Fortran, whatever),
allowing some other thread to run in the PVM while it's doing its C
(whatever) work. The Python implementation routinely does this around
potentially blocking C-level I/O operations.
> Is the Python Virtual Machine it's own thread?
No. All threads run in the PVM, but the GIL serializes their PVM time.
> How do threads aquire the global interpreter lock once they are awoken?
I'm not sure what you mean by "awoken". Python doesn't normally put threads
to sleep <wink>. Threads that don't hold the GIL are normally blocked
trying to acquire the GIL. If by "awoken" you mean "become unblocked", then
awakening and acquiring the GIL are exactly the same thing -- one doesn't
precede the other. The details of how the GIL gets acquired live inside
your platform thread implementation. On Linux, Python may use either
pthread condition variables, or POSIX semaphores, to implement the GIL.
> How exactly do the byte-codes for a particular thread get run by the
> Python interpreter?
See ceval.c. This isn't a thread issue beyond that threads in the PVM are
serialized by the GIL.
> Does each thread run the interpreter?
Yes, but they're serialized via the GIL.
anton wilson
From: anton wilson <anton....@camotion.com>
To: Tim Peters <tim...@comcast.net>
Date: Mon, 1 Jul 2002 16:52:12 -0400
> > How do threads aquire the global interpreter lock once they are awoken?
>
> I'm not sure what you mean by "awoken". Python doesn't normally put
> threads to sleep <wink>. Threads that don't hold the GIL are normally
> blocked trying to acquire the GIL. If by "awoken" you mean "become
> unblocked", then awakening and acquiring the GIL are exactly the same thing
> -- one doesn't precede the other. The details of how the GIL gets acquired
> live inside your platform thread implementation. On Linux, Python may use
> either pthread condition variables, or POSIX semaphores, to implement the
> GIL.
I think the reason why I'm confused here is that when the
PyThread_release_lock call ias executed, it simply locks the global int lock,
changes the locked variable to 0, unlocks the global interpreter lock, and
then signals a thread waiting on the unlocked mutex condition to wake up. I
can't find where a thread that gets swapped out by the interpreter blocks
when it tries to require the lock again.
Where does a swapped out thread block?
From thread_pthread.h:
status = pthread_mutex_lock( &thelock->mut );
CHECK_STATUS("pthread_mutex_lock[3]");
thelock->locked = 0;
status = pthread_mutex_unlock( &thelock->mut );
CHECK_STATUS("pthread_mutex_unlock[3]");
/* wake up someone (anyone, if any) waiting on the lock */
status = pthread_cond_signal( &thelock->lock_released );
CHECK_STATUS("pthread_cond_signal");
Joseph A Knapka
>
> [anton wilson]
> > ...
> > Is the Python Virtual Machine it's own thread?
>
> No. All threads run in the PVM, but the GIL serializes their PVM time.
Does Jython lift this limitation, by any chance? Or is
the GIL a universal and unchangable Python artifact?
Thanks,
-- Joe
- c o v e n t r y -
>
> Does Jython lift this limitation, by any chance? Or is
> the GIL a universal and unchangable Python artifact?
I don't know about jython, but _nothing_ is unchangeable (well, almost
nothing) - the GIL is whats used right now... Theres no reason to
believe that a non-GIL solution couldn't be found if people put thier
minds to it - it would be *hard*, and might break compatability (ACK!),
but it is possible.
How about a read-lock on global data? - all threads could maintain thier
own locals-state, and just lock on trying to read/write a global...
of course, first, all code in the current implementation would need to
be rentrant for true (OS) threads...
-c
holger krekel
>
> >
> > Does Jython lift this limitation, by any chance? Or is
> > the GIL a universal and unchangable Python artifact?
>
> I don't know about jython, but _nothing_ is unchangeable (well, almost
> nothing) - the GIL is whats used right now... Theres no reason to
> believe that a non-GIL solution couldn't be found if people put thier
> minds to it - it would be *hard*, and might break compatability (ACK!),
> but it is possible.
I have been told (at EuroPython and anywhere else) that removing the GIL
is just too hard. I'd still like to agree with you :-)
> How about a read-lock on global data? - all threads could maintain thier
> own locals-state, and just lock on trying to read/write a global...
> of course, first, all code in the current implementation would need to
> be rentrant for true (OS) threads...
I think its not that much about global data. Any object can be accessed from
threads. Refcounting is the first problem. Modifying dictionaries
and lists is the second. Coming to think of it, these two types are really
the two types that underly *any* modification!?! Except maybe for the
global and local namespaces which are highly optimized because of their
importance. Other than dicts, lists, namespaces and refcounts what
objects does python actually modify? (ints,strings,tuples,floats etc. are immutable).
Based on these observations i'd guess that interesting stuff could be done.
I'd like to check out if it is possible to make an object and everything
it recursively references 'read-only'. This requires hacking dict-proxy
objects at the C-level. They'd have to prevent write-access to themselves
and everything they pass out (e.g. the value of an attribute lookup).
Note that there have been discussions on python-dev about adding an
observer-pattern to python objects (get notifications on object
changes). I think this is much too heavy-weight and a simple read-only
property would be enough for many use-cases. Most important it would
support Copy-On-Write (COW) - semantics. Very important for transactional
systems which have a lot more reads than writes (like e.g. Zope).
Ultimately i'd really like to have extremly lightweight transactional
behaviour at the language level.
Of course this doesn't really answer any of your questions about the GIL but
i have the gut feeling that having read-only mechanisms at the type
level would be a step towards a relaxed GIL handling ...
really just my 2c,
holger
Ype Kingma
Jython does not have this limitation. Jython threads are Java threads.
Some basic types of the jython interpreter are java-synchronized,
eg. the dictionary type used for namespaces and threading.Lock.
For synchronisation you can use the standard python things, and also
others, eg util.concurrent by Doug Lea.
Finally, it's easy to manipulate the java thread priority from jython.
Have fun,
Ype
Tim Peters
> I have been told (at EuroPython and anywhere else) that removing the GIL
> is just too hard.
Greg Stein did it once, long ago. Search for "free threading" in old
archives, perhaps particularly in the defunct Thread-SIG. A single-threaded
Python program ran about 2x slower as a result; the need for mutual
exclusion doesn't go away, and in effect one honking giant hammer of a lock
(the GIL) got replaced with many teensy short-lived locks. That's got
substantial overhead of its own, although I'm sure it could have been
significantly reduced if anyone had mountains of time to throw at it.
Removing the GIL would be substantially harder today. There are both gross
and exceedingly subtle reliances on that the GIL is held, all over Python's
C code, and the extent of that reliance has only grown since Greg did his
experiment.
multiple-processes-remain-your-best-friends-ly y'rs - tim