thread/queue bug
phil
I have a very strange bug. A thread in a .pyc stops dead.
This program has many threads and queues and has worked
great for months.
One thread listens for UDP messages from other programs,
and puts the messages in listenq.
Procmsgs gets from listenq and for a certain kind of
message creates another mq = Queue(0).
I don't wish to distribute the source to pnetx.py
so I have a short program pnet.py import it, that is
pnetx.pyc.
Then the procmsgs thread dies at the Queue statement.
But if I run pnetx.py as the main program, all works fine.
So if pnetx.py or .pyc is imported this thread dies. ???
I've tried this with 2.3.3 and 2.3.4. Anyone?
Peter Hansen
A few random thoughts:
1. You haven't provided nearly enough info for anyone
to be able to help much, I suspect.
2. Threading problems can be incredibly complex and race
conditions are frequently next to impossible to troubleshoot,
let alone reproduce reliably.
3. Distributing a .pyc does very little to prevent someone
from getting your source, since it is possible to "decompile"
.pyc files into something fairly closely resembling the
original source.
4. The fact that you have a .pyc file instead of a .py
file very likely has *nothing* to do with any threading
problem you are facing, so I suggest you get past that mental
block and look elsewhere.
5. Sorry I can't be more help. You don't give anyone much
to go on. All that stuff about "Queue(0)" and "listenq"
is pretty much meaningless to us, you know...
-Peter
phil
> file very likely has *nothing* to do with any threading
> problem you are facing, so I suggest you get past that mental
> block and look elsewhere.
Well, I tried to make it clear that the ONLY difference between
working and not working was the pnetx.pyc when imported did not
work when
#!/usr/bin/python
# pnet.py, exists only to import pnetx.py
import pnetx.py
Otherwise the program has worked fine for months.
I don't think its a mental block, its the ONLY difference.
It freezes on the following statement:
mq = Queue(0)
and I haven't a clue why. All the other threads continue to run.
> 5. Sorry I can't be more help. You don't give anyone much
> to go on. All that stuff about "Queue(0)" and "listenq"
> is pretty much meaningless to us, you know...
You know, I get this all the time on language support groups.
All of my Linux support groups, if they don't understand, say
why and ask for elaboration. I tried to explain what was going on,
without incuding the source. I have about 200 man hours in the source
and I bet it would take longer to understand it.
If my explanation was insufficient, I'm sorry.
ALSO, you did not respond to my email, so I didn't know how to reply.
There is nothing on the archives page which gives me a clue as to
how to respond.
SO, if there is ZERO chance there is some sort of inadvertent lock
occuring in saved byte code, I just kludge around and move on.
Maybe 2.4
Thanks.
phil
>someone posts unclear confusing questions and extraneous
>information. The noive!
I would be happy to elaborate. No one asked to me to elaborate.
I was simply told I didn't give enough information.
I wasn't given an idea of what additional information
was needed or some clue as to how to better phrase my question.
>I also find it remarkable that so many different people are
>all doing the same thing to you. It must be a conspiracy.
>Certainly not a problem with, say, you...
I didn't say so many people are doing the same thing to me.
Some language support groups make me feel like I am asking
stupid questions in a stupid way, not all. And the Linux
and FreeBSD support groups never do. In fact it seems
on those groups the more newbie and inadequate your questions
are the more patience and help you get.
I've never before on any group seen anyone told they
had a mental block, because they were fishing for info.
I have a genuine problem here, which I have no clue how to approach,
and I am very sorry I started a flame over protocol.
THAT has certainly never happened to me before. signing off.
Peter Hansen
> You know, I get this all the time on language support groups.
> All of my Linux support groups, if they don't understand, say
> why and ask for elaboration.
Wow, amazing! Imagine that... asking for elaboration when
someone posts unclear confusing questions and extraneous
information. The noive!
I also find it remarkable that so many different people are
Peter Hansen
> I've never before on any group seen anyone told they
> had a mental block, because they were fishing for info.
I'm sorry if I offended you by using that term. That
wasn't my intention. Communications can be difficult
in an online forum. For example, I would normally find
the phrase "fishing for info" to be offensive, but in
this case I'll assume we got off on the wrong foot and
that there are some different thought patterns involved,
and I'm sure you didn't intend to offend any more than
I did.
My intention with "mental block" was to emphasize as
strongly as I could my conviction that your problem
has *nothing* to do with the fact that you have a .pyc
file instead of a .py, and to encourage you to search
elsewhere, to find another way of getting more information
about the problem.
The one key difference that I can see on the face of
it is that a .pyc, being already compiled, doesn't
need a compilation phase the way a .py file would if
it didn't already have an up-to-date .pyc file
present.
I really doubt this small difference in time is the
source of the trouble, but it's possible. On the
other hand, when you run with the .py file, if you
run a second time and it still fails, then this is
even less likely to be the problem because it will
compile the .py file only the first time, and the
second time it will use the .pyc file just like
when you don't even have the .py file around.
And I could help more, being fairly experienced with
threading issues and race conditions and such, but
as I tried to indicate in the first place, you've
provided next to no useful (IMHO) information to
let anyone help you more than this... sorry. (And,
again, no offense intended by that statement.)
-Peter
phil
>And I could help more, being fairly experienced with
>threading issues and race conditions and such, but
>as I tried to indicate in the first place, you've
>provided next to no useful (IMHO) information to
>let anyone help you more than this
This is about 5% of the code.
Uses no locks.
I am mystified, I have written probably 100,000 lines
of Python and never seen a thread just lock up and quit
running. It happens on a Queue() statement so my suspicion
is a bug. ??
I have kludged around it by putting all the thread/queue stuff
in the main program and import the stuff I don't want to
distribute. But mysteries haunt your dreams, sooo...
#!/usr/local/bin/python
# pnetx.py
from threading import *
from time import *
from Queue import Queue
from socket import *
import sys
import os
# glob is a DUMMY CLASS
glob.listenq = Queue(1000)
def listener():
while 1:
msg,addrport = listenersocket.recvfrom(BUFSIZE)
addr = addrport[0]
glob.listenq.put (( msg,addr),1)
if msg == 'DIE': break
def procmsgs():
while 1:
msg,addr = glob.listenq.get(1)
if msg == 'DIE': break
wds = msg.split(';')
if wds[0] == 'S': recvsucc(msg); continue
if wds[0] == 'E': recvevent(msg); continue
if wds[0] == 'ACK': recvack(msg[4:]); continue
if wds[0] == 'MONITOR':
if addr not in monitorlist:
print 'This line ALWAYS PRINTS'
queuelist.append( Queue(0) )
## The above fails if this code is imported
## It doesn't matter whether it is imported
## as .py or .pyc
## also mq = Queue(0); queuelist.append(mq) # fails
print 'This line ALWAYS PRINTS if i execute this source'
print 'but NEVER PRINTS if I import '
print 'this source from a 1 line program'
print 'Thats weird'
monitoron.append( 1 )
monitorlist.append( addr )
queuelist = [Queue(0)]
listenersocket = socket(AF_INET,SOCK_DGRAM)
listenersocket.bind(ListenAddr)
procm = Thread(target=procmsgs)
procm.start()
listenr = Thread(target=listener)
listenr.start()
## Then start a bunch of other threads and queuea.
Don't spend a lot of time on this, not worth it.
I just thought someone might have experienced
something similar.
Peter Hansen
> Uses no locks.
It does use locks implicitly, though, since even just
importing threading will do that, and creating a Queue
does too.
> I am mystified, I have written probably 100,000 lines
> of Python and never seen a thread just lock up and quit
> running. It happens on a Queue() statement so my suspicion
> is a bug. ??
You have the source to Queue.py in your standard library
folder. Why not throw a few more print statements into
its __init__ and see what you learn?
> I have kludged around it by putting all the thread/queue stuff
> in the main program and import the stuff I don't want to
> distribute. But mysteries haunt your dreams, sooo...
> #!/usr/local/bin/python
[snip source]
I cobbled together a "working" version of your code
and ran it just fine, whether imported or run directly.
No lockups. On Windows XP.
Are you by any chance running on a new version of the
Linux kernel, where the threading model has changed?
(Or was it just RedHat 9.0?)
I don't know the details, but I know folks have had trouble
with this and Python... For example, I found this
reference to the issue:
http://groups.google.ca/groups?th=fe9a064ffeb38adc&seekm=bpi438%24qta%2408%241%40news.t-online.com#link2
-Peter
phil
>You have the source to Queue.py in your standard library
>folder. Why not throw a few more print statements into
>its __init__ and see what you learn?
Yeah I put some print statements in init and it seems
to complete.
>Are you by any chance running on a new version of the
>Linux kernel, where the threading model has changed?
>(Or was it just RedHat 9.0?)
RedHat 7.2, 2.4.20 kernel. Get the same hangup on
Win2000 Pro. Which leads me to believe it has nothing
to do with kernel threads.
Well its an anomaly. I sent to bug list.
Probably never see it again.
I think some sort of semaphore thingy, which I know
nothing about, is sneaking in under unreproducible
conditions. I'm moving on.
I'm love python for multi-thread, simple socket programming
and lots of reasons. This is the first thing that's looked
like a bug in 5 years (1.5), I'm not worried.
Thanks for your help.
Peter Hansen
> Well its an anomaly. I sent to bug list.
> Probably never see it again.
> I think some sort of semaphore thingy, which I know
> nothing about, is sneaking in under unreproducible
> conditions. I'm moving on.
If you want to try one more thing, try mucking with
a call to sys.setcheckinterval(). This used to be
10 by default but is now 100. In any case, changing
it will quite often significantly change the code's
behaviour in the presence of race conditions. Try
values of 1 and 1000 and see if anything makes a
difference.
-Peter
Bengt Richter
>And sorry I got ticked, frustrating week
>
threading problems can do that ;-)
You are obviusly deeper into this than I can get from a cursory scan,
but I'll make some general debugging comments ;-)
> >And I could help more, being fairly experienced with
> >threading issues and race conditions and such, but
> >as I tried to indicate in the first place, you've
> >provided next to no useful (IMHO) information to
> >let anyone help you more than this
>
>This is about 5% of the code.
>Uses no locks.
>I am mystified, I have written probably 100,000 lines
>of Python and never seen a thread just lock up and quit
>running. It happens on a Queue() statement so my suspicion
>is a bug. ??
Or a once-in-a-blue-moon resource access deadlock of some kind?
>
>I have kludged around it by putting all the thread/queue stuff
>in the main program and import the stuff I don't want to
>distribute. But mysteries haunt your dreams, sooo...
>#!/usr/local/bin/python
>
># pnetx.py
>
>from threading import *
>from time import *
>from Queue import Queue
>from socket import *
Do you really need to import * ? Though the above should be safe import-wise
you are setting yourself up for name-clash problems by putting so many in
the same space. E.g., if something happened to match a misspelling typo in
your program, you wouldn't get a name error exception. Etc., etc.
One of the first rules of debugging is to eliminate the unnecessary ;-)
At least there doesn't seem to be name clashes between the above (except
starting with '_' which shouldn't get imported with *
>>> d = {}
>>> for imp in 'threading time Queue socket'.split():
... m = __import__(imp)
... for name in m.__dict__.keys():
... d.setdefault(name, []).append(imp)
...
>>> for k,v in d.items():
... if len(v)!=1: print k,v
...
_sleep ['threading', 'Queue']
__file__ ['threading', 'Queue', 'socket']
__all__ ['threading', 'Queue', 'socket']
__builtins__ ['threading', 'Queue', 'socket']
__name__ ['threading', 'time', 'Queue', 'socket']
_time ['threading', 'Queue']
__doc__ ['threading', 'time', 'Queue', 'socket']
Just verifying an assumption ;-)
OTOH, do you really need (ignoring wid and name and the first content of dir()):
>>> dir()
['__builtins__', '__doc__', '__name__']
>>> from threading import *
>>> from time import *
>>> from Queue import *
>>> from socket import *
>>> wid = 0
>>> for name in dir():
... print repr(name),
... wid += len(repr(name))+1
... if wid>60:
... print
... wid = 0
...
'AF_APPLETALK' 'AF_INET' 'AF_IPX' 'AF_UNSPEC' 'AI_ADDRCONFIG'
'AI_ALL' 'AI_CANONNAME' 'AI_DEFAULT' 'AI_MASK' 'AI_NUMERICHOST'
'AI_PASSIVE' 'AI_V4MAPPED' 'AI_V4MAPPED_CFG' 'BoundedSemaphore'
'CAPI' 'Condition' 'EAI_ADDRFAMILY' 'EAI_AGAIN' 'EAI_BADFLAGS'
'EAI_BADHINTS' 'EAI_FAIL' 'EAI_FAMILY' 'EAI_MAX' 'EAI_MEMORY'
'EAI_NODATA' 'EAI_NONAME' 'EAI_PROTOCOL' 'EAI_SERVICE' 'EAI_SOCKTYPE'
'EAI_SYSTEM' 'Empty' 'Event' 'Full' 'INADDR_ALLHOSTS_GROUP' 'INADDR_ANY'
'INADDR_BROADCAST' 'INADDR_LOOPBACK' 'INADDR_MAX_LOCAL_GROUP'
'INADDR_NONE' 'INADDR_UNSPEC_GROUP' 'IPPORT_RESERVED' 'IPPORT_USERRESERVED'
'IPPROTO_GGP' 'IPPROTO_ICMP' 'IPPROTO_IDP' 'IPPROTO_IGMP' 'IPPROTO_IP'
'IPPROTO_MAX' 'IPPROTO_ND' 'IPPROTO_PUP' 'IPPROTO_RAW' 'IPPROTO_TCP'
'IPPROTO_UDP' 'IP_ADD_MEMBERSHIP' 'IP_DEFAULT_MULTICAST_LOOP'
'IP_DEFAULT_MULTICAST_TTL' 'IP_DROP_MEMBERSHIP' 'IP_MAX_MEMBERSHIPS'
'IP_MULTICAST_IF' 'IP_MULTICAST_LOOP' 'IP_MULTICAST_TTL' 'IP_OPTIONS'
'IP_TOS' 'IP_TTL' 'Lock' 'MSG_DONTROUTE' 'MSG_OOB' 'MSG_PEEK'
'NI_DGRAM' 'NI_MAXHOST' 'NI_MAXSERV' 'NI_NAMEREQD' 'NI_NOFQDN'
'NI_NUMERICHOST' 'NI_NUMERICSERV' 'Queue' 'RAND_add' 'RAND_egd'
'RAND_status' 'RLock' 'SOCK_DGRAM' 'SOCK_RAW' 'SOCK_RDM' 'SOCK_SEQPACKET'
'SOCK_STREAM' 'SOL_IP' 'SOL_SOCKET' 'SOL_TCP' 'SOL_UDP' 'SOMAXCONN'
'SO_ACCEPTCONN' 'SO_BROADCAST' 'SO_DEBUG' 'SO_DONTROUTE' 'SO_ERROR'
'SO_KEEPALIVE' 'SO_LINGER' 'SO_OOBINLINE' 'SO_RCVBUF' 'SO_RCVLOWAT'
'SO_RCVTIMEO' 'SO_REUSEADDR' 'SO_SNDBUF' 'SO_SNDLOWAT' 'SO_SNDTIMEO'
'SO_TYPE' 'SO_USELOOPBACK' 'SSLType' 'SSL_ERROR_EOF' 'SSL_ERROR_INVALID_ERROR_CODE'
'SSL_ERROR_SSL' 'SSL_ERROR_SYSCALL' 'SSL_ERROR_WANT_CONNECT'
'SSL_ERROR_WANT_READ' 'SSL_ERROR_WANT_WRITE' 'SSL_ERROR_WANT_X509_LOOKUP'
'SSL_ERROR_ZERO_RETURN' 'Semaphore' 'SocketType' 'TCP_NODELAY'
'Thread' 'Timer' '__builtins__' '__doc__' '__name__' 'accept2dyear'
'activeCount' 'altzone' 'asctime' 'clock' 'ctime' 'currentThread'
'daylight' 'enumerate' 'error' 'errorTab' 'gaierror' 'getaddrinfo'
'getdefaulttimeout' 'getfqdn' 'gethostbyaddr' 'gethostbyname'
'gethostbyname_ex' 'gethostname' 'getnameinfo' 'getprotobyname'
'getservbyname' 'gmtime' 'has_ipv6' 'herror' 'htonl' 'htons'
'inet_aton' 'inet_ntoa' 'localtime' 'mktime' 'ntohl' 'ntohs'
'setdefaulttimeout' 'setprofile' 'settrace' 'sleep' 'socket'
'ssl' 'sslerror' 'strftime' 'strptime' 'struct_time' 'time' 'timeout'
'timezone' 'tzname' 'wid'
>>>
(Hm, should have pre-tested wid+len(current) to limit width ;-)
>import sys
>import os
>
># glob is a DUMMY CLASS
where from? accidentally snipped?
>
>glob.listenq = Queue(1000)
>
>def listener():
> while 1:
> msg,addrport = listenersocket.recvfrom(BUFSIZE)
> addr = addrport[0]
what is the usage level on glob.listenq? It is finite 1000, so
you could theoretically block on the put. What do you get
if you print glob.listenq.qsize here?
> glob.listenq.put (( msg,addr),1)
> if msg == 'DIE': break
>
>def procmsgs():
> while 1:
> msg,addr = glob.listenq.get(1)
> if msg == 'DIE': break
Might be interesting to get some output when 'DIE' is recognized. Above too.
> wds = msg.split(';')
assert wds[0] in ['S', 'E', 'ACK', 'MONITOR'] # check assumptions. E.g.,
# split on ';' means spaces around
# the ';' are not eliminated
> if wds[0] == 'S': recvsucc(msg); continue
> if wds[0] == 'E': recvevent(msg); continue
> if wds[0] == 'ACK': recvack(msg[4:]); continue
> if wds[0] == 'MONITOR':
> if addr not in monitorlist:
> print 'This line ALWAYS PRINTS'
> queuelist.append( Queue(0) )
Why are you apparently creating *new* queues in this thread that you are not using
and appending them to queuelist in the main thread? Are you testing to see how many
can be created, or for some Queue creation bug?
What do you get if you print 'queuelist length = %s'%len(queuelist) here?
How often are you getting 'MONITOR'?
> ## The above fails if this code is imported
Imported from where? Do you mean that this code works if it
is embedded in some larger code, and if you put an import of this
at the top of that code it doesn't work? That would not only be
moving the source, but also the point of invocation. I.e., the
import *executes* what you import once, putting the result in
the module's global name space (other than code that explicitly
accesses other modules etc.) Do you need to wrap part of this
in a function that you can call *after* importing the definitions?
If your real app has multiple threads accessing a main thread queuelist
without a lock, you might want to look closely and see if that needs
to be a queue mechanism itself.
> ## It doesn't matter whether it is imported
> ## as .py or .pyc
> ## also mq = Queue(0); queuelist.append(mq) # fails
> print 'This line ALWAYS PRINTS if i execute this source'
> print 'but NEVER PRINTS if I import '
> print 'this source from a 1 line program'
> print 'Thats weird'
> monitoron.append( 1 )
> monitorlist.append( addr )
>
Do you want all the following to be executed once at the point of importing this?
>queuelist = [Queue(0)]
What is this for? (oops, I hope I restored the above line correctly)
>
>
>listenersocket = socket(AF_INET,SOCK_DGRAM)
>listenersocket.bind(ListenAddr)
>
>procm = Thread(target=procmsgs)
>procm.start()
>listenr = Thread(target=listener)
>listenr.start()
>
>## Then start a bunch of other threads and queuea.
>
>
>Don't spend a lot of time on this, not worth it.
>I just thought someone might have experienced
>something similar.
>
Just some thoughts. HTH.
Regards,
Bengt Richter
Peter Otten
> And sorry I got ticked, frustrating week
>
> >And I could help more, being fairly experienced with
> >threading issues and race conditions and such, but
> >as I tried to indicate in the first place, you've
> >provided next to no useful (IMHO) information to
> >let anyone help you more than this
>
> This is about 5% of the code.
> Uses no locks.
A lock is acquired in Queue.__init__().
> I am mystified, I have written probably 100,000 lines
> of Python and never seen a thread just lock up and quit
> running. It happens on a Queue() statement so my suspicion
> is a bug. ??
>
> I have kludged around it by putting all the thread/queue stuff
> in the main program and import the stuff I don't want to
> distribute. But mysteries haunt your dreams, sooo...
What I believe to be a minimal example:
<freeze.py>
import Queue
import threading
import time
q = Queue.Queue(4)
def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"
threading.Thread(target=proc).start()
while True:
print "yadda"
q.put(None)
time.sleep(1)
</freeze.py>
<freezemain.py>
import freeze
</freezemain.py>
Invoking freezemain.py produces
yadda
yadda
yadda
yadda
yadda
yadda
i. e. consistently q.maxsize + 2. One item about to be put, one already
taken before Queue.Queue(). Deferring execution of the module-level code
until after the import
<nofreeze.py>
import Queue
import threading
import time
q = Queue.Queue(4)
def proc():
while True:
q.get(1)
Queue.Queue()
print "YADDA"
def start():
threading.Thread(target=proc).start()
while True:
print "yadda"
q.put(None)
time.sleep(1)
</nofreeze.py>
<nofreezemain.py>
import nofreeze
nofreeze.start()
</nofreezemain.py>
and invoking nofreezemain.py produces
yadda
YADDA
yadda
YADDA
yadda
YADDA
yadda
YADDA
apparently ad infinitum. Conclusion? None so far, though you might welcome
the confirmation that this is not a "once in a blue moon" accident as Bengt
Richter surmised.
Import _is_ a sensitive phase...
As an experiment I moved
try:
import thread
except ImportError:
import dummy_thread as thread
from the Queue.Queue.__init__() method to the module body -- and now
freezemain.py seems to work, too. So that would be an easy remedy, but sure
there is a reason why that import statement is in such an unusual place?
Peter
Tim Peters
> What I believe to be a minimal example:
>
> <freeze.py>
> import Queue
> import threading
> import time
>
> q = Queue.Queue(4)
>
> def proc():
> while True:
> q.get(1)
> Queue.Queue()
> print "YADDA"
>
> threading.Thread(target=proc).start()
>
> while True:
> print "yadda"
> q.put(None)
> time.sleep(1)
> </freeze.py>
>
> <freezemain.py>
> import freeze
> </freezemain.py>
CPython has an internal, reentrant import lock. When a thread does an
import, it acquires this lock, and the lock remains held until that
import is complete. As a consequence, no *other* thread can do an
import (it blocks waiting to obtain the internal import lock) until
the original import completes. So until "import freeze" returns in
the main thread, no other thread can do an import.
Partly for that reason, it's generally a Horrible Idea to start a
thread as a side effect of importing a module. That's what freeze.py
does, and you get the expected deadlock as a result. The main thread
is hung waiting for "import freeze" to return, and the spawned thread
is hung at an import in Queue.__init__() waiting for the main thread
to release the import lock.
Right. Note that this is the same reason threaded tests in Python's
standard regression suite define a 'test_main()' function, called by
the regrtest.py driver after import of the test's module completes.
It's generally suicidal to start a thread as a side effect of an
import.
...
> Import _is_ a sensitive phase...
It's quite easy to avoid thread problems in imports: never start a
thread as a side effect of importing, and you'll never get a deadlock
due to importing.
> As an experiment I moved
>
> try:
> import thread
> except ImportError:
> import dummy_thread as thread
>
> from the Queue.Queue.__init__() method to the module body --
> and now freezemain.py seems to work, too. So that would be an
> easy remedy, but sure there is a reason why that import
> statement is in such an unusual place?
I think you'd have to ask Brett (who did most of the work on
dummy_thread and dummy_threading). It doesn't really matter, though:
it's a general truth that starting a thread as a side effect of
importing is a recipe for deadlock, and hacking specific methods and
functions to avoid imports just moves the problem around. It's not a
goal that anything in the standard Python library cater to bad thread
practice here (the bad thread practice being, again, starting a thread
as a side effect of importing).
Steve Holden
[...]
>
> > 5. Sorry I can't be more help. You don't give anyone much
> > to go on. All that stuff about "Queue(0)" and "listenq"
> > is pretty much meaningless to us, you know...
>
> You know, I get this all the time on language support groups.
This might be a clue.
> All of my Linux support groups, if they don't understand, say
> why and ask for elaboration. I tried to explain what was going on,
> without incuding the source. I have about 200 man hours in the source
> and I bet it would take longer to understand it.
> If my explanation was insufficient, I'm sorry.
>
It *was* pretty sketchy. Basically you shouldn't expect people to debug
code they can't see. There are some very able programmers active on this
newsgroup, but their psychic abilities are limited :-)
> ALSO, you did not respond to my email, so I didn't know how to reply.
> There is nothing on the archives page which gives me a clue as to
> how to respond.
>
Once yo post in the newsgroup (or mailing list) then use of the same
channel is advisable.
> SO, if there is ZERO chance there is some sort of inadvertent lock
> occuring in saved byte code, I just kludge around and move on.
> Maybe 2.4
>
If you suspect the problem is a bug in Python then you might consider
searching the bugs database on sourceforge. If nobody's reported a bug
then you are being very optimistic assuming that migrating to 2.4 will
fix the problem.
> Thanks.
>
>
regards
Steve
--
http://www.holdenweb.com
http://pydish.holdenweb.com
Holden Web LLC +1 800 494 3119
Antoon Pardon
>
> I think you'd have to ask Brett (who did most of the work on
> dummy_thread and dummy_threading). It doesn't really matter, though:
> it's a general truth that starting a thread as a side effect of
> importing is a recipe for deadlock, and hacking specific methods and
> functions to avoid imports just moves the problem around. It's not a
> goal that anything in the standard Python library cater to bad thread
> practice here (the bad thread practice being, again, starting a thread
> as a side effect of importing).
I don't see why starting a thread as a side effect of importing is
bad thread practice. Sure python doesn't cater for it, but IMO
that seems to be python failing.
--
Antoon Pardon
Tim Peters
> I don't see why starting a thread as a side effect of importing is
> bad thread practice. Sure python doesn't cater for it, but IMO
> that seems to be python failing.
Obviously, it's bad practice in Python because it can lead to
deadlocks in Python. It's nearly tautological. Import is an
executable statement in Python, not, e.g., as in many other languages,
a declaration directed at the system linker. With that power comes
opportunities to shoot yourself, although they're generally easy to
avoid. Come up with a practical design that doesn't have this
limitation, and then perhaps your characterization of the current
design as "a failing" would be both credible and constructive.
Apart from that, ya, I do think it would *uisually* be poor practice
to start a thread as a side effect of importing anyway. It's too
mysterious, and IME has caused trouble even when it didn't lead to
deadlocks. The fundamental purpose of import in Python is to add
useful names to the importer's namespace, and users of a module
generally think of it as doing no more than that.
Note that the OP's example had a module that, upon the first attempt
to import it, ran an infinite loop (even if it hadn't deadlocked), and
it's clearly severe abuse of import's purpose.to write a module M such
that "import M" *never* returns. Indeed, that's the other half of how
deadlock occurs: not only that the imported module spawn a thread as
a side effect of importing, but also that the imported module refuse
to allow the import to complete.
The current design actually supports spawning all the threads you like
as a side effect of importing, provided you ensure also that the
import ompletes. The easiest way to avoid trouble remains not to
spawn threads as a side effect of importing to begin with, although a
programmer determined to demonstrate their bad taste <wink> can easily
enough make it work.
Antoon Pardon
> [Antoon Pardon]
>> I don't see why starting a thread as a side effect of importing is
>> bad thread practice. Sure python doesn't cater for it, but IMO
>> that seems to be python failing.
>
> Obviously, it's bad practice in Python because it can lead to
> deadlocks in Python.
By that argument any use of locks is bad practice because it
can lead to deadlock.
> It's nearly tautological. Import is an
> executable statement in Python, not, e.g., as in many other languages,
> a declaration directed at the system linker. With that power comes
> opportunities to shoot yourself, although they're generally easy to
> avoid. Come up with a practical design that doesn't have this
> limitation, and then perhaps your characterization of the current
> design as "a failing" would be both credible and constructive.
If a car model has cranky brakes, I think I can call that a failing
even without having the ability to come up with a pratical design
that doesn's has those limitations.
I judge a language by what it can and cannot do, not by my ability
to correct the things I perceive as failings. For all I know python
may have taken some design decisions that might have seen perfectly
logical but now prohibit a a practical design that doesn't have this
limitation. I don't see why something like that would make this
any less a failing then when a practical design was easy in the
current implemenation.
> Apart from that, ya, I do think it would *uisually* be poor practice
> to start a thread as a side effect of importing anyway. It's too
> mysterious, and IME has caused trouble even when it didn't lead to
> deadlocks. The fundamental purpose of import in Python is to add
> useful names to the importer's namespace, and users of a module
> generally think of it as doing no more than that.
Importing a module in general also does some kind of initialisation.
If starting a thread is a logical thing to do in this initialisation
fase I see nothing wrong with it.
> Note that the OP's example had a module that, upon the first attempt
> to import it, ran an infinite loop (even if it hadn't deadlocked), and
> it's clearly severe abuse of import's purpose.to write a module M such
> that "import M" *never* returns. Indeed, that's the other half of how
> deadlock occurs: not only that the imported module spawn a thread as
> a side effect of importing, but also that the imported module refuse
> to allow the import to complete.
Well I'll agree here. An import that has as a side effect that the
import doesn't return is bad practice.
> The current design actually supports spawning all the threads you like
> as a side effect of importing, provided you ensure also that the
> import ompletes.
Well in that case I don't have any problems with it. The perceived
failing was because of only knowing part of it based on what I had
read in this thread.
> The easiest way to avoid trouble remains not to
> spawn threads as a side effect of importing to begin with, although a
> programmer determined to demonstrate their bad taste <wink> can easily
> enough make it work.
Well then probably I have a bad taste. I have though of designs in which
it seemed very natural to have a thread started as part of the
initialisation in a module. Other limitations of python didn't make
it workable but in priciple I saw nothing wrong with doing it.
--
Antoon Pardon
Steve Holden
> Op 2004-12-13, Tim Peters schreef <tim.p...@gmail.com>:
>
>>[Antoon Pardon]
>>
>>>I don't see why starting a thread as a side effect of importing is
>>>bad thread practice. Sure python doesn't cater for it, but IMO
>>>that seems to be python failing.
>>
>>Obviously, it's bad practice in Python because it can lead to
>>deadlocks in Python.
>
>
> By that argument any use of locks is bad practice because it
> can lead to deadlock.
>
Not at all. You mentioned locks, not Tim. The reason it can lead to
deadlocks is because import, being an executable statement, must
terminate correctly in order to return control to the module that
executed the import statement.
>
>>It's nearly tautological. Import is an
>>executable statement in Python, not, e.g., as in many other languages,
>>a declaration directed at the system linker. With that power comes
>>opportunities to shoot yourself, although they're generally easy to
>>avoid. Come up with a practical design that doesn't have this
>>limitation, and then perhaps your characterization of the current
>>design as "a failing" would be both credible and constructive.
>
>
> If a car model has cranky brakes, I think I can call that a failing
> even without having the ability to come up with a pratical design
> that doesn's has those limitations.
>
But in fact your situation is more closely analogous to a customer who's
bought a car that can be stopped by pressing on the brake pedal now
complaining that sideways pressure on the brake pedal doesn;t affect the
car's motion.
> I judge a language by what it can and cannot do, not by my ability
> to correct the things I perceive as failings. For all I know python
> may have taken some design decisions that might have seen perfectly
> logical but now prohibit a a practical design that doesn't have this
> limitation. I don't see why something like that would make this
> any less a failing then when a practical design was easy in the
> current implemenation.
>
All that Tim was suggesting is that it's MORE SENSIBLE to start a thread
as the result of a specific call to programmed functionality rather than
as the side effect of an import. The reason for this is due to the
semantics of the import statement. If you perceive that as a failing
then you'd be better rewarded by an attempt to modify your perceptions.
I've always found "don't argue with Tim about Python" to be a useful
rule of thumb. He's wrong much less often than I am. I suspect he's also
wrong much less often than you ;-)
>
>>Apart from that, ya, I do think it would *uisually* be poor practice
>>to start a thread as a side effect of importing anyway. It's too
>>mysterious, and IME has caused trouble even when it didn't lead to
>>deadlocks. The fundamental purpose of import in Python is to add
>>useful names to the importer's namespace, and users of a module
>>generally think of it as doing no more than that.
>
>
> Importing a module in general also does some kind of initialisation.
> If starting a thread is a logical thing to do in this initialisation
> fase I see nothing wrong with it.
>
But the reason, I suspect, that it's being suggested this is bad design
is because it would be tricky (in any language) to ensure that the
satisfactory conclusion of the import didn't depend on successful thread
startup (and possibly termination).
Plus, of course, in Python the side-effect of the import (creating the
thread) can happen precisely once, because the module body is executed
precisely once no matter how many times it's imported. That's not really
the issue, however.
>
>>Note that the OP's example had a module that, upon the first attempt
>>to import it, ran an infinite loop (even if it hadn't deadlocked), and
>>it's clearly severe abuse of import's purpose.to write a module M such
>>that "import M" *never* returns. Indeed, that's the other half of how
>>deadlock occurs: not only that the imported module spawn a thread as
>>a side effect of importing, but also that the imported module refuse
>>to allow the import to complete.
>
>
> Well I'll agree here. An import that has as a side effect that the
> import doesn't return is bad practice.
>
But that's precisely the risk you run when starting up threads!
>
>>The current design actually supports spawning all the threads you like
>>as a side effect of importing, provided you ensure also that the
>>import ompletes.
>
>
> Well in that case I don't have any problems with it. The perceived
> failing was because of only knowing part of it based on what I had
> read in this thread.
>
>
>>The easiest way to avoid trouble remains not to
>>spawn threads as a side effect of importing to begin with, although a
>>programmer determined to demonstrate their bad taste <wink> can easily
>>enough make it work.
>
>
> Well then probably I have a bad taste. I have though of designs in which
> it seemed very natural to have a thread started as part of the
> initialisation in a module. Other limitations of python didn't make
> it workable but in priciple I saw nothing wrong with doing it.
>
Bearing in mind that module initialisation is always one-off, relying on
import to trigger such complex behavior is probably a design that will
mislead users into false expectations.
regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Antoon Pardon
> Antoon Pardon wrote:
>
>> Op 2004-12-13, Tim Peters schreef <tim.p...@gmail.com>:
>>
>>>[Antoon Pardon]
>>>
>>>>I don't see why starting a thread as a side effect of importing is
>>>>bad thread practice. Sure python doesn't cater for it, but IMO
>>>>that seems to be python failing.
>>>
>>>Obviously, it's bad practice in Python because it can lead to
>>>deadlocks in Python.
>>
>>
>> By that argument any use of locks is bad practice because it
>> can lead to deadlock.
>>
> Not at all. You mentioned locks, not Tim.
That is beside the point. The argument was that starting a thread
as a side effect of importing is bad practice because it can lead to
deadlock. This suggest that a general condition for something
to be bad practice is if that something can lead to deadlock.
Locks are in that case.
>>>It's nearly tautological. Import is an
>>>executable statement in Python, not, e.g., as in many other languages,
>>>a declaration directed at the system linker. With that power comes
>>>opportunities to shoot yourself, although they're generally easy to
>>>avoid. Come up with a practical design that doesn't have this
>>>limitation, and then perhaps your characterization of the current
>>>design as "a failing" would be both credible and constructive.
>>
>>
>> If a car model has cranky brakes, I think I can call that a failing
>> even without having the ability to come up with a pratical design
>> that doesn's has those limitations.
>>
> But in fact your situation is more closely analogous to a customer who's
> bought a car that can be stopped by pressing on the brake pedal now
> complaining that sideways pressure on the brake pedal doesn;t affect the
> car's motion.
You will have to explain how you come by that analogy.
>> I judge a language by what it can and cannot do, not by my ability
>> to correct the things I perceive as failings. For all I know python
>> may have taken some design decisions that might have seen perfectly
>> logical but now prohibit a a practical design that doesn't have this
>> limitation. I don't see why something like that would make this
>> any less a failing then when a practical design was easy in the
>> current implemenation.
>>
> All that Tim was suggesting is that it's MORE SENSIBLE to start a thread
> as the result of a specific call to programmed functionality rather than
> as the side effect of an import. The reason for this is due to the
> semantics of the import statement. If you perceive that as a failing
> then you'd be better rewarded by an attempt to modify your perceptions.
> I've always found "don't argue with Tim about Python" to be a useful
> rule of thumb. He's wrong much less often than I am. I suspect he's also
> wrong much less often than you ;-)
With all respect I find that lousy advise. I don't mind that it will turn
out I'll be wrong most of the time. But I'll probably will have gained
a better understanding by the responses to my argument than by merely
accepting the word of Tim.
[ ... ]
>>>Note that the OP's example had a module that, upon the first attempt
>>>to import it, ran an infinite loop (even if it hadn't deadlocked), and
>>>it's clearly severe abuse of import's purpose.to write a module M such
>>>that "import M" *never* returns. Indeed, that's the other half of how
>>>deadlock occurs: not only that the imported module spawn a thread as
>>>a side effect of importing, but also that the imported module refuse
>>>to allow the import to complete.
>>
>>
>> Well I'll agree here. An import that has as a side effect that the
>> import doesn't return is bad practice.
>>
> But that's precisely the risk you run when starting up threads!
Now you are confusing me. Is it a problem of an import that doesn't
return of is it a case of a race condition where the import has to
return in a timely fashion?
--
Antoon Pardon