killing thread ?
Uwe Schmitt
Hi,
AFAIK there is no method in the threading module to kill a thread.
Each thread has to terminate itself by watching status variables.
But: what happens if I start a thread, and when I want to kill
this thread, I delete all references to the according Thread object,
and call the garbage collector.
Will this work ??
My motivation is to watch some functions, where I have no
access to the functions implementation.
And another question: how fast is the garbage collector ?
Greetings, Uwe.
--
Dr. rer. nat. Uwe Schmitt Computer science is no more about Computers,
uwe.s...@num.uni-sb.de than astronomy is about telescopes. (Dijkstra)
http://www.procoders.net
Martin v. Löwis
> AFAIK there is no method in the threading module to kill a thread.
Correct.
> Will this work ??
No. The Thread object is just a wrapper around the underlying system
resource. Deleting the Thread object won't kill the thread, since
thread cancellation is not supported.
> And another question: how fast is the garbage collector ?
Answer: very fast :-)
Regards,
Martin
Erik Max Francis
> AFAIK there is no method in the threading module to kill a thread.
> Each thread has to terminate itself by watching status variables.
> But: what happens if I start a thread, and when I want to kill
> this thread, I delete all references to the according Thread object,
> and call the garbage collector.
> Will this work ??
No. A thread needs to suicide cooperatively; have it, say, periodically
check a flag that, when set, will cause the thread to stop itself. When
you want to kill the thread, set the flag.
> And another question: how fast is the garbage collector ?
If you're asking how quickly __del__ methods get called, it's an
implementation detail. You're not even really guaranteed that the
__del__ method will ever get called under some circumstances. (Either
way it's a side issue here, even removing the last reference to a thread
will not kill it.)
--
Erik Max Francis / m...@alcyone.com / http://www.alcyone.com/max/
__ San Jose, CA, USA / 37 20 N 121 53 W / &tSftDotIotE
/ \ I am a gentlemen: I live by robbing the poor.
\__/ George Bernard Shaw
Kepler's laws / http://www.alcyone.com/max/physics/kepler/
A proof of Kepler's laws.
Laura Creighton
> Hi,
>
> AFAIK there is no method in the threading module to kill a thread.
> Each thread has to terminate itself by watching status variables.
> But: what happens if I start a thread, and when I want to kill
> this thread, I delete all references to the according Thread object,
> and call the garbage collector.
> Will this work ??
No. You cannot stand outside your application and kill a thread.
Instead you have to arrange for your thread to receive the message
to go commit suicide (and then your thread has to know how to do
that). If your threads are already communicating by Queues, sticking
a 'do_die' in there is often cleanest.
<snip>
> Greetings, Uwe.
>
> --
> Dr. rer. nat. Uwe Schmitt Computer science is no more about Computers,
> uwe.s...@num.uni-sb.de than astronomy is about telescopes. (Dijkstra)
> http://www.procoders.net
> --
> http://mail.python.org/mailman/listinfo/python-list
Laura Creighton
Uwe Schmitt
>>
>> Hi,
>>
>> AFAIK there is no method in the threading module to kill a thread.
>> Each thread has to terminate itself by watching status variables.
>> But: what happens if I start a thread, and when I want to kill
>> this thread, I delete all references to the according Thread object,
>> and call the garbage collector.
>> Will this work ??
> No. You cannot stand outside your application and kill a thread.
> Instead you have to arrange for your thread to receive the message
> to go commit suicide (and then your thread has to know how to do
> that). If your threads are already communicating by Queues, sticking
> a 'do_die' in there is often cleanest.
> <snip>
But what if I want to end a thread which hangs because
it uses 'exec' on some code which loops forever ?
I discovered the _Thread__stop() method of threading.Thread object.
It works. But that seems to be some dirty hack...
Greetings, Uwe
"Martin v. Löwis"
> But what if I want to end a thread which hangs because
> it uses 'exec' on some code which loops forever ?
You lose. There is no way to cancel a thread, period.
> I discovered the _Thread__stop() method of threading.Thread object.
> It works. But that seems to be some dirty hack...
What do you mean by "it works"? It does not terminate the thread (just
read its source code if you don't believe me). If you call it in the
context of another thread, it has all kinds of funny effects, which is
why it is a private method (and really spelled __stop).
Regards,
Martin
Irmen de Jong
> Uwe Schmitt wrote:
>
>> But what if I want to end a thread which hangs because
>> it uses 'exec' on some code which loops forever ?
>
>
> You lose. There is no way to cancel a thread, period.
>
Can't you cheat on linux by sending SIGKILL to the thread's PID?? ;-)
Irmen
Paul Rubin
> > But what if I want to end a thread which hangs because
> > it uses 'exec' on some code which loops forever ?
>
> You lose. There is no way to cancel a thread, period.
Is there some inherent reason for that? Or is it just a deficiency,
either in Python or in the way OS threads work? I don't understand
why Python's thread switching code can't just look for a flag
indicating that the thread should be killed instead of allowed to run.
Peter Hansen
>
> But what if I want to end a thread which hangs because
> it uses 'exec' on some code which loops forever ?
What does the code do in the loop? If it calls some other
routines, you could perhaps find a way to dynamically insert
a "raise SystemExit()" into that subroutine, thereby causing
the thread to terminate (assuming it does *not* catch this
exception!) even without changing the code therein.
If you're worried about some kind of denial of service type
thing with code written by someone else: you're out of luck.
Don't use Python. (Or, better yet, get over the worry and just
accept this possibility.)
> I discovered the _Thread__stop() method of threading.Thread object.
> It works. But that seems to be some dirty hack...
As Martin said, it does not work for what you want. It's just
an internal utility function called *when a thread is already
stopping*.
-Peter
"Martin v. Löwis"
> Can't you cheat on linux by sending SIGKILL to the thread's PID?? ;-)
Only in the current implementation. In NGPTL (next generation's posix
threads), all threads will have the same PID, but different tids.
Likewise, on Windows, you can probably use some Win32 API, if that is
exposed through PythonWin.
Regards,
Martin
"Martin v. Löwis"
> Is there some inherent reason for that? Or is it just a deficiency,
> either in Python or in the way OS threads work? I don't understand
> why Python's thread switching code can't just look for a flag
> indicating that the thread should be killed instead of allowed to run.
Python does not implement a thread switching code; these are operating
system threads.
If you are thinking that the interpreter loop could look whether it
shall terminate: this works only if the thread is not in a blocking
system call.
There are thread cancellation functions in some of the underlying thread
APIs. The main reason why those are not used is that they don't
implement proper stack unwinding. In Python, you have to explicitly
complete all C stack frames, because they may need to DECREF objects (in
particular, argument tuples). So a plain thread termination, as Irmen
suggests it, leaves garbage behind.
If these issues (blocking system calls and proper unwinding) are
addressed, the feature could be provided. Most likely, it would be
exposed as an exception that appears out of nothing in the target
thread, so if the target thread choses to catch the exception, you still
couldn't terminate it.
Regards,
Martin
Paul Rubin
> If you're worried about some kind of denial of service type
> thing with code written by someone else: you're out of luck.
> Don't use Python. (Or, better yet, get over the worry and just
> accept this possibility.)
Wha? Python provides the rexec and bastion modules specifically to
have a way to run hostile code in a sandbox. So what kind of advice
is "don't use Python"? If sandboxed code can be prevented from
accessing sensitive data or doing i/o, but can't be prevented from
looping unkillably, that's a bug in Python.
Skip Montanaro
>> You lose. There is no way to cancel a thread, period.
Paul> Is there some inherent reason for that?
Not all threads packages on which Python runs implement the desired
functionality. This topic has come up before, and the answer has always
been the same. Here are some references:
http://mail.python.org/pipermail/python-dev/2001-May/014980.html
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&th=dc349060ffdf6b24&seekm=a5ld2m%24554%241%40panix2.panix.com&frame=off
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&th=15e0381c34730afc&rnum=4
Skip
Laura Creighton
> >>
> >> Hi,
> >>
> >> AFAIK there is no method in the threading module to kill a thread.
> >> Each thread has to terminate itself by watching status variables.
> >> But: what happens if I start a thread, and when I want to kill
> >> this thread, I delete all references to the according Thread object,
> >> and call the garbage collector.
> >> Will this work ??
>
> > No. You cannot stand outside your application and kill a thread.
> > Instead you have to arrange for your thread to receive the message
> > to go commit suicide (and then your thread has to know how to do
> > that). If your threads are already communicating by Queues, sticking
> > a 'do_die' in there is often cleanest.
>
> > <snip>
>
> But what if I want to end a thread which hangs because
> it uses 'exec' on some code which loops forever ?
You lose. That thread doesn't know how to receive a message that it
ought to die.
>
> I discovered the _Thread__stop() method of threading.Thread object.
> It works. But that seems to be some dirty hack...
I don't think that this does what you think it does. If you call it in
the context of another thread, all sorts of weird things happen, but
none of them is what you want. What you want is a way to stand outside
of your program and shoot the running process that you don't need any
more, and _you_ _can't_ _have_ _that_, because there is no way to do
that, Sorry.
> Dr. rer. nat. Uwe Schmitt Computer science is no more about Computers,
> uwe.s...@num.uni-sb.de than astronomy is about telescopes. (Dijkstra)
> http://www.procoders.net
Laura
Paul Rubin
> operating system threads.
>
> If you are thinking that the interpreter loop could look whether it
> shall terminate: this works only if the thread is not in a blocking
> system call.
How does this sound:
The interpreter loop can check whether there's a kill pending for
any other threads (that's just looking at a flag that will be false
almost all the time).
If there's a kill pending, the interpreter sends a signal to
the target thread. That should unblock any blocked i/o.
A global signal handler then raises an exception in the
target thread which gets handled the usual way.
That does leave the target thread able to catch the exception
and keep looping. Even still, it improves on the present situation.
I can imagine some ways to hair up the exception mechanism to stop
that from happening, but that starts to mess with exception semantics
rather deeply.
Paul Rubin
> >> You lose. There is no way to cancel a thread, period.
>
> Paul> Is there some inherent reason for that?
>
> Not all threads packages on which Python runs implement the desired
> functionality.
Well, there's all kinds of functionality that some OS's implement and
others don't. Python should use the functionality when it's
available.
"Martin v. Löwis"
> The interpreter loop can check whether there's a kill pending for
> any other threads (that's just looking at a flag that will be false
> almost all the time).
No need for that: the implementation of thread.cancel could do that
right away. However, it cannot know whether the target thread is in a
blocking system call - nor could the interpreter loop of another thread
know for sure.
> If there's a kill pending, the interpreter sends a signal to
> the target thread. That should unblock any blocked i/o.
Which signal specifically?
> A global signal handler then raises an exception in the
> target thread which gets handled the usual way.
What if the signal is blocked?
> That does leave the target thread able to catch the exception
> and keep looping. Even still, it improves on the present situation.
How do I do that on Windows? On HP-UX? On Irix? With Gnu threads?
Regards,
Martin
Paul Rubin
> > If there's a kill pending, the interpreter sends a signal to
> > the target thread. That should unblock any blocked i/o.
>
> Which signal specifically?
[phr@localhost]$ kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL
5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE
9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2
13) SIGPIPE 14) SIGALRM 15) SIGTERM 17) SIGCHLD
18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN
22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO
30) SIGPWR 31) SIGSYS 32) SIGRTMIN 33) SIGRTMIN+1
34) SIGRTMIN+2 35) SIGRTMIN+3 36) SIGRTMIN+4 37) SIGRTMIN+5
38) SIGRTMIN+6 39) SIGRTMIN+7 40) SIGRTMIN+8 41) SIGRTMIN+9
42) SIGRTMIN+10 43) SIGRTMIN+11 44) SIGRTMIN+12 45) SIGRTMIN+13
46) SIGRTMIN+14 47) SIGRTMIN+15 48) SIGRTMAX-15 49) SIGRTMAX-14
50) SIGRTMAX-13 51) SIGRTMAX-12 52) SIGRTMAX-11 53) SIGRTMAX-10
54) SIGRTMAX-9 55) SIGRTMAX-8 56) SIGRTMAX-7 57) SIGRTMAX-6
58) SIGRTMAX-5 59) SIGRTMAX-4 60) SIGRTMAX-3 61) SIGRTMAX-2
Pick one of the unused ones from the above (probably from the 33-61 range).
> > A global signal handler then raises an exception in the
> > target thread which gets handled the usual way.
>
> What if the signal is blocked?
Then you lose. Normally the interpreter itself would set up the handler.
> > That does leave the target thread able to catch the exception
> > and keep looping. Even still, it improves on the present situation.
>
> How do I do that on Windows? On HP-UX? On Irix? With Gnu threads?
I don't know. I expect it's OS specific. Maybe it can't be done on
every OS. That's no reason not to do it on the OS's that can support it.
Skip Montanaro
Paul> [phr@localhost]$ kill -l
...
Paul> Pick one of the unused ones from the above (probably from the
Paul> 33-61 range).
My MacOSX system only goes up as high as 31 (SIGUSR2). Is calling kill(47)
going to be defined?
Skip
Paul Rubin
> Paul> Pick one of the unused ones from the above (probably from the
> Paul> 33-61 range).
>
> My MacOSX system only goes up as high as 31 (SIGUSR2). Is calling kill(47)
> going to be defined?
On the Mac, you'd use SIGUSR2. I don't see what's so hard to
understand about this. Threads are an OS-specific feature to begin
with. You have to expect the internal interface to them to also
not necessarily be the same for every OS.
Irmen de Jong
<offtopic>
This discussion makes me think back with pleasure at the way threads
were implemented in AmigaDOS... Every process was essentially a thread.
(AmigaDOS didn't have memory protection/processes). The threads could
listen on a flexible set of self-defined "message ports", and various
signals.
Thread signaling took place by sending a message to a message port
(zero-copy by the way), or signaling the thread (for instance, CTRL-C
was a special signal). Thread woke up because AmigaDOS signalled
it "hey wake up, these-and-these message ports have a message for you".
Asynchronous I/O was done using the exact same message port mechanism.
I don't remember if there was a way to kill a thread. There probably
was. With severe side-effects ofcourse (non-freed resources).
If there wasn't, you could ofcourse hack the Exec task list and
manually remove the process node ;-)
Back when I did AmigaPython, I never tried to build AmigaDOS threading
support into the interpreter. Could it have been hard to do so?
</offtopic>
Forget my rambling ;-)
Irmen.
Uwe Schmitt
> But what if I want to end a thread which hangs because
> it uses 'exec' on some code which loops forever ?
so, what about a new PEP : extend the exec method by limiting
the execution time and memory usage ...
greetings, Uwe
Skip Montanaro
Paul> On the Mac, you'd use SIGUSR2. I don't see what's so hard to
Paul> understand about this. Threads are an OS-specific feature to
Paul> begin with. You have to expect the internal interface to them to
Paul> also not necessarily be the same for every OS.
I don't think most people misunderstand what you're saying, it's just that
they have a different take on things than you. Python tries very hard to
make threads less OS-specific. That necessarily means you miss out on some
capavility. Maybe that's not the way it should be. If you'd like things
done differently, I suspect you'll have to convince Guido.
Skip
Rich Harkins
>
> Paul> [phr@localhost]$ kill -l
>
> ...
>
> Paul> 33-61 range).
>
> My MacOSX system only goes up as high as 31 (SIGUSR2). Is calling kill(47)
> going to be defined?
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
Sadly, the only way to actually accomplish this in a cross-platform way
(AFAIK) is to route ALL blocking calls through select/poll (or
equivilent) and maintain a tickler UDP socket that is merged with any
pending reads so that a kill can force thread wakeup by sending through
the tickler. It's messy, doesn't stop the infinite loop case, difficult
to do right, and can kill performance -- but it's not impossible.
BTW, is this what Java did for a while back in the 1.0 days? I seem to
recall it claiming to be able to kill threads and I always wondered
how. I notice that Java no longer implies that killing a thread is
possible or safe anymore.
Rich
Jp Calderone
Yes, that's why Python provides them. But this misconception (that they
work) is also why it will soon -not- provide them :P And neither RExec nor
Bastion ever protected you against "while 1: pass" or "100L ** 100L ** 100L
** 100L ** 100L". It is currently not possible to write a secure sandbox in
Python. I'm not sure if you can call that a bug; it's certainly
unfortunate.
Jp
--
A sad spectacle. If they be inhabited, what a scope for misery
and folly. If they be not inhabited, what a waste of space.
-- Thomas Carlyle, looking at the stars
--
up 39 days, 21:49, 1 user, load average: 0.81, 0.64, 0.57
Peter Hansen
That's a judgment call, and one which was made differently by those
responsible for Python's design.
It's not a question of right or wrong: Python was intended to be very
cross-platform, as opposed to many other languages which try to provide
support for every nuance of every different platform which results in a
situation in which programs are absolutely *not* cross-platform because
even relatively simple programs, e.g. those using threads, cannot run
the same on more than one supported platform.
There are actually very few platforms in which it is possibly safely
to terminate threads from the outside, so it's no great loss to most...
-Peter
Jack Diederich
> On Fri, Jan 24, 2003 at 07:52:08AM -0800, Paul Rubin wrote:
> work) is also why it will soon -not- provide them :P And neither RExec nor
> Bastion ever protected you against "while 1: pass" or "100L ** 100L ** 100L
> ** 100L ** 100L". It is currently not possible to write a secure sandbox in
> Python. I'm not sure if you can call that a bug; it's certainly
> unfortunate.
>
As has been said elsewhere, the correct [read: only way to do it realiably
without rewriting the language] way to do this is by using OS features.
Disk quotas, CPU quotas, chroot filesystems, apache configurations.
If you've ever worked on an LPC MUD or thought about implementing one
you have to deal with giving 'untrusted' users code-level access to the system.
You want to give them access to the codebase and at the same time not give
them a way to hack or DOS the system. If anyone feels like implmenting
a setup like this, I'd be a giddy beta tester. It would be a
combination of setup scripts (add_sanbox_user.p[ly]) that would be language
agnostic plus some language specific RPC libraries.
LPC didn't do it correctly, but it was close enough because coders
were 'trusted,' they had something to lose by exploiting the system so the
restrictions were mainly to prevent them from doing major damage. Minor
but reversible damage was still possible.
-jackdied
Tim Peters
> ...
> It is currently not possible to write a secure sandbox in
> Python. I'm not sure if you can call that a bug; it's certainly
> unfortunate.
Just curious: does there exist a programming language in which it is
possible do this? If so, who guarantees it, and via what kind of validation
process?
Bjorn Pettersen
For the marketing version:
http://java.sun.com/marketing/collateral/security.html
insert-appropriate-smiley'ly y'rs
-- bjorn
Paul Rubin
> It's not a question of right or wrong: Python was intended to be very
> cross-platform, as opposed to many other languages which try to provide
> support for every nuance of every different platform which results in a
> situation in which programs are absolutely *not* cross-platform because
> even relatively simple programs, e.g. those using threads, cannot run
> the same on more than one supported platform.
Deciding whether to use features that arespecific to a particular OS
should be the prerogative of the application designer, not the
programming language. The reason we have OS wars all the time and
everyone has opinions about what OS is best, is that OS's are not all
the same. And if your favorite OS has some advantage over other OS's,
but Python won't let you use your OS's advantage to make your
application better, that kind of defeats the idea of Python as a
platform for writing the best applications that you can.
All kinds of stuff like process control, file i/o operations, etc.
are OS dependent and Python supports OS specific calls to deal with
those things. Why on earth should threads have to get a bigger dose
of OS-independence religion than ioctl gets? This just baffles me.
Paul Rubin
That's the claim for javascript as used in web browsers. People keep
finding holes, but they do tend to get patched pretty fast.
Jp Calderone
> [Jp Calderone]
> > ...
> > It is currently not possible to write a secure sandbox in
> > Python. I'm not sure if you can call that a bug; it's certainly
> > unfortunate.
>
> Just curious: does there exist a programming language in which it is
> possible do this? If so, who guarantees it, and via what kind of validation
> process?
>
Implementations tend to fall short, even if the language specifies that
such security is possible. Java and Javascript claim it, and fall short.
You -could- make a Python implementation that claimed it, there's nothing
about the language that inherently prevents it that I know about. Actually
there's been talk recently on python-dev on how to handle the secure parts
of Python, possibly doing a fork of the interpreter, or a separate dev tree
to try to make the supposedly secure features actually secure, etc. What's
my point? I'm not sure :) I guess my answer is this: I don't know of any
language implementation that tries and succeeds at this goal.
Jp
--
"Pascal is Pascal is Pascal is dog meat."
-- M. Devine and P. Larson, Computer Science 340
--
up 39 days, 23:49, 2 users, load average: 0.74, 0.69, 0.60
Jp Calderone
> Peter Hansen <pe...@engcorp.com> writes:
> > It's not a question of right or wrong: Python was intended to be very
> > cross-platform, as opposed to many other languages which try to provide
> > support for every nuance of every different platform which results in a
> > situation in which programs are absolutely *not* cross-platform because
> > even relatively simple programs, e.g. those using threads, cannot run
> > the same on more than one supported platform.
>
> Deciding whether to use features that arespecific to a particular OS
> should be the prerogative of the application designer, not the
> programming language.
Absolutely. And this is the case with Python. If your application needs
a platform specific feature, you can write an extension module to support
it.
Whether a feature is supported *in the core language* and the *official
distribution* -is- up to the language developers, though, especially in the
case of free software.
Paul Rubin
> Whether a feature is supported *in the core language* and the
> *official distribution* -is- up to the language developers, though,
> especially in the case of free software.
Yes, indeed, the language developers are perfectly entitled to release
programs that don't live up to the marketing claims written by those
same developers. However, nobody should then be surprised when users
complain that their expectations have been let down.
Peter Hansen
>
> On Fri, Jan 24, 2003 at 07:52:08AM -0800, Paul Rubin wrote:
> > Wha? Python provides the rexec and bastion modules specifically to
> > have a way to run hostile code in a sandbox. So what kind of advice
> > is "don't use Python"? If sandboxed code can be prevented from
> > accessing sensitive data or doing i/o, but can't be prevented from
> > looping unkillably, that's a bug in Python.
>
> Yes, that's why Python provides them. But this misconception (that they
> work) is also why it will soon -not- provide them :P And neither RExec nor
> Bastion ever protected you against "while 1: pass" or "100L ** 100L ** 100L
> ** 100L ** 100L". It is currently not possible to write a secure sandbox in
> Python. I'm not sure if you can call that a bug; it's certainly
> unfortunate.
I always hate to describe something I know next to nothing about, but
I vaguely recall reading about something Zope does, I think, with munging
the generated *bytecode* to ensure that infinite loops and certain other
resource-stealing attacks (like range(1000000)) cannot work effectively.
That's not really using pure *Python*, although perhaps it can be effective
and maybe even reliable as a solution to this tricky problem.
-Peter
Peter Hansen
>
> Uwe Schmitt <uwe.s...@procoders.net> wrote:
>
> > But what if I want to end a thread which hangs because
> > it uses 'exec' on some code which loops forever ?
>
> so, what about a new PEP : extend the exec method by limiting
> the execution time and memory usage ...
I'd support a PEP that proposed being able to set a flag in
a thread or in the interpreter so that a SystemExit exception
would be thrown at the very next bytecode instruction that gets
executed.
It's not foolproof, for those attempting to defeat it, and it
still can't handle threads that are blocked in OS calls, but
it works for many of the cases that people think they want it
for, and it would suit me in all cases that I care about.
-Peter
Peter Hansen
I agree. Fortunately, as far as I can tell no one was surprised
in this case when a user didn't have his expectations met. I
did see lots of explanations as to *why* he can't do what he wants,
but very little suggesting surprise.
Anyway, the argument is kind of moot, isn't it? Especially considering
how often it has been raised, and how consistent the answer is to date.
-Peter
Paul Rubin
There's an entry in Python's bug tracking system on Sourceforge about
supporting asynchronous exceptions in multi-threaded programs, showing
the developers consider it something worth looking for ways to
implement, rather than something that can't be done or something that
shouldn't be done for portability reasons.
Since the new "minimal Python" is supposed to be stackless (all
activation records are in the heap), maybe that implementation can
more gracefully deal with the problem of killing a thread failing to
unwind the C stack.
Peter Hansen
>
> There's an entry in Python's bug tracking system on Sourceforge about
> supporting asynchronous exceptions in multi-threaded programs, showing
> the developers consider it something worth looking for ways to
> implement, rather than something that can't be done or something that
> shouldn't be done for portability reasons.
>
> Since the new "minimal Python" is supposed to be stackless (all
> activation records are in the heap), maybe that implementation can
> more gracefully deal with the problem of killing a thread failing to
> unwind the C stack.
That would be a nice thought. I doubt it would cleanly be able
to handle C extensions reliably though, but maybe ignoring some of
the limit cases would be acceptable for the value it provides in
the more common situations.
-Peter
Terry Reedy
"Jack Diederich" <ja...@performancedrivers.com> wrote in message
news:mailman.104343214...@python.org...
> On Fri, Jan 24, 2003 at 12:44:24PM -0500, Jp Calderone wrote:
> > On Fri, Jan 24, 2003 at 07:52:08AM -0800, Paul Rubin wrote:
> > > Peter Hansen <pe...@engcorp.com> writes:
> > > > If you're worried about some kind of denial of service type
> > > > thing with code written by someone else: you're out of luck.
> > > > Don't use Python. (Or, better yet, get over the worry and
just
> > > > accept this possibility.)
> > >
> > > Wha? Python provides the rexec and bastion modules specifically
to
> > > have a way to run hostile code in a sandbox. So what kind of
advice
> > > is "don't use Python"? If sandboxed code can be prevented from
> > > accessing sensitive data or doing i/o, but can't be prevented
from
> > > looping unkillably, that's a bug in Python.
> >
> > Yes, that's why Python provides them. But this misconception
(that they
> > work) is also why it will soon -not- provide them :P And neither
RExec nor
> > Bastion ever protected you against "while 1: pass"
Hmmm. Disable 'while': force use of for loops, and limit list
lengths, and recursion depth and ....
>> or "100L ** 100L ** 100L ** 100L ** 100L".
Hmmm. Remove longs. But then language is not quite Python.
> If you've ever worked on an LPC MUD or thought about implementing
one
> you have to deal with giving 'untrusted' users code-level access to
the system.
I have sometimes thought I would like to play (or write) a game
user-scriptable in Python. But I keep seeing problems like the above.
Removing parts of the language, to make it *not* Turing-complete, as
well as some builtin functions, is only thing I can think of.
Terry
Uwe Schmitt
AFAIK ruby does it.
Greetings, Uwe.
Tim Peters
> ...
> It is currently not possible to write a secure sandbox in
> Python. I'm not sure if you can call that a bug; it's certainly
> unfortunate.
[Tim]
> Just curious: does there exist a programming language in which it is
> possible [to] do this? If so, who guarantees it, and via what kind
> of validation process?
[Bjorn Pettersen]
> For the marketing version:
> http://java.sun.com/marketing/collateral/security.html
>
> insert-appropriate-smiley'ly y'rs
The original branch point for this thread was talking about
denial-of-service attacks, such as sucking up too much CPU. To Sun's
credit, near the bottom of the linked page they plainly say that Java's
protection against DOS is "weak"; heck, they even list 5 specific ways Java
applets can render host machines unusable, and it doesn't take much
imagination to dream up many more.
I don't expect JavaScript/ECMAScript is any better in this respect. For
that matter, I'm having a hard time imagining a *usable* programming
language that could be.
Paul Rubin
> > > Bastion ever protected you against "while 1: pass"
>
> Hmmm. Disable 'while': force use of for loops, and limit list
> lengths, and recursion depth and ....
Recursion depth is already limited. Normally you handle infinite
loops by having the interpreter count the bytecodes it executes,
and stop after some limit.
"Martin v. Löwis"
> On the Mac, you'd use SIGUSR2. I don't see what's so hard to
> understand about this. Threads are an OS-specific feature to begin
> with. You have to expect the internal interface to them to also
> not necessarily be the same for every OS.
Not in Python; Python gives a uniform view to threading on all platforms
where threading is supported in the first place. Portability would go to
hell if every tiny feature could or could not be available on a certain
system.
However, this is not the issue: it probably would be acceptable if the
feature was not available on a certain system. It would be unacceptable
if the feature was available but would cause interpreter crashes.
So the really hard issue is not understanding it, but implementing it.
Feel free to contribute a patch.
Regards,
Martin
"Martin v. Löwis"
> so, what about a new PEP : extend the exec method by limiting
> the execution time and memory usage ...
That PEP would have to come with a suggested implementation strategy.
Regards,
Martin
Laura Creighton
Question -- have we already determined that Uwe Schmitt cannot
use a fork like he seems to want to? What system does this have to
run on?
Laura Creighton
Terry Reedy
"Paul Rubin" <phr-n...@NOSPAMnightsong.com> wrote in message
news:7xn0lq3...@ruckus.brouhaha.com...
> Recursion depth is already limited.
I was thinking of something more limited, like
sys.setrecursionlimit(10), instead of the current default of 1000 (on
my box).
TJR
Jp Calderone
import time
start = time.time()
try:
(lambda x: x(x))(lambda x: x(x))
except RuntimeError:
print 'Took %0.2f seconds' % (time.time() - start)
import sys
sys.setrecursionlimit(10)
start = time.time()
try:
(lambda x: x(x))(lambda x: x(x))
except RuntimeError:
print 'Took %0.2f seconds' % (time.time() - start)
On my machine, at least, the interval is so short that it is more or less
random which of the two blocks takes longer to execute, with times varying
from a hundredth of a second to four times that. On hardware that hasn't
been obsolete since the last millenium, it's probably even less noticable.
And who knows, your user might want to execute code for processing a
reasonably sized tree. Are you really going to be so mean, and force them
to write an iterative version of the algorithm? :)
Jp
--
"There is no reason for any individual to have a computer in their
home."
-- Ken Olson, President of DEC, World Future Society
Convention, 1977
--
up 40 days, 7:49, 7 users, load average: 0.55, 0.53, 0.50
Paul Rubin
> > so, what about a new PEP : extend the exec method by limiting
> > the execution time and memory usage ...
>
> That PEP would have to come with a suggested implementation strategy.
Neither seems terribly difficult; just have the interpreter keep track
of how much time and memory have been consumed. CPU time may be hard
to measure on a per-thread basis, but certainly it can count byte
codes executed, which should be enough for the purpose.
Christian Tismer
> Peter Hansen <pe...@engcorp.com> writes:
[about the thread kiling prob]
> Since the new "minimal Python" is supposed to be stackless (all
> activation records are in the heap), maybe that implementation can
> more gracefully deal with the problem of killing a thread failing to
> unwind the C stack.
Well, whether Stackless or not.
Stackless 2.0 itself has the problem to be by
principle unable to kill a tasklet (which is my
tiny kind of threads), since it cannot have full
track of all stack entries, unless we do a ful stack
analysis.
Stackless 3.0 *will* be able to do this, for those
tasklets which are running without stack copies,
using the "stack-avoidance" scheme of the 1.0 version.
But this is also only a partial solution.
With Minimal Python, we have almost full control over
the whole system. If we provide the code generator, then
we know where the objects are, and we can fotunately
kill threads safely, since we know the stack/object
layout very well.
This still is not a complete solution, since there will
be non-cooperative extension modules which have to be
supported.
hoping-to-suck-them-all-up-and-to-rewrite-the-OS-
-in-Python - ly y'rs -- chris
Martin v. Löwis
How about memory usage?
Regards,
Martin
Paul Rubin
> > Neither seems terribly difficult; just have the interpreter keep track
> > of how much time and memory have been consumed. CPU time may be hard
> > to measure on a per-thread basis, but certainly it can count byte
> > codes executed, which should be enough for the purpose.
>
> How about memory usage?
Is there an obstacle to instrumenting the memory allocator to remember
how much memory has been allocated by sandboxed code?
Ype Kingma
And how would calls to eg. C (CPython) or Java (Jython) code be handled?
--
email at xs4all.nl
Tim Peters
> ...
> CPU time may be hard to measure on a per-thread basis, but certainly it
> can count byte codes executed, which should be enough for the purpose.
Count of bytecodes executed is monotonic with CPU cycles consumed, but
there's no definable relationship beyond "monotonic with" -- there's no
upper bound on how much work a single bytecode can require before the PVM is
entered again. For example, a single bytecode can kick off a (very) long
int division requiring CPU minutes, or a regexp search requiring CPU
millennia. Here's a cute one:
a = []
for i in range(4): a.append(a)
b = []
for i in range(4): b.append(b)
equal = a == b
The opcode for "==" there won't return until about a century has passed:
counting bytecodes is simply useless as a protection against DOS attacks of
a CPU-hogging nature.
resource-limitations-are-an-os's-job-ly y'rs - tim
Martin v. Löwis
> Is there an obstacle to instrumenting the memory allocator to remember
> how much memory has been allocated by sandboxed code?
Yes. It is difficult to find out whom to attribute any allocation
to. It is likely also very expensive even if not used.
Regards,
Martin
Tim Peters
> Is there an obstacle to instrumenting the memory allocator to remember
> how much memory has been allocated by sandboxed code?
Well, "the memory allocator" doesn't exist -- there's no choke point for
allocations, outside of the platform C malloc. If the latter has
instrumentation, then it can be asked. The Python object allocator knows
how many "arenas" it has in hand, so that part is easy, but not everything
goes thru the Python object allocator. Even so, the Python object allocator
has no idea how much memory is actually consumed by the platform C malloc
when it asks the latter for "an arena". Bottom line: all memory ultimately
comes from C malloc, and only C malloc knows what C malloc really does.
Irmen de Jong
> Or, to use Amiga terms, the bottom of the hierarchy was an Exec task;
> AmigaDOS processes were tasks with extra data in the task control block
> -- primarily entries for stdin, stdout, and some other AmigaDOS level
> data (Exec was the task switch/memory allocation/IPC [messages] kernel
> library; I/O and other stuff were libraries built using Exec
> capabilities. AmigaDOS was the "user level" view, the shell commands
> and file concepts.)
You're right ofcourse. Though I always refer to the Amiga's OS with
"AmigaDOS". Others use "Kickstart", which was actually the name
for the 512Kbyte ROM that contained the core OS. ;-)
> In a way, the Exec message port and signal bits were a close match to
> DEC VMS mailboxs (with the above "zero-copy" constraint -- what was
> passed was a list node with the address of the message data) and
> process local event flags (local, as you could not share a signal bit
> between tasks -- though there likely was some scheme by which one task
> could locate/modify the bits of another).
I don't know much about VMS, but I do know that a lot of nice stuff
was borrowed from VMS (and Unix) and used in the creation of AmigaDOS.
( 'Assigns' , anyone?)
> Of course, even signal bits were only effective if the program
> explicitly checked them -- a common problem for neophyte coders is that
> <ctrl-c> had no effect on a process bound loop as the signal check was
> done as part of the C I/O library.
>
> while (1)
> x = y;
>
> could not be broken from the command line as it never tests the signal
> bits.
You hit the nail on the head.
Thanks for pointing this out again.
Because of this requirement, the SIGNAL solution for aborting a thread,
proposed elsewhere in this discussion, would never work on the Amiga
if the thread doesn't check for the signal explicitly.
Though I now remember the following: it was possible to set a special
function for a designated "interrupt" signal. (there is a field for
this in the Exec task structure). I even used this in
AmigaPython to catch and process the CTRL-C (or "break") signal,
if I'm not mistaken. So it might be possible after all!
Well, it doesn't matter anymore. AmigaPython died at version 2.0,
without thread support. ;-)
Irmen de Jong
Jacob Hallen
>> cross-platform, as opposed to many other languages which try to provide
>> support for every nuance of every different platform which results in a
>> situation in which programs are absolutely *not* cross-platform because
>> even relatively simple programs, e.g. those using threads, cannot run
>> the same on more than one supported platform.
>
>Deciding whether to use features that arespecific to a particular OS
>should be the prerogative of the application designer, not the
>everyone has opinions about what OS is best, is that OS's are not all
>the same. And if your favorite OS has some advantage over other OS's,
>but Python won't let you use your OS's advantage to make your
>application better, that kind of defeats the idea of Python as a
>platform for writing the best applications that you can.
Local optimisation without regard to global optimisation is usually
a bad idea. You argue that reaping the benefits of being able to
use OS specific features is worth the price of not being able to run
your software on other platforms.
I think you should reconsider this argument, if you have ever used a
single part of the Python library, or somebody elses Python software.
It worked on your platform _because_ the Python philosophy is to be
platform independent.
As an aside, threading is not part of the Python language. You are free
to build your own thread library that is very platform specific. However,
it will be your design decision and your package to maintain.
>
>All kinds of stuff like process control, file i/o operations, etc.
>are OS dependent and Python supports OS specific calls to deal with
>those things. Why on earth should threads have to get a bigger dose
>of OS-independence religion than ioctl gets? This just baffles me.
You have things backwards. Ioctl is a deplorable example of when OS
independence was too hard to achieve. If it can reasonably be redone in
an OS independent way it should. The functionality itself is way too
important to rip out.
Jacob Hallén
--
Aahz
Uwe Schmitt <uwe.s...@procoders.net> wrote:
>Tim Peters <tim...@comcast.net> wrote:
>> [Jp Calderone]
>>>
>>> I'm not sure if you can call that a bug; it's certainly unfortunate.
>>
>> Just curious: does there exist a programming language in which it is
>> possible do this? If so, who guarantees it, and via what kind of
>> validation process?
>
>AFAIK ruby does it.
What exactly does Ruby do? Allow forced shutdown of threads? No way;
Ruby uses cooperative threading that's even more restrictive than
Python's. Have a secure sandbox? Then Tim's question applies with
force: who guarantees the secure sandbox, and what validation proves
this guarantee?
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
"Argue for your limitations, and sure enough they're yours." --Richard Bach
Skip Montanaro
>> All kinds of stuff like process control, file i/o operations, etc.
>> are OS dependent and Python supports OS specific calls to deal with
>> those things. Why on earth should threads have to get a bigger dose
>> of OS-independence religion than ioctl gets? This just baffles me.
Jacob> You have things backwards. Ioctl is a deplorable example of when
Jacob> OS independence was too hard to achieve. If it can reasonably be
Jacob> redone in an OS independent way it should. The functionality
Jacob> itself is way too important to rip out.
It seems to me that the general way platform-dependent stuff is handled is
on a module-by-module basis. Accordingly, an entire module is either
platform-dependent or platform-independent. Thus fcntl is Unix-only, as is
win32all. Once a decision is made that a module will be
platform-independent, it's rare that platform-dependent extensions are made
to it. That may mean some sacrifice in functionality (e.g., no thread
killing).
The os module is a bit of a weird beast because it provides a hint of
platform independence on top of the posix module and its ilk, but the
programmer must still be careful which module-level objects are used if
concerned about platform independence. I view this is a compromise. It
would break too much code to go back now and try to make the os module truly
platform-independent.
Skip
Skip Montanaro
Should of course have read something like
Thus fcntl is Unix-only, and win32all is Windows-specific.
S
anton wilson
> Uwe Schmitt wrote:
> > Uwe Schmitt <uwe.s...@procoders.net> wrote:
> > > But what if I want to end a thread which hangs because
> > > it uses 'exec' on some code which loops forever ?
> >
> > so, what about a new PEP : extend the exec method by limiting
> > the execution time and memory usage ...
>
> a thread or in the interpreter so that a SystemExit exception
> would be thrown at the very next bytecode instruction that gets
> executed.
>
> It's not foolproof, for those attempting to defeat it, and it
> still can't handle threads that are blocked in OS calls, but
> it works for many of the cases that people think they want it
> for, and it would suit me in all cases that I care about.
>
> -Peter
I'm somewhat interested in attempting to do this myself. You have any ideas
on how to communicate the kill message between threads? One way I can think
of right now is with a global list of threads to be killed that get's checked
in the interpreter loop. But then that adds one more global variable that is
protected by the global interpreter lock and more overhead to check the list
. . . See any other problems with this idea?
The way you suggested would, in a non-cross-platform manner, set up a
"A global signal handler then raises an exception in the
target thread which gets handled the usual way." I'm guessing that for this
to work, the thread that get's the signal would catch it in some imported c
module and then rasise the SystemExit exception from within the module.
Correct me if I'm wrong.
Anton
Peter Hansen
>
> On Friday 24 January 2003 04:36 pm, Peter Hansen wrote:
> > I'd support a PEP that proposed being able to set a flag in
> > a thread or in the interpreter so that a SystemExit exception
> > would be thrown at the very next bytecode instruction that gets
> > executed.
>
> on how to communicate the kill message between threads? One way I can think
> of right now is with a global list of threads to be killed that get's checked
> in the interpreter loop.
> The way you suggested would, in a non-cross-platform manner, set up a
> "A global signal handler then raises an exception in the
> target thread which gets handled the usual way." I'm guessing that for this
> to work, the thread that get's the signal would catch it in some imported c
> module and then rasise the SystemExit exception from within the module.
> Correct me if I'm wrong.
I believe you're wrong, although my idea was not very concrete, so
it's quite open to interpretation, so maybe you're right. ;-)
It's been a while since I looked at the interpreter, but I believe
it has direct knowledge of Python threads. I believe it should
therefore be possible to arrange for a flag that is checked,
maybe only when the 'checkinterval' is reached (number of bytecode
instructions before a context switch is considered), and if
it's true, the exception is raised as though the next bytecode
instruction caused it.
The thread object itself could have the flag, so you don't need to
have a global "hit list" of threads with the added overhead of
checking the GIL. Add a method to the thread class and you probably
have all you need to let another thread (or the thread itself,
if you like suicide) trigger the behaviour.
It would be 100% cross-platform, as it would be either pure Python,
or require only the (presumably already cross-platform) core thread
structure to support the flag, plus generic changes to the interpreter.
-Peter
anton wilson
>
> The thread object itself could have the flag, so you don't need to
> have a global "hit list" of threads with the added overhead of
> checking the GIL.
Everything seems fine except that if I'm thread A and I want to tell thread
B to jump off a cliff, I need to communicate that to thread B somehow, and as
far as I can tell, there's no way for thread A to do this unless I change
thread.start_new to return some object with a pointer to the thread's t_state
context. To make matters even worse, the t_state struct that is used in the
interpretter loop is created by the new thread itself, and not the function
that actually makes the OS call to create the thread. So currently, there is
no way for me to get my hands on the t_state without some semi-non-trivial
change. I guess if I wanted to, I could pass a pointer to some object that
could be changed by the newly created thread, use locks to protect the
information inside that object, and hope that by the time I'm ready to kill
thread B, thread B has already written a pointer to it's t_state structure.
Or I could use a global dictionary, to get my hands on the t_state....
Anton
Peter Hansen
Ah... I've always followed the advice to stick to the threading.Thread
class instead of anything in the thread.py module, and more than that
I've always done threads by _subclassing_ that class, rather than passing
in my target function as a reference... as such, my relatively narrow
experience with threads in Python leads me to feel this is simpler than
it probably is. :-(
-Peter
Laura Creighton
> >
> > The thread object itself could have the flag, so you don't need to
> > have a global "hit list" of threads with the added overhead of
> > checking the GIL.
>
> B to jump off a cliff, I need to communicate that to thread B somehow, and as
>
> far as I can tell, there's no way for thread A to do this unless I change
> thread.start_new to return some object with a pointer to the thread's t_state
>
> context. To make matters even worse, the t_state struct that is used in the
> interpretter loop is created by the new thread itself, and not the function
> that actually makes the OS call to create the thread. So currently, there is
> no way for me to get my hands on the t_state without some semi-non-trivial
> change. I guess if I wanted to, I could pass a pointer to some object that
> could be changed by the newly created thread, use locks to protect the
> information inside that object, and hope that by the time I'm ready to kill
> thread B, thread B has already written a pointer to it's t_state structure.
> Or I could use a global dictionary, to get my hands on the t_state....
>
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
Use a queue to communicate between threads. Pass a message
that says 'go die'. You may need 2 queues, if you need to
get messages back as well.
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/82965
anton wilson
>
> Use a queue to communicate between threads. Pass a message
> that says 'go die'. You may need 2 queues, if you need to
> get messages back as well.
>
The problem is not making a thread die, but making a thread I didn't write
die. For that, I'm going to have to do interpreter changes.
Anton