realtime design
Will Stuyvesant
'''
A lost ounce of gold may be found, a lost moment of time never.
'''
James J. Besemer
Will Stuyvesant wrote:
>We would like to prototype a realtime system. Is anybody doing this using Python?
>
Depends on what you mean by "real time". ;o)
I am using Python for a home automation system. Very low-level digital,
analog and X10 I/O is handled by hardware or custom hardware/firmware
modules but the overall command and control logic is handled in Python.
In particular, a Real Time event actuator commands all of the
autonomous events, such as lighting being turned on or off on a preset
schedule. Non-x10 signals can be processed in around 10 msec.; X10 by
it's nature takes on the order of 1 second per command code sent or
received.
The base control system runs on a dedicated Linux computer but all the
basic controls operate via TCP/IP sockets and thus can be monitored or
operated upon by any computer on the network. The idea is that each
"client" can connect to the system and monitor and change just those few
signals it cares about. For most services, I start out with a unix
"command line" interface and then follow later with more elaborate GUI
front ends.
I feel Python is eminently suitable for many real time applications. It
only begins to break down in those circumstances where the real time
constraints are very tight. Even then, C modules can be written to
handle some of the time-critical functions. Only on the high-end of
real time performance would I say that Python is probably not the right
choice.
Regards
--jb
--
James J. Besemer 503-280-0838 voice
2727 NE Skidmore St. 503-280-0375 fax
Portland, Oregon 97211-6557 mailto:j...@cascade-sys.com
http://cascade-sys.com
Peter Hansen
> We would like to prototype a realtime system. Is anybody doing this using Python?
We are using Python for a soft realtime system. Whether it is suitable
for your purpose or not depends entirely on what your constraints are.
Can you provide more background? Hard or soft realtime? How much
latency can you afford? Safety critical? User interaction? Etc...
-Peter
Will Stuyvesant
We would like to do prototyping of realtime systems. Is
anybody doing something like this using Python?
</OP>
That posting has been reformulated to a programming contest:
James Besemer asked what we mean with realtime. Realtime is
about responsiveness, for our purposes. What we would like to
have is a function (or maybe a class, but a function seems
enough for now) that would make existing Python software
suitable for a realtime application. The function should call
a function in the existing software. The function should
return within a specified time limit. That would mean a
function call could only take a maximum amount of time, after
that it is ignored or killed or whatever and a default value
is used. So you can reuse existing software without modifying
anything in its source code.
I think this can be done. I need a function, lets call it
rcall(...), that accepts a *maximumtime* parameter, a value
representing milliseconds, and that calls a function *f* in
the existing software. If *f* returns before maximumtime
passes everything is just like when calling the *f* function
without rcall. Else rcall returns 'timeout' at about time ==
*maximumtime*. I can think of many other useful functions for
realtime software, but something like rcall(...) is the first
I would like to have.
For implementing this the signal module did look interesting.
But the signal module is not complete for Windows: e.g.
signal.alarm() and signal.pause() are UNIX only, so the signal
module seems not useful. And it was mentioned that
time.sleep() does not work with signals on a Linux box.
Here is my first setup. What I am asking is this: can anybody
write the realtime.rcall function body so it fits its
description?
<file realtime.py>
import time
def current_time():
return time.time() * 1000 # ms since Januari 1 1970
def rcall(ms, func, *args, **kw):
''' Return func(*args, **kw), unless over ms milliseconds
time passes. In that case return 'timeout' in about ms
milliseconds plus the time it takes for returning.
NOT functioning now, just returning func(*args)
'''
starttime = current_time()
# YOUR CODE HERE TO REPLACE THIS LINE :-)
returnvalue = func(*args, **kw) # How to set a timelimit here?
endtime = current_time()
if endtime - starttime > ms:
return 'timeout'
else:
return returnvalue
</file realtime.py>
Testing it:
<file testrealtime.py>
import random, time
from realtime import rcall
from realtime import current_time
class TestMe:
def waitForSeconds(self, seconds):
time.sleep(seconds)
pass
def returnImmediately(self):
return 1
def bigCalculation(self):
# to make sure the cpu is busy
x = [ [[]] * 5000 +
[random.random()] for i in range(1000)]
x.sort() # takes about 5 seconds on my computer
return 1
starttime = current_time()
t = rcall(500, TestMe)
if t != 'timeout':
print rcall(500, t.returnImmediately),
print rcall(500, t.waitForSeconds, 2),
print rcall(500, t.bigCalculation),
endtime = current_time()
print endtime - starttime
</file testrealtime.py>
-- DESIRED output:
1 timeout timeout 1050.0
-- or another value around 1050.0, only t.waitForSeconds and
-- t.bigCalculation should take a little over 500 ms, the
-- others should be very fast
-- The output now is:
1 timeout timeout 5938.0
-- of course, since there the time limit in realtimecall.rcall
-- is not implemented.
QOTD
'''
=== ALL CSH USERS PLEASE NOTE ========================
Set the variable $LOSERS to all the people that you think are losers.
This will cause all said losers to have the variable
$PEOPLE-WHO-THINK-I-AM-A-LOSER updated in their .login file. Should you
attempt to execute a job on a machine with poor response time and a
machine on your local net is currently populated by losers, that machine
will be freed up for your job through a cold boot process.
'''
Peter Hansen
> <OP>
> We would like to do prototyping of realtime systems. Is
> anybody doing something like this using Python?
> </OP>
>
> That posting has been reformulated to a programming contest:
>
> James Besemer asked what we mean with realtime. Realtime is
> about responsiveness, for our purposes. What we would like to
> have is a function (or maybe a class, but a function seems
> enough for now) that would make existing Python software
> suitable for a realtime application. The function should call
> a function in the existing software. The function should
> return within a specified time limit. That would mean a
> function call could only take a maximum amount of time, after
> that it is ignored or killed or whatever and a default value
> is used.
That sounds like a pretty bizarre thing to do, considering
how most systems described as "realtime" are actually
designed (that is to say, they are unlike the above approach).
Could you tell us what you are really trying to accomplish?
-Peter
Peter Hansen
> I think this can be done. I need a function, lets call it
> rcall(...), that accepts a *maximumtime* parameter, a value
> representing milliseconds, and that calls a function *f* in
> the existing software. If *f* returns before maximumtime
> passes everything is just like when calling the *f* function
> without rcall. Else rcall returns 'timeout' at about time ==
> *maximumtime*. I can think of many other useful functions for
> realtime software, but something like rcall(...) is the first
> I would like to have.
This sounds like a pretty simple thing to do, using a thread
to call the function, but that doesn't mean the resulting
system will in any way be considered "realtime" unless you
have a fairly simplistic definition. You can't get around
the underlying fact that Python itself has aspects that are
not deterministic, nor that unless you are running on top of
a realtime operating system, no application can be considered
realtime, by the usual definitions. More info required...
-Peter
Kragen Sitaker
> I feel Python is eminently suitable for many real time applications.
> It only begins to break down in those circumstances where the real
> time constraints are very tight. Even then, C modules can be written
> to handle some of the time-critical functions. Only on the high-end
> of real time performance would I say that Python is probably not the
> right choice.
Different people mean different things by "real time", as you pointed
out. For your meaning of "real time", I agree with you. But for
another meaning of "real time" --- "guaranteed worst-case response
time" --- I don't think Python is suitable, because a programmer
cannot correctly calculate a worst-case response time for any but the
smallest Python programs.
Chris Liechti
news:83it06i...@panacea.canonical.org:
and why do you think is python different than other languages?
i think it has clear semantics, you know whats going on. e.g. you know
exactly when memory is freed (CPython) etc.
what you need is an underlying realtime OS, that guaratees that the real
time tasks get enough CPU power at the right time. without that absolutely
no programming language can help out.
chris
--
Chris <clie...@gmx.net>
Brian Quinlan
> and why do you think is python different than other languages?
> i think it has clear semantics, you know whats going on. e.g. you know
> exactly when memory is freed (CPython) etc.
But the variable timing of memory allocation/deallocation might be
unacceptable.
Cheers,
Brian
Richard Jones
What do you mean by variable timing? If you avoid circular references, CPython
(the C version, not the Java version) has deterministic memory handling.
Richard
Brian Quinlan
> > But the variable timing of memory allocation/deallocation might be
> > unacceptable.
> What do you mean by variable timing? If you avoid circular references,
> CPython
> (the C version, not the Java version) has deterministic memory
handling.
CPython uses malloc/free to allocate/deallocate Python objects. The
malloc routine must search a free list for a memory block of sufficient
size. So the timing of trivial Python operations can vary significantly,
depending on the malloc implementation and the history of previous
memory allocations/deallocations.
Cheers,
Brian
Bradley D. Larson
dependent upon the
need of the underlying system.
The underlying premise is that the calculations are done..."in order that results
of the computation can be
used in guiding the physical process" (see gov def below). If you can calculate
appropriate results in 30
minutes and they are used to make changes to the process every 45 minutes... That
is real time.
In the following definition I would underline: "user senses as sufficiently
immediate" but don't want to
get yelled at by those utilizing deficient email readers.
Real time is a level of computer responsiveness that a user senses as sufficiently
immediate or
that enables the computer to keep up with some external process (for example, to
present
visualizations of the weather as it constantly changes). Real-time is an adjective
pertaining to
computers or processes that operate in real time. Real time describes a human
rather than a
machine sense of time.
In the days when mainframe batch computers were predominant, an expression for a
mainframe
that interacted immediately with users working from connected terminals was online
in real time.
Source: http://whatis.techtarget.com/definition/0,289893,sid9_gci214344,00.html
Government Definition:
real time: 1. The actual time during which a physical process occurs. (188) 2.
Pertaining to the performance of a computation during the actual time that the
related physical process occurs, in order that results of the computation can be
used in guiding the physical process.
Source: http://www.its.bldrdoc.gov/fs-1037/dir-030/_4450.htm
Chris Liechti wrote:
> ... what you need is an underlying realtime OS, that guaratees that the real
Chris Liechti
news:mailman.103454743...@python.org:
ok, sure. but i want to make clear that has nothing to do with Python
suitability for real time applications if the underlying OS causes the
troubles. if the underlying memory management is that bad, even the
smallest programm in C suffers from the same problem.
chris
--
Chris <clie...@gmx.net>
Chris Liechti
news:mailman.1034549538...@python.org:
> Real-time does not always mean "micro" or nano second response. The
> granularity is dependent upon the need of the underlying system.
i know. i didn't wanted to imply otherwise. it just that the discussion
seemed to go in the direction of fast responses...
i would say that a normal OS (Windows, Linux, others) is suitable for real
time down to the range to 1s timesteps. longer times beeing easy. there's
of course no guarantee that even that can be met. there much longer delays
(e.g. accessing the floppy under windows or deleting a large file under
linux ext2, spinning up a HD from low power when accessed)
if one needs a _guaranteed_ response in the range of seconds or shorter,
then there is no way around a real time OS (or such a primitive 'thing' - i
don't want to call that an OS ;) - like DOS)
(i think there are real time extensions in both, Linux and Windows, but
they're not accessible to user programs. both OS have features for very
short delays etc. for the use in kernel drivers)
i'm working with a real time system in the 125ms range and it works quite
good. it runs on machines from 200Mhz Pentiums up to much faster ones.
but it's not critical when one timeslot is not met and uses longer.
what i'm saying is that for hard real time you need a real time OS ("hard"
meanung that you have time constraints that must be met in _every_ case)
for soft real time a normal OS without special extensions is suitable too
("soft" means it does not hurt when the constraint is missed once, but held
in most cases)
the fact that it uses Python or an other languages does not matter all that
much. some make it easier, some harder. of course, Java with it's gc in the
background makes predictions how long something takes harder, especialy for
short times, that's a problem in Jython. CPython is much easier to
understand with it's reference counting.
> Government Definition:
>
> real time: 1. The actual time during which a physical process occurs.
> (188) 2. Pertaining to the performance of a computation during the
> actual time that the related physical process occurs, in order that
> results of the computation can be used in guiding the physical
> process.
>
> Source: http://www.its.bldrdoc.gov/fs-1037/dir-030/_4450.htm
heh, if we take the second definition, then Python meets the requirements.
we just have to choose an appropriate problem, though ;-)
chris
> Chris Liechti wrote:
>
>> ... what you need is an underlying realtime OS, that guaratees that
>> the real time tasks get enough CPU power at the right time. without
>> that absolutely no programming language can help out.
--
Chris <clie...@gmx.net>
Richard Jones
> Richard Jones wrote:
> > > But the variable timing of memory allocation/deallocation might be
> > > unacceptable.
> >
> > What do you mean by variable timing? If you avoid circular references,
> > CPython
> > (the C version, not the Java version) has deterministic memory
>
> handling.
>
> CPython uses malloc/free to allocate/deallocate Python objects. The
> malloc routine must search a free list for a memory block of sufficient
> size. So the timing of trivial Python operations can vary significantly,
> depending on the malloc implementation and the history of previous
> memory allocations/deallocations.
As other posters have pointed out, for Python to satisfy realitime
constraints, the underling operating system must also satisfy those
constraints.
Richard
Brian Quinlan
>> CPython uses malloc/free to allocate/deallocate Python objects.
>> The malloc routine must search a free list for a memory block
>> of sufficient size. So the timing of trivial Python operations
>> can vary significantly, depending on the malloc implementation
>> and the history of previous memory allocations/deallocations.
> ok, sure. but i want to make clear that has nothing to do with
> Python suitability for real time applications if the underlying OS
> causes the troubles. if the underlying memory management is that bad,
> even the smallest programm in C suffers from the same problem.
1. I'm not arguing that CPython is not appropriate for realtime
applications.
2. The C standard library is not (usually) part of the OS.
3. The smallest program in C need not allocate memory dynamically at
all.
And it might be possible to structure larger programs such that the
portions requiring realtime performance need not allocate memory
dynamically. Currently I am using working with an embedded realtime
system where the dynamic memory allocation is only performed at
system startup time.
Cheers,
Brian
Will Stuyvesant
>
> This sounds like a pretty simple thing to do, using a thread
> to call the function, but that doesn't mean the resulting
> system will in any way be considered "realtime" unless you
> have a fairly simplistic definition. You can't get around
> the underlying fact that Python itself has aspects that are
> not deterministic, nor that unless you are running on top of
> a realtime operating system, no application can be considered
> realtime, by the usual definitions. More info required...
>
> -Peter
Well if you think it is easy, could you please show me how?
Because I can not, probably not enough programming skills here...
There a quite a couple of postings here about using Python itself
for realtime and other aspects. I think I did not make it clear
that we just want to *model* a realtime system, for designing
purposes. With that model we can try to do things like
correctness proofs. To build such a model we need basic building
blocks, and the ``rcall'' function below is one of those building
blocks, but I do not know how to implement it!
The *real* realtime system would be implemented later, probably on
top of a realtime OS, but that is not important now. The first
goal is to build some kind of simulator for a realtime system. If
it works with seconds instead of milliseconds that would even be
fine too.
Here is the ``programming contest'' again, can you show how to
program the rcall function so it fits its desciption?
I put some comments around the line to ``replace'', I hope I
explain now more clearly what I am looking for?
--------------- cut here ------------------------------------------
import time
def current_time():
return time.time() * 1000 # ms since Januari 1 1970
def rcall(ms, func, *args, **kw):
''' Return func(*args, **kw), unless over ms milliseconds
time passes. In that case return 'timeout' in about ms
milliseconds plus the time it takes for returning.
NOT functioning now, just returning func(*args)
'''
starttime = current_time()
# YOUR CODE HERE to replace this line :-)
returnvalue = func(*args, **kw) # How to set a timelimit here?
# A new thread with func?
# Just starting a new thread does not take that long, that
# will stay below ms milliseconds. The idea is that this line
# of code should not take over ms milliseconds. But I do not
# know how to start a new thread with func(...) and get the
# result from it if it is finished in time: Suppose func
# hangs, runs forever, how to kill that thread?
# Or start another process with func?
endtime = current_time()
if endtime - starttime > ms:
return 'timeout'
else:
return returnvalue
--------------- cut here ------------------------------------------
'''
QOTD
All great discoveries are made by mistake.
-- Young
'''
Armin Steinhoff
> We would like to prototype a realtime system. Is anybody doing this using >Python?
Have a look to http://www.sf.net/projects/pyqnx
(QNX is a real-time OS which is free for non commercial use ..)
There is a port of Python 2.2 and an extension module (SRR for Python)
for message passing with external QNX processes. These processes can
provide any services ... a send time out can be archived by using a so
called kernel time-out.
We provide also CAN and PROFIBUS DP (fieldbus IO) support with a time
resolution of 1-2ms. (The limiting factor is the speed of Python ..)
Cheers
Armin
http://www.steinhoff-automation.com
Will Stuyvesant
found out there is probably none. Argh!
The problem:
I am trying to come up with a method to run a function in such a way
that the whole thing will take at most x milliseconds, with x a
reasonable value like 200. If the function is not finished by that
time, a default value should be used. If the function *does* finish
in time, the return value of the function should be used. The main
point is that the calling program should be able to continue with
the next statement after x milliseconds. And it would be nice if
the called function is interrupted or something if it has not
finished, so it will not eat CPU resources.
But I found this:
From: Tim Peters (tim...@home.com)
Subject: RE: embedded python - cancel "while 1: pass"
Newsgroups: comp.lang.python
Date: 2001-01-08 00:07:17 PST
Java's thread .stop(), .resume(), and .suspend() methods were all
deprecated, because they proved to be "inherently unsafe" in
practice. This isn't real surprising: a thread is cheaper than an
OS process largely because the runtime *doesn't* save enough info
to clean up a thread safely, short of the death of the process
it's running in.
Aha! So what I am trying to do is basically impossible with threads!
I guess some people have been here before. Anyway, thank you Tim
Peters. There should be a collection with Tim Peters' c.l.p.
postings you can search on subject. His postings have been useful for
me before! I found this one using google groups search.
What I am going to look into is how to write a function that will
start the target function in another *process* in Windows, and use a
default value if the process does not finish in time.
Any ideas?
and-now-I-am-thinking-about-how-to-kill-a-Windows-process-ly y'rs
'''
Multithreading will rot your teeth.
-- Michael Swaine
'''
Aahz
Richard Jones <rjo...@ekit-inc.com> wrote:
>
>What do you mean by variable timing? If you avoid circular references,
>CPython (the C version, not the Java version) has deterministic memory
>handling.
That's true, but only partly. Thing is, normally one programs Python
with the assumption that checking memory sizes and such doesn't really
matter. But freeing a large dict, particularly one that contains
complex objects, can have serious consequences for timing. Defensively
programming against that isn't normal Python style, and in the absence
of such defensive programming, I'd say that Python isn't really
deterministic.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
Project Vote Smart: http://www.vote-smart.org/
Aahz
Will Stuyvesant <hw...@hotmail.com> wrote:
>
>I am trying to come up with a method to run a function in such a
>way that the whole thing will take at most x milliseconds, with x a
>reasonable value like 200. If the function is not finished by that
>time, a default value should be used. If the function *does* finish in
>time, the return value of the function should be used. The main point
>is that the calling program should be able to continue with the next
>statement after x milliseconds. And it would be nice if the called
>function is interrupted or something if it has not finished, so it will
>not eat CPU resources.
That last part isn't really possible, not if there are statements that
depend on external connectivity like I/O. But it's certainly possible
to make your main loop work the way you want: have your function put its
result on a Queue.Queue. In your main function, time.sleep() for
however long it's supposed to, then do a non-blocking get() on the
queue. If the get() fails, do whatever's appropriate. This gets
trickier if your main program needs to be responsive during the 200ms,
but it's not impossible.
If your function is running in a loop, have it check the time at the top
or bottom of the loop and exit early if it takes too long. That's not
perfect, but it should keep your program from consuming endless
resources.
>and-now-I-am-thinking-about-how-to-kill-a-Windows-process-ly y'rs
Not cleanly.
Bjorn Pettersen
> In article <cb035744.02101...@posting.google.com>,
> Will Stuyvesant <hw...@hotmail.com> wrote:
> >
> >I am trying to come up with a method to run a function in such a way
> >that the whole thing will take at most x milliseconds, with x a
> >reasonable value like 200. If the function is not finished by that
> >time, a default value should be used. If the function *does* finish
in
> >time, the return value of the function should be used.
>
> statements that depend on external connectivity like I/O.
> But it's certainly possible to make your main loop work the
> way you want: have your function put its result on a
> Queue.Queue. In your main function, time.sleep() for however
> long it's supposed to, then do a non-blocking get() on the
> queue. If the get() fails, do whatever's appropriate. This
> gets trickier if your main program needs to be responsive
> during the 200ms, but it's not impossible.
You should be able to re-work the "Futures" receipe
(http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/84317) into
doing what you want... (basically set a timestamp in __init__ and
conditionally return a default value in __call__).
-- bjorn
Dennis Lee Bieber
07:29 am:
>
> What I am going to look into is how to write a function that will
> start the target function in another *process* in Windows, and use a
> default value if the process does not finish in time.
> Any ideas?
>
Not easy either -- Windows itself doesn't support many ways for a
parent to wait on a sub-process ("Python Standard Library" indicates
that os.system() waits, but my experience says it didn't). os.spawnv()
has a WAIT mode, but then you are stuck just like in a thread...
If you do NOWAIT, you could set up a timer... But again Windows
doesn't have an easy way to kill the process (or I couldn't find it --
UNIX/Linux support os.kill(pid)). And you've no doubt seen just how
long it takes from using the task manager (ctrl-alt-del) to pick a
process to kill, until the "program is not responding..." pop-up.
You'd also have the difficulty of how to get the result returned to
you. Oh, besides having to write the "function" as a stand-alone main
program.
(Oh, I should clarify -- I do NOT know what is available in the win32
extension package regarding process control; my book is on loan <G>)
So... while I can visualize a scheme in which I can properly select
which return is valid, I can not kill off the incomplete thread, and
also have the restriction that a second call is unsafe before all of
the first one has finished (well, I may have a work around for that too)
... a half hour later... (hope the formatting survives -- ignore the
KNode "box")
,----[ /home/wulfraed/tt.py ]
| from time import sleep
| import threading
| import Queue
|
|
| def Worker(*args, **kwargs):
| retq = kwargs["retq"]
| func = kwargs["ToDo"]
| del kwargs["retq"]
| del kwargs["ToDo"]
| retq.put(func(*args, **kwargs))
|
| def rcall(ms, func, *ar, **kw):
| q = Queue.Queue()
| kw["retq"] = q
| kw["ToDo"] = func
| w = threading.Thread(target=Worker, args=ar, kwargs=kw)
| w.start()
| w.join(float(ms) / 1000.0)
| if q.empty():
| return "TimeOut" # leaves a dangling Queue and Worker
Thread
# had a line wrap on the comment, beware!
| else:
| return q.get()
|
|
|
| # dummy func
|
| def dfunc(ms):
| sleep(float(ms) / 1000.0)
| return "dfunc is Defunct"
|
| if __name__ == "__main__":
| print rcall(10, dfunc, 5)
| print rcall(10, dfunc, 50)
| print rcall(50, dfunc, 50)
| print rcall(50, dfunc, 49)
| print rcall(50, dfunc, 51)
|
`----
>
> and-now-I-am-thinking-about-how-to-kill-a-Windows-process-ly y'rs
>
>
> '''
> Multithreading will rot your teeth.
> -- Michael Swaine
> '''
--
--
> ============================================================== <
> wlf...@ix.netcom.com | Wulfraed Dennis Lee Bieber KD6MOG <
> wulf...@dm.net | Bestiaria Support Staff <
> ============================================================== <
> Bestiaria Home Page: http://www.beastie.dm.net/ <
> Home Page: http://www.dm.net/~wulfraed/ <
Tim Daneliuk
> We would like to prototype a realtime system. Is anybody doing this using Python?
>
> '''
> A lost ounce of gold may be found, a lost moment of time never.
> '''
Much has already been posted here but I would add this:
There is an inherent problem with the term "realtime" if it is not
further qualified. All realtime systems must typically cope with external
asynchronous events as their principal paradigm - i.e., Asynchronous
events determine control flow of the code not some master algorithm.
However, all such systems are not equal. Generally, we can think about
two classes of realtime systems:
1) "Hard" realtime - cases where timing must be predictable and guaranteed.
For example, "You must respond to the Flabergast Interrupt in less than
100 us and guarantee it is serviced completely in 1ms." This kind of
thing is common in machine control and military fire control systems.
2) "Soft" realtime - cases where you must respond to external events in
a reasonable amount of time and preserve proper *sequencing*, but the
absolute amount of time is neither guranteed nor particularly important
unless it gets ridiculously long. A good example of this is the
message processing and dispatching common in a GUI manager.
I would argue that Python is possibly suited to 2) and probably poorly suited
to 1). The reason I say this is that things like background garbage
collection, interaction with lower-level system services like I/O and so on
make guaranteed, predictable service times essentially impossible (unless
the guarantees are so weak and the permissible times so long that it does
not matter - but this is not ordinarily the case in hard realtime).
It might be possible to shoehorn Python into a hard realtime environment
for some functions like dispatching if one wrote a sufficient amount of
binder code in a lower level language like 'C' or Assembler, but why
bother? Python is a marvelous language with many merits, but it cannot do
everything - no language can, nor should it be expected to. - Shameless
promotion of my ideas on *that* subject can be found at:
http://www.tundraware.com/Technology/Bullet/
If you shudder at writing in Assembler or 'C', give Forth a good look
for this sort of thing.
My view is that higher abstraction langauges, especially those involving a lot
of dynamic runtime behavior are poor fits for hard realtime and even (often)
very challenging for soft realtime.
HOWEVER, you said you wanted to "prototype" such a system. Depending on how
functional your prototype needs to be, Python may well be a good choice.
It is relatively easy and quite appropriate to *model* realtime systems, hard
and soft, in other languages. I once wrote a operating system VM simulation
in 'C' which had nothing to do with a real OS, but was an exercise designed
to examine paging behavior and get some sense of how different swapping algorithms
handled different page working sets. It was a useful and interesting
project which is sorta-kinda-somewhat realtime-ish.
Just my 2 bits worth ...
--
------------------------------------------------------------------------------
Tim Daneliuk
tun...@tundraware.com
Tim Daneliuk
> In article <mailman.1034545938...@python.org>,
> Richard Jones <rjo...@ekit-inc.com> wrote:
>
>>What do you mean by variable timing? If you avoid circular references,
>>CPython (the C version, not the Java version) has deterministic memory
>>handling.
>
>
> That's true, but only partly. Thing is, normally one programs Python
> with the assumption that checking memory sizes and such doesn't really
> matter. But freeing a large dict, particularly one that contains
> complex objects, can have serious consequences for timing. Defensively
> programming against that isn't normal Python style, and in the absence
> of such defensive programming, I'd say that Python isn't really
> deterministic.
I agree completely. Hard realtime requires *guarantees* == determinism.
Moreover, the very nature of most hard realtime systems requires direct
programmatic interaction with the interrupt infrastructure of the target
system. With enough glue code, one might be able to do this, but it would
be pretty ugly, I suspect.
Also, I feel it is impossible to talk about any kind of "realtime" and only
discuss the language. Realtime behavior (and the consequent guarantees of
timing and sequencing) is the sum total of the language AND the hardware AND
the operating system (if any). Absent the right hardware support you cannot
write correct hard realtime systems in *Assembler*! DAMHIKT ;))
Will Stuyvesant
> ...
> Will Stuyvesant fed this fish to the penguins on Monday 14 October
> ...
Yay, and the fish turns out to be a whale. Oh no, thats not a fish,
that is a mammal. A shark then.
> ...
> import Queue
> ...
Much better to use Queue. Thank you for showing your code.
> def testFunctionC(a):
> # takes about 4 seconds with a == 19 on my computer,
> # keeps the cpu very busy
> tmp = 2L
> for t2 in range(a):
> tmp = tmp * 2L
> while tmp > 0:
> t2 = 2L * t2
> tmp = tmp - 1
> return t2 <> 12345L
I have been looking for some kind of function that keeps the cpu doing C
stuff **very** busy (demonstrates treading.Tread.join(timeout) failure),
with a parameter that gives a linear behavior in time, so
testFunction(2*a) would take twice as long. But such a test function is
hard to find isn't it.
> ...
> # your exponential equation, apparently the function took 12743msec
> ...
Not mine...I think I found the equation in a Tim Peters posting. And it
turns out this notebook is doing that function about 3 times faster than
yours. *grin*
> ...
> IMPORTANT NOTICE -- Even though ALL cases resulted in a TIMEOUT
> return, the program itself cannot exit until thread-4 finishes its
> calculation!
> ...
That was expected. Unless that thread can be killed, but it just
is not listening and even keeps the caller from doing things!
What I thought was possible was to have the main program executing
statements at least every 200ms, or 500ms, or even 5000ms for that
matter. And that is just not to be. Not using threads anyway.
So for now it is byebye to my idea for modeling realtime software
this way.
But...it has been fun experimenting and I guess we did learn a
thing or two. Thanks!
'''
QOTD, about experiment
Bowie's Theorem:
If an experiment works, you must be using the wrong equipment.
'''
Will Stuyvesant
> ...
> There is an inherent problem with the term "realtime" if it is
> not further qualified.
> 1) "Hard" realtime - ...
> ...
> 2) "Soft" realtime - ...
> ...
I was looking for a primitive that can be used for modeling *hard*
realtime, but without the ``service completed'' requirement. ``I
cannot do this in time'' is also acceptable at timeout. The point
is that the caller should be able to carry on.
> ...
> http://www.tundraware.com/Technology/Bullet/
> ...
And you have more interesting stuff there.
> HOWEVER, you said you wanted to "prototype" such a system.
> Depending on how functional your prototype needs to be, Python
> may well be a good choice. It is relatively easy and quite
> appropriate to *model* realtime systems, hard and soft, in other
> languages.
Well as you can see in this thread I failed using Python! Maybe
my requirements are too strict. For modeling ``ordinary''
function calls a combination of Queue.Queue and
threading.Thread.join(timeout=...) can be used. But if the
function call winds up doing cpu intensive stuff (for example
pow()) at the C level then the Python threading timeout fails, so
the caller can not carry on in time. And *that* is essential for
the realtime modeling primitive. Performance is no issue, the
simulation is allowed to be very slow, as long as it does
implement the primitive correctly.
If I have more time I am going to look into doing this with
processes instead of threads.
Thank you for your comments!
'''
QOTD, about modeling
Recession is when your neighbor loses his job. Depression is when
you lose your job. These economic downturns are very difficult to
predict, but sophisticated econometric modeling houses like Data
Resources and Chase Econometrics have successfully predicted 14 of
the last 3 recessions.
'''
Aahz
>requirements are too strict. For modeling ``ordinary'' function calls
>a combination of Queue.Queue and threading.Thread.join(timeout=...) can
>be used. But if the function call winds up doing cpu intensive stuff
>(for example pow()) at the C level then the Python threading timeout
>fails, so the caller can not carry on in time. And *that* is essential
>for the realtime modeling primitive. Performance is no issue, the
>simulation is allowed to be very slow, as long as it does implement the
>primitive correctly.
Ah! You need to set up your C primitives to release the GIL. That
should be easy as long as your C functions don't need to communicate
with other threads *or* Python while they operate.
Dennis Lee Bieber
2002 08:41 am:
>
> Much better to use Queue. Thank you for showing your code.
>
You've probably run into my later post, with the quote from Python
Essential Reference...
> it turns out this notebook is doing that function about 3 times faster
> than yours. *grin*
>
I've got an old machine (I've never bought /the/ top-of-line, always
one step down)... Only a mere 733MHz with pc700-ecc RDRAM -- not that
the RAM would have been involved for an equation that should have fit
inside the processor cache...
Greg Ewing
> But if the
> function call winds up doing cpu intensive stuff (for example
> pow()) at the C level then the Python threading timeout fails
> ...
> Performance is no issue, the
> simulation is allowed to be very slow, as long as it does
> implement the primitive correctly.
In that case, just don't do anything intensive in C!
Keep it all in pure Python.
--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
Dennis Lee Bieber
03:21 pm:
>
> In that case, just don't do anything intensive in C!
> Keep it all in pure Python.
>
As my last listing demonstrated -- where I very crudely replaced the
** math with a loop using *; each multiply being short enough to let
the Python thread swapping to activate in time to catch the time-out
(but it did leave the work thread crunching for another 30 minutes <G>).
--
Will Stuyvesant
> > Will Stuyvesant wrote:
> >
> > But if the
> > function call winds up doing cpu intensive stuff (for example
> > pow()) at the C level then the Python threading timeout fails
> >
> > ...
> >
> > Performance is no issue, the
> > simulation is allowed to be very slow, as long as it does
> > implement the primitive correctly.
>
> In that case, just don't do anything intensive in C!
> Keep it all in pure Python.
I did hope that would be a reliable solution. But alas it is not:
doing something like ``x = 2L * 2L * 32'' will send control to the C
level for too long. For ordinary purposes it may not be much of a
problem. But for a realtime simulator...
'''
Magic is always the best solution -- especially reliable magic.
'''
Will Stuyvesant
> Ah! You need to set up your C primitives to release the GIL. That
> should be easy as long as your C functions don't need to communicate
> with other threads *or* Python while they operate.
I am only doing Python programming, so I have no idea how to release a
GIL. Also, I am on Windows (without M$ compiler).
Can you point to a list of Python functions that release the GIL? If
it exists... For example the ** operator does not...as can be seen in
an earlier post. A list with safe and unsafe functions would be
useful!
'''
Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are, by
definition, not smart enough to debug it.
-- Brian W. Kernighan
'''
Bengt Richter
It sounds a bit funny to me to be worrying about "too long" (presumably
_real_ time?) in a realtime *simulator* (especially after saying performance
is no issue). I guess you're not simulating time itself in your realtime simulator?
(I probably missed something, jumping in just on this post ;-)
Regards,
Bengt Richter
Aahz
>>
>> Ah! You need to set up your C primitives to release the GIL. That
>> should be easy as long as your C functions don't need to communicate
>> with other threads *or* Python while they operate.
>
>I am only doing Python programming, so I have no idea how to release a
>GIL. Also, I am on Windows (without M$ compiler).
>
>Can you point to a list of Python functions that release the GIL? If
>it exists... For example the ** operator does not...as can be seen in
>an earlier post. A list with safe and unsafe functions would be
>useful!
Generally speaking, only/all I/O related functions release the GIL.
Greg Ewing
> It sounds a bit funny to me to be worrying about "too long" (presumably
> _real_ time?) in a realtime *simulator* (especially after saying performance
> is no issue). I guess you're not simulating time itself in your realtime simulator?
I've been wondering about that, too. If you're seriously
trying to simulate the behaviour of a real-time system,
you want the timeout to occur after a certain amount of
*simulated* time, not actual time. To be able to do that,
your simulator needs to have a notion of how much
simulated time each operation takes, and keep track
of it.
So, before going any further, the question which must
be answered is: how accurately does the behaviour of
the simulated system in simulated time have to reflect
the behaviour of the real system in real time?
Tim Daneliuk
> Bengt Richter wrote:
>
>> It sounds a bit funny to me to be worrying about "too long" (presumably
>> _real_ time?) in a realtime *simulator* (especially after saying
>> performance
>> is no issue). I guess you're not simulating time itself in your
>> realtime simulator?
>
>
>
> I've been wondering about that, too. If you're seriously
> trying to simulate the behaviour of a real-time system,
> you want the timeout to occur after a certain amount of
> *simulated* time, not actual time. To be able to do that,
> your simulator needs to have a notion of how much
> simulated time each operation takes, and keep track
> of it.
>
> So, before going any further, the question which must
> be answered is: how accurately does the behaviour of
> the simulated system in simulated time have to reflect
> the behaviour of the real system in real time?
>
All that is true, but the problem here is not absolute time response but
*determinism*. If I understand the commentary above (and I may well not!)
the issue is that you cannot get predictable (== deterministic) behavior
when python hands control down to a lower layer of code written in 'C'.
IMHO, this has much less to do with the way the code is implemented in 'C'
and more to do with the fact that the underlying *system* (unix, win32,
macos) is not deterministic. For example, if the C function invoked happens
to do some disk I/O you will get rather large variations in response times
depending on how busy the VM system and disk I/O subsystem are. This has
little to do with C and everything to do with the aforementioned
subsystems.
Similarly, and having nothing to do with 'C' code, the simulation itself
will have variable timing even if written in pure Python simply because
housekeeping like garbage collection can influence the 'realtime'
determinism of the system.
Tim Daneliuk
One last thing: The scenarios and problems outlined above can be avoided if
the simulation in fact simulates *time* itself (I think Mr. Ewing was
hinting at this as I reread his post). In this case, a clock "tick" is
merely a software counter which moves things forward one system quantum. In
such simulations, actual "real" time is not preserved but determinism and
sequencing are. This can be very useful in examining the behavior of a
proposed or existing real world system.
In this case Python absolutely could and would serve nicely as an
implementation vehicle. Some very interesting OO design issues emerged as I
thought about this. What should the central set of objects include, for
example? Clearly hardware events like interrupts need to be represented,
but so does time itself - or at least the behavior of hardware clocks.
'Mighty interesting project indeed...
Dennis Lee Bieber
04:10 pm:
>
> One last thing: The scenarios and problems outlined above can be
> avoided if the simulation in fact simulates *time* itself (I think Mr.
> Ewing was hinting at this as I reread his post). In this case, a clock
> "tick" is merely a software counter which moves things forward one
> system quantum. In such simulations, actual "real" time is not
> preserved but determinism and sequencing are. This can be very useful
> in examining the behavior of a proposed or existing real world system.
>
> In this case Python absolutely could and would serve nicely as an
> implementation vehicle. Some very interesting OO design issues emerged
> as I thought about this. What should the central set of objects
> include, for example? Clearly hardware events like interrupts need to
> be represented, but so does time itself - or at least the behavior of
> hardware clocks. 'Mighty interesting project indeed...
>
Would the announced SimPy be a starting place? Of course, you'd likely
have to build a model of the Python interpreter in order to simulate
the threading with emulated time <G>
Will Stuyvesant
> ...a certain amount of *simulated* time, not actual
> time...your simulator needs to have a notion of how much
> simulated time each operation takes, and keep track
> of it.
>
> So, before going any further, the question which must
> be answered is: how accurately does the behaviour of
> the simulated system in simulated time have to reflect
> the behaviour of the real system in real time?
Yes indeed there are several approaches to simulation realtime. What
I am working on is an academic project (with industrial partners
though) and I am not tied to existing methodology but free to 'invent'
something new.
Sorry I cannot show you much more since they have this disclosure
thing. Also am busy today and I am also preparing a response to Tim
Daneliuk on this thread since it looks like he is on target too with
ideas.
Maybe you do know the CSP realtime book by Steve Schneider? That also
covers realtime issues and our research is going in that direction.
One of my ideas was to drop the variables and logic for simulating
time and using *real time*. It would not matter how much slower that
would be. And it also would not matter if the 'rcall' function
described earlier in this thread would return timeouts more often than
in a real realtime system with a realtime OS.
But it looks like we found the limit of that approach already (also
see the posts by mr. Bieber on this thread).
'''
Simulations are like miniskirts, they show a lot and hide the
essentials.
-- Hubert Kirrman
'''
Armin Steinhoff
> Will Stuyvesant wrote:
> > We would like to prototype a realtime system. Is anybody doing this using Python?
> >
> > '''
> > A lost ounce of gold may be found, a lost moment of time never.
> > '''
>
> Much has already been posted here but I would add this:
>
> There is an inherent problem with the term "realtime" if it is not
> further qualified. All realtime systems must typically cope with external
> asynchronous events as their principal paradigm - i.e., Asynchronous
> events determine control flow of the code not some master algorithm.
> However, all such systems are not equal. Generally, we can think about
> two classes of realtime systems:
>
> 1) "Hard" realtime - cases where timing must be predictable and guaranteed.
hard real-time systems results must be provided in a timely manner.
> For example, "You must respond to the Flabergast Interrupt in less than
> 100 us and guarantee it is serviced completely in 1ms."
No ... speed has nothing to do with hard real-time ... it is only a
technological attribute which says something about the possible time
resolution of a real-time system. The interrupt could be handled in
100h ... but the service _must_ be completed by meeting its deadline.
Only timely provided results a valid results ... that's simply hard
real-time.
> This kind of
> thing is common in machine control and military fire control systems.
>
> 2) "Soft" realtime - cases where you must respond to external events in
> a reasonable amount of time and preserve proper *sequencing*, but the
> absolute amount of time is neither guranteed nor particularly important
> unless it gets ridiculously long. A good example of this is the
> message processing and dispatching common in a GUI manager.
Results are valid in a soft real-time if the results are provided with
a defined jitter around a deadline.
>
> I would argue that Python is possibly suited to 2) and probably poorly suited
> to 1).
Yes ... the CPU time for the garbage collection is completely
unpredictable, so Python could be used just for soft real-time
applications.
But ... when I remember rigth, there is a possibilty to switch off
Python's garbage collection .. isn't it?
> The reason I say this is that things like background garbage
> collection, interaction with lower-level system services like I/O and so on
> make guaranteed, predictable service times essentially impossible (unless
> the guarantees are so weak and the permissible times so long that it does
> not matter - but this is not ordinarily the case in hard realtime).
>
> It might be possible to shoehorn Python into a hard realtime environment
> for some functions like dispatching if one wrote a sufficient amount of
> binder code in a lower level language like 'C' or Assembler, but why
> bother? Python is a marvelous language with many merits, but it cannot do
> everything - no language can, nor should it be expected to. - Shameless
> promotion of my ideas on *that* subject can be found at:
>
> http://www.tundraware.com/Technology/Bullet/
Under QNX you can use a C coded message passing extension in order to
communicate with an external process which can do the hard real-time
(HRT) stuff for Python. In such a case Python and the HRT process are
running in seperate protected address spaces. Sending a message with
~64 bytes and receiving it back takes 5 microseconds ... so it is
realy fast.
This is a very good alternative to build an other kind of dynamical
extensions for Python for robust and secure RT applications.
Use Python for the top level logic and RT extensions for the HRT
processing :)
> If you shudder at writing in Assembler or 'C', give Forth a good look
> for this sort of thing.
Pyrex is much better than Forth :)
Cheers
Armin Steinhoff
Greg Ewing
>> the problem here is not absolute time response but
>> *determinism*. If I understand the commentary above (and I may well not!)
>> the issue is that you cannot get predictable (== deterministic) behavior
>> when python hands control down to a lower layer of code written in 'C'.
If determinism is really *all* that's needed, it could be
achieved by equating one unit of simulated time with one
Python bytecode. Stackless Python may be able to help here,
with its bytecode-count-based task switching.
Dennis Lee Bieber
2002 12:08 am:
> see the posts by mr. Bieber on this thread).
>
<My Heart!>
/I've/ been cited as a witness in this matter? How frightening! <G>
Johannes Nix
> i would say that a normal OS (Windows, Linux, others) is suitable for real
> time down to the range to 1s timesteps. longer times beeing easy. there's
> of course no guarantee that even that can be met. there much longer delays
> (e.g. accessing the floppy under windows or deleting a large file under
> linux ext2, spinning up a HD from low power when accessed)
I am doing real-time audio processing with Python. The OS is Linux
2.4.17 with Andrew Mortons low-latency patches. The audio driver is
ALSA-0.9. Sound hardware are 32-Bit RME sound cards with optical
connections and external AD/DA converter, about 120 dB dynamic range
and up to 8 channels. The driver binding is written in C, most of the
other processing (noise supression and hearing aid algorithms) is
written in Python using the Numeric module. The worst-case latencies
are in the range of 25 - 50 milliseconds. "Latencies" here does not
mean interrupt latencies but the time lag between the input signal and
the output signal. Part of this is because the RME cards have very
small buffers with fixed fragment sizes so that an extra I/O thread
with buffers on the applications side is needed. I guess very much
that lower latencies in the 2 - 4 millisecond range are with modest
effort reachable on the same system. The main problem here is that for
standard FFT processing one needs about 600 samples of buffering.
Linux-2.4 with low-latency patches provides worst case latencies well
below 0.5 milliseconds. It's possible that one has to use SCSI disks
because not all ATA disks can be tuned by hdparm.
CPU hardware has been a dual-processor UP2000 with 750 MHz Alpha CPUs,
but now we've switched to AMD and Pentium III CPU with 1800 MHz, in
some cases they are still a bit slower. In terms of computing power
the system is largely equivalent to a VME-Bus based Pentek System with
a SUN sparc host workstation and five Texas Instrumens TMSC40320 DSP
Processors (40 and 50 MHz), but perhaps twenty times faster to program
and at the fifteenth part of the cost. I've reduced about 10000 lines
of C to 1000 lines Python. For example, a standard spectral
subtraction algorithm needs for the core part (not counted window
analysis / overlap add synthesis) about 120 lines, including a
socket-based interface to change parameters at runtime.
Because the Global Interpreter Lock (GIL), parallel processing in Python is
somewhat complicated. The solution I found was to fork the Python
process and exchange data by shared memory access (there does exist an
"shm" module ).
> the fact that it uses Python or an other languages does not matter all that
> much. some make it easier, some harder. of course, Java with it's gc in the
> background makes predictions how long something takes harder, especialy for
> short times, that's a problem in Jython. CPython is much easier to
> understand with it's reference counting.
I think reference counting helps also to get quite deterministic
behaviour.
I'd say that Python with Numeric and psyco is an excellent language
for many real-time purposes once the GIL is replaced by per-object
locks. I already was thinking about a PEP.
Once there is an easy-to use, powerful graphics package for scientific
graphics, Python will be also a very good engineering language which
has all requirements to replace many applications of Matlab (which is
the de facto standard). I guess you can do all what you need with the
Gnuplot, Dislin and OpenDX bindings but it's still too complicated for
many people.
Johannes