Events
Dan Sugalski
want to address it quickly so we don't end up going too far afield
yet.
Events are notice of something happening (though different from
notficications, which are separate, I think). IO was complete,
signals pinged, timers fired, whatever. Something happened.
Events should be checked and processed in several instances:
1) When a blocking IO op is being performed. All IO is async, so a
blocking request is really a "queue IO request and drain event queue
until notice of completion is returned"
2) When the queue is explicitly drained. A draineventqueueforever op,
or something much like it
3) When we do a quick check to see if there are pending events to be processed
The first is done in the case of readw or writew, for example. The
second for event-driven programs that set up callbacks and park
themselves forever in one big ProcessEvent call. (Tk programs come to
mind) The third is to be scattered in generated code to make sure
that events are occasionally checked for, and will probably drain
only high-priority events and at most a single low-priorty event.
It's been suggested that a "checkevent" keyword get added to the
signature of ops that should check for events as part of their
processing, which isn't a bad idea either.
So the deal is that everything that generates events throws an entry
in the event queue, and the interpreter as it runs grabs events out
of the queue when it chooses and processes the events.
We still need to talk about event records, callbacks, what blocks
what, and privilege issues with events, which we ought to get out of
the way now.
--
Dan
--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Damien Neil
> The first is done in the case of readw or writew, for example. The
> second for event-driven programs that set up callbacks and park
> themselves forever in one big ProcessEvent call. (Tk programs come to
> mind) The third is to be scattered in generated code to make sure
> that events are occasionally checked for, and will probably drain
> only high-priority events and at most a single low-priorty event.
While it's possibly politically impossible (many people are very
attached to Unix signals), I'd really rather work with a system
that does async event dispatch exclusively through threads.
Interrupting a thread in the middle of its execution and sending
it haring off to an interrupt handler is not only clumsy and difficut
to implement, it's a recipe for buggy code.
Much nicer would be if events were always dispatched in one of two ways:
- Synchronously, by calling a GetNextEvent or ProcessEvent function.
- Asynchronously, by spawning a new thread and executing the signal
handler within it.
Is there any hope for rethinking the desire to expose the ugliness of
Unix signals in the Parrot VM?
- Damien
Dan Sugalski
>On Thu, Jul 17, 2003 at 12:58:12PM -0400, Dan Sugalski wrote:
>> The first is done in the case of readw or writew, for example. The
>> second for event-driven programs that set up callbacks and park
>> themselves forever in one big ProcessEvent call. (Tk programs come to
>> mind) The third is to be scattered in generated code to make sure
>> that events are occasionally checked for, and will probably drain
>> only high-priority events and at most a single low-priorty event.
>
>While it's possibly politically impossible (many people are very
>attached to Unix signals), I'd really rather work with a system
>that does async event dispatch exclusively through threads.
>Interrupting a thread in the middle of its execution and sending
>it haring off to an interrupt handler is not only clumsy and difficut
>to implement, it's a recipe for buggy code.
We're not doing the traditional C-style interrupt. That's infeasable
at the level we operate at generally, so we're not. We *are*
simulating it, since potentially some of Parrot's instructions will
be 'interruptable' or check interrupts, or whatever you want to call
it. (I call it annoying but necessary :)
>Much nicer would be if events were always dispatched in one of two ways:
> - Synchronously, by calling a GetNextEvent or ProcessEvent function.
> - Asynchronously, by spawning a new thread and executing the signal
> handler within it.
Nope, that won't work. A lot of what's done is really "main thread
with interrupts" and that doesn't map well to doing all signal
handling in separate threads. You really do want to pause the main
program to handle the events that are coming in, if they're events of
sufficient importance. Generally I put them in three classes--hard
interrupts (signals), soft interrupts (IO completion stuff), and
events (fuzzy user-level stuff). Hard and soft interrupts should get
dealt with as soon as possible, events should probably wait until
something explicitly decides to process an event.
>Is there any hope for rethinking the desire to expose the ugliness of
>Unix signals in the Parrot VM?
It's not just signals, there's a lot of stuff that falls into this
category. We've got to deal with it, and deal with it properly, since
not dealing with it gets you an 80% solution.
Leopold Toetsch
> Events should be checked and processed in several instances:
> 3) When we do a quick check to see if there are pending events to be processed
I have now a patch ready, which does 3) without additioal performacne
penalty for all cores except JIT.
(A short description was in an different thread WRT Event handling from
today).
$ diffstat -w 65 event\ handling-2
classes/timer.pmc | 5 -
config/gen/config_h/config_h.in | 4
core.ops | 2
events.c | 68 +++++++++++--
exceptions.c | 2
include/parrot/events.h | 10 +-
include/parrot/interpreter.h | 9 +
include/parrot/oplib.h | 11 +-
interpreter.c | 125 +++++++++++++++++--------
ops2c.pl | 124 ++++++++++++++++++++++--
t/op/hacks.t | 36 ++++++-
11 files changed, 324 insertions(+), 72 deletions(-)
$ l event\ handling-2
-rw-r--r-- 1 lt users 22139 Jul 18 17:53 event handling-2
Should I send it to the list first (or anyone personally) or put it in?
leo
Dan Sugalski
List first, please.
Damien Neil
> Nope, that won't work. A lot of what's done is really "main thread
> with interrupts" and that doesn't map well to doing all signal
> handling in separate threads. You really do want to pause the main
> program to handle the events that are coming in, if they're events of
> sufficient importance. Generally I put them in three classes--hard
> interrupts (signals), soft interrupts (IO completion stuff), and
> events (fuzzy user-level stuff). Hard and soft interrupts should get
> dealt with as soon as possible, events should probably wait until
> something explicitly decides to process an event.
In my experience, interrupt handlers in Perl code generally fall
into three categories: Ones that set a flag to be checked later,
ones that perform an action and terminate the program, and buggy
ones subject to race conditions.
IO completion events in particular should not be handled by
interrupting the main execution thread. The appropriate action
required to handle these events will almost invariably require
access to data structures shared between the interrrupt handler
and the main thread. If you place the interrupt handler in the
main thread, you can't use locks to control access to these
structures (as the handler will wait on the main thread's lock,
while the main thread will block on the handler returning).
This leads to Unix-style signal masks, where interrupts are
blocked during critical sections. While this works, I strongly
feel that a platform with thread support is better off dispatching
interrupts to a separate thread and using the existing interthread
synchronization mechanisms, rather than introducing a separate
interrupt masking system.
Also, given that asynchronous IO is a fairly unpopular programming
technique these days (non-blocking event-loop IO and blocking
threaded IO are far more common), I would think long and hard before
placing support for it as a core design goal of the VM. If there
is a compelling reason to use AIO, the implementation may better
be handled at a lower level than Parrot; even if Parrot itself does
not support AIO at the opcode level, Parrot programs could still
use it by calling down to a C library.
> It's not just signals, there's a lot of stuff that falls into this
> category. We've got to deal with it, and deal with it properly, since
> not dealing with it gets you an 80% solution.
Outside of signals and AIO, what requires async event dispatch?
User events, as you pointed out above, are better handled through
an explicit request for the next event.
- Damien
Benjamin Goldberg
[snip]
> Also, given that asynchronous IO is a fairly unpopular programming
> technique these days (non-blocking event-loop IO and blocking
> threaded IO are far more common), I would think long and hard before
> placing support for it as a core design goal of the VM. If there
> is a compelling reason to use AIO, the implementation may better
> be handled at a lower level than Parrot; even if Parrot itself does
> not support AIO at the opcode level, Parrot programs could still
> use it by calling down to a C library.
AIO is unpopular because it's not widely/portably supported, whereas
non-blocking event-loop IO is, (one generally does have either select or
poll), as is blocking threaded IO (even if the thread starting stuff may
need to be different on some platform, it's *mostly* portable).
If we make it a core part of parrot, it will become more popular, simply
because of parrot's availability.
> > It's not just signals, there's a lot of stuff that falls into this
> > category. We've got to deal with it, and deal with it properly, since
> > not dealing with it gets you an 80% solution.
>
> Outside of signals and AIO, what requires async event dispatch?
> User events, as you pointed out above, are better handled through
> an explicit request for the next event.
Inter-thread notification and timers? True, these *could* be considered
to be "user events", but IMHO, there are times when we want a "user
event" to have the same (high) priority as a system signal.
--
$a=24;split//,240513;s/\B/ => /for@@=qw(ac ab bc ba cb ca
);{push(@b,$a),($a-=6)^=1 for 2..$a/6x--$|;print "$@[$a%6
]\n";((6<=($a-=6))?$a+=$_[$a%6]-$a%6:($a=pop @b))&&redo;}
Uri Guttman
BG> Damien Neil wrote:
BG> [snip]
>> Also, given that asynchronous IO is a fairly unpopular programming
>> technique these days (non-blocking event-loop IO and blocking
>> threaded IO are far more common), I would think long and hard before
>> placing support for it as a core design goal of the VM. If there
>> is a compelling reason to use AIO, the implementation may better
>> be handled at a lower level than Parrot; even if Parrot itself does
>> not support AIO at the opcode level, Parrot programs could still
>> use it by calling down to a C library.
BG> AIO is unpopular because it's not widely/portably supported, whereas
BG> non-blocking event-loop IO is, (one generally does have either select or
BG> poll), as is blocking threaded IO (even if the thread starting stuff may
BG> need to be different on some platform, it's *mostly* portable).
and event loop i/o doesn't usually support async file i/o. many people
conflate the two. event i/o handles sockets, pipes and such but none
support files. the issue is that the async file i/o api is so different
(or non-existant) on all the platforms. dan wants to make clean async
file i/o in the core by using a blocking thread on each i/o request and
synchronizing them with this event queue in the main thread. it has the
advantage of being easier to code and should work on most platforms
which have threads. and he wants async file i/o in parrot core since it
is faster and has to be in the core to be properly supported at higher
levels.
BG> If we make it a core part of parrot, it will become more popular,
BG> simply because of parrot's availability.
that is a major win since no other system i (or dan) has heard of has
portable async file i/o. and it will be integrated into the core event
handling so you will be able to mix and match async socket, terminal (on
unix at least) and file i/o with timers. this is what i want. :)
uri
--
Uri Guttman ------ u...@stemsystems.com -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
Adam Turoff
> Also, given that asynchronous IO is a fairly unpopular programming
> technique these days (non-blocking event-loop IO and blocking
> threaded IO are far more common), I would think long and hard before
> placing support for it as a core design goal of the VM.
What makes you think that Dan hasn't already thought long and hard about AIO?
This design issue has been settled for months now. And if you don't
think it's a particularly *good* design decision, think about this quote
from Alan Perlis[1]:
Symmetry is a complexity-reducing concept (co-routines include
subroutines); seek it everywhere.
AIO with files simplifies *all* IO handling. You get sync IO through a
thin wrapper over AIO. Socket IO is already async, so why have multiple
incompatible models? Why repeat the mistakes of the C standard library
for the umteenth time?
Z.
[1] Alan Perlis: (n) definately not an idiot
Damien Neil
> AIO is unpopular because it's not widely/portably supported, whereas
> non-blocking event-loop IO is, (one generally does have either select or
> poll), as is blocking threaded IO (even if the thread starting stuff may
> need to be different on some platform, it's *mostly* portable).
I disagree; AIO is not widely/portably supported because it is
unpopular. Threading (on Unix systems, at least) is a much newer
concept than AIO, and yet it is now nigh-ubiquitous; any modern OS
needs to have solid threading support to be taken seriously. Portable
libraries wrapping system-specific thread models are common. The only
reason this hasn't happened with AIO is lack of user demand.
The problem with AIO is that it has all the synchronization pain
of threading combined with the code flow complexity of an event-based
IO system. There are certianly occasions when AIO may prove to be
the best or most elegant solution to a problem, but in most cases
there are other approaches which are substantially simpler for the
programmer.
> If we make it a core part of parrot, it will become more popular, simply
> because of parrot's availability.
I'd be interested in seeing specific examples of problems that will
be solved by adding AIO support to the VM layer. How will this feature
be used in real-world programs?
> > Outside of signals and AIO, what requires async event dispatch?
> > User events, as you pointed out above, are better handled through
> > an explicit request for the next event.
>
> Inter-thread notification and timers? True, these *could* be considered
> to be "user events", but IMHO, there are times when we want a "user
> event" to have the same (high) priority as a system signal.
I'd like a specific example (general pseudocode fine) of inter-thread
notification implemented using interrupts that a) doesn't include
any race conditions, and b) can't be written more clearly using
non-interrupt based code.
I think you're vastly underestimating the difficulty of writing
interrupt-based code that doesn't include race conditions. Consider
that Parrot itself has given up on trying to do this: internally,
interrupts (signals) will simply result in an event being added to
a queue for later processing.
- Damien
Damien Neil
> and event loop i/o doesn't usually support async file i/o. many people
> conflate the two. event i/o handles sockets, pipes and such but none
> support files. the issue is that the async file i/o api is so different
> (or non-existant) on all the platforms. dan wants to make clean async
> file i/o in the core by using a blocking thread on each i/o request and
> synchronizing them with this event queue in the main thread. it has the
> advantage of being easier to code and should work on most platforms
> which have threads. and he wants async file i/o in parrot core since it
> is faster and has to be in the core to be properly supported at higher
> levels.
Right, there are two independent issues here: Support for asynchronous
IO (an OS feature distinct from non-blocking IO), and VM-level support
for interrupts in Parrot. The latter is what I am questioning.
It would be entirely possible for Parrot (or a Parrot library) to
use AIO at a low level, without introducing interrupts to the VM layer.
The fact that most event loops do not support async file IO on Unix
systems is due to a combination of deficiencies in the Unix APIs
(select() and poll() don't work on files), and lack implementation
in the event library. There is certainly no reason a traditional
event loop (such at Tcl's, which is an excellent example of a
well-done event system) can't use AIO at a low level to support
async file IO on Unix.
(I specifically refer to Unix above because many non-Unix systems
have perfectly good support for monitoring files in their equivalents
of select(). So do some Unixes, for that matter.)
Regarding AIO being faster: Beware premature optimization. And are
you referring to OS-level AIO (which often does have performance
advantages), or application-level AIO using a collection of threads
as you describe above (which I suspect will be slower than
single-threaded non-blocking IO, owing to synchronization costs
between threads)?
> that is a major win since no other system i (or dan) has heard of has
> portable async file i/o. and it will be integrated into the core event
> handling so you will be able to mix and match async socket, terminal (on
> unix at least) and file i/o with timers. this is what i want. :)
Do you want to use interrupt-based IO at the VM level, or do you
want an event system which will function cleanly on sockets,
terminals, and files?
- Damien
Dan Sugalski
>On Fri, Jul 18, 2003 at 05:58:41PM -0400, Uri Guttman wrote:
>> and event loop i/o doesn't usually support async file i/o. many people
>> conflate the two. event i/o handles sockets, pipes and such but none
>> support files. the issue is that the async file i/o api is so different
>> (or non-existant) on all the platforms. dan wants to make clean async
>> file i/o in the core by using a blocking thread on each i/o request and
>> synchronizing them with this event queue in the main thread. it has the
>> advantage of being easier to code and should work on most platforms
>> which have threads. and he wants async file i/o in parrot core since it
>> is faster and has to be in the core to be properly supported at higher
>> levels.
>
>Right, there are two independent issues here: Support for asynchronous
>IO (an OS feature distinct from non-blocking IO), and VM-level support
>for interrupts in Parrot. The latter is what I am questioning.
So we'll skip all the rest for the moment, and we can get into that
if we need to.
We're supporting interrupts at the interpreter level because we must.
It doesn't matter much whether we like them, or we think they're a
good idea, the languages we target require them to be there. Perl 5,
Perl 6, Python, and Ruby all have support for Unix signals in pretty
much the way you'd get them if you were writing C code. (That is to
say, broken by design, dangerous to use, and of far less utility than
they seem on the surface)
>It would be entirely possible for Parrot (or a Parrot library) to
>use AIO at a low level, without introducing interrupts to the VM layer.
Sure. But what'd be the point? Adding in interrupts allows a number
of high-performance idioms that aren't available without them. They
certainly won't be required, and most of the IO you'll see done will
be entirely synchronous, since that's what most compilers will be
spitting out. You don't *have* to use IO callbacks, you just can if
you want to.
>The fact that most event loops do not support async file IO on Unix
>systems is due to a combination of deficiencies in the Unix APIs
>(select() and poll() don't work on files), and lack implementation
>in the event library. There is certainly no reason a traditional
>event loop (such at Tcl's, which is an excellent example of a
>well-done event system) can't use AIO at a low level to support
>async file IO on Unix.
Yep, definitely. This is why there's the unified event system, so you
can do async I/O on files, on sockets, handle window system events,
interprocess messages and interthread messages with a single point of
coordination.
>Regarding AIO being faster: Beware premature optimization.
I'm going to start carrying a nerf bat around and smack people who
trot this one out.
Ignoring the fact that you can often get a 2x or 3x improvement in
throughput with async IO, the original quote referred to
*implementation* optimization, and was generally referring to fun
tricks like splitting data elements across the unused bits of other
struct members, bizarre bitwise ops, and odd side-effects. (And at
the time the quote came, hardware had a lot of odd side-effects which
were fun to use)
While it's not inappropriate to apply it to design, we're nowhere
near that point. This isn't premature optimization, or optimization
of any sort--it's design, and it should be done now. This is what
we're *supposed* to be doing. It's certainly reasonable to posit that
async IO is a bad design choice (won't get you very far, but you can
posit it :) but please don't trot out the "premature optimization"
quote.
> And are
>you referring to OS-level AIO (which often does have performance
>advantages), or application-level AIO using a collection of threads
>as you describe above (which I suspect will be slower than
>single-threaded non-blocking IO, owing to synchronization costs
>between threads)?
You may suspect, but you'd turn out to be incorrect--using threads to
simulate a real async IO system still has performance wins. And we're
going to be using native async stuff when we can.
Michal Wallace
> >It would be entirely possible for Parrot (or a Parrot library) to
> >use AIO at a low level, without introducing interrupts to the VM layer.
> Sure. But what'd be the point? Adding in interrupts allows a number
> of high-performance idioms that aren't available without them. They
> certainly won't be required, and most of the IO you'll see done will
> be entirely synchronous, since that's what most compilers will be
> spitting out. You don't *have* to use IO callbacks, you just can if
> you want to.
Hey Dan,
I'm just curious... When you say IO callbacks, does that
mean external C code calling back into the parrot VM?
I wrote a python interface to the win32 MIDI routines a
while back. At first I just called the DLL and that worked
fine for output, but if I tried to read MIDI events, it
didn't work: I'd press a note on the keyboard and python
would crash. The fix was to actually write a C module
that locked the interpreter whenever it received
a callback.
Will that kind of locking be necessary in parrot, or
would these IO callbacks allow outside DLL's to
callback to parrot code automatically?
Sincerely,
Michal J Wallace
Sabren Enterprises, Inc.
-------------------------------------
contact: mic...@sabren.com
hosting: http://www.cornerhost.com/
my site: http://www.withoutane.com/
--------------------------------------
Dan Sugalski
>On Sun, 20 Jul 2003, Dan Sugalski wrote:
>
>> >It would be entirely possible for Parrot (or a Parrot library) to
>> >use AIO at a low level, without introducing interrupts to the VM layer.
>
>> Sure. But what'd be the point? Adding in interrupts allows a number
>> of high-performance idioms that aren't available without them. They
>> certainly won't be required, and most of the IO you'll see done will
>> be entirely synchronous, since that's what most compilers will be
>> spitting out. You don't *have* to use IO callbacks, you just can if
>> you want to.
>
>Hey Dan,
>
>I'm just curious... When you say IO callbacks, does that
>mean external C code calling back into the parrot VM?
That should work just fine, though it wasn't what I was talking about.
>Will that kind of locking be necessary in parrot, or
>would these IO callbacks allow outside DLL's to
>callback to parrot code automatically?
There'll have to be some mechanism defined, depending on what you
want to do--you'll be able to choose either posting an event to the
queue for processing, or calling directly back in. There'll be
limitations with both, but they shouldn't be bad.
Damien Neil
> We're supporting interrupts at the interpreter level because we must.
> It doesn't matter much whether we like them, or we think they're a
> good idea, the languages we target require them to be there. Perl 5,
> Perl 6, Python, and Ruby all have support for Unix signals in pretty
> much the way you'd get them if you were writing C code. (That is to
> say, broken by design, dangerous to use, and of far less utility than
> they seem on the surface)
Right, which is why I said in my initial message that dropping
interrupts might be politically impossible.
I still think that including something that is broken by design,
dangerous to use, and of questionable utility isn't a good idea,
but I can accept the argument that it may be necessary.
> >It would be entirely possible for Parrot (or a Parrot library) to
> >use AIO at a low level, without introducing interrupts to the VM layer.
>
> Sure. But what'd be the point? Adding in interrupts allows a number
> of high-performance idioms that aren't available without them. They
> certainly won't be required, and most of the IO you'll see done will
> be entirely synchronous, since that's what most compilers will be
> spitting out. You don't *have* to use IO callbacks, you just can if
> you want to.
Could point me at a reference for these high-performance idioms?
While I've heard of significant gains being realized through AIO,
it was my understanding that this is generally related to disk IO,
where Unix doesn't provide support for non-blocking IO. The
performance gains come not from a different code flow, but from the
ability to perform disk access in the background.
(I'm not disputing that such idioms exist; if there's a better
way to do things that I don't know of, I want to know more about it!)
> >Regarding AIO being faster: Beware premature optimization.
>
> I'm going to start carrying a nerf bat around and smack people who
> trot this one out.
The fact that it is often said does not make it any less true.
You've asserted that Parrot will be faster (in at least some
situations) with interrupt-driven IO than it will be with
non-interrupt-driven IO. I'm unconvinced of this claim. In
particular, I feel that support for interrupts will come at an
overall performance penalty, and I am unconvinced that this penalty
will not outweigh any benefits that interrupt-driven IO would bring.
Now, you can ignore me if you want; you're the designer. Hitting
me isn't going to convince me of anything, however.
> While it's not inappropriate to apply it to design, we're nowhere
> near that point. This isn't premature optimization, or optimization
> of any sort--it's design, and it should be done now. This is what
> we're *supposed* to be doing. It's certainly reasonable to posit that
> async IO is a bad design choice (won't get you very far, but you can
> posit it :) but please don't trot out the "premature optimization"
> quote.
This is *exactly* the time when that quote is appropriate to apply.
When a design decision is made "because it'll be faster that way",
it is always worth examining the question of whether it WILL be
faster or not. (I am aware that there is a second reason for
supporting interrupts in Parrot--Unix signals; I was addressing the
argument that support for AIO is sufficient reason to include
interrupts.)
For example: If it turns out that Parrot, sans interrupt-driven IO,
is capable of saturating the system bus when writing to a device,
there is little point in optimizing Parrot's IO system.
> You may suspect, but you'd turn out to be incorrect--using threads to
> simulate a real async IO system still has performance wins. And we're
> going to be using native async stuff when we can.
Do you know of a program that does this (simulated AIO via threads)?
(Again, I'm not disputing your claim--it's just that this is
completely contrary to my experience, and I'd like to know more
about it.)
- Damien
Jos Visser
>
> Do you know of a program that does this (simulated AIO via threads)?
> (Again, I'm not disputing your claim--it's just that this is
> completely contrary to my experience, and I'd like to know more
> about it.)
>
Lots of Java programs simulate AIO by using threads. Almost all Java IO
mechanisms are blocking and this means that doing interesting work while
the IO is in progress requires some mechanism that involves a separate
thread with a callback routine.
In fact, JMS (the Java Messaging Standard) supports a MessageListener
interface where the JMS system will call you back (the onMessage method)
when a new message has arrived.
++Jos.nl
--
ek is so lug jy vlieg deur my
sonder jou is ek sonder patroon
"Breyten Breytenbach"
Damien Neil
> Lots of Java programs simulate AIO by using threads. Almost all Java IO
> mechanisms are blocking and this means that doing interesting work while
> the IO is in progress requires some mechanism that involves a separate
> thread with a callback routine.
This is not AIO.
I am using "asynchronous I/O" (AIO) to refer to interrupt-driven
facilities such as the Unix aio_*() syscalls. (I recognize that
it is not uncommon to refer to non-blocking IO as asynchronous IO,
but I believe that I have been careful about clearly defining terms
in this thread.) The standard Java libraries perform blocking IO
dispatched from threads. Some Java programs, as you say, build a
single-threaded, non-blocking, event-dispatched IO mechanism on top
of this. Java does not, however, have any support for interrupts;
it is not possible to do AIO in Java.
- Damien
Jos Visser
> Some Java programs, as you say, build a
> single-threaded, non-blocking, event-dispatched IO mechanism on top
> of this. Java does not, however, have any support for interrupts;
> it is not possible to do AIO in Java.
You are (of course :-) correct. Java has no support for "true" AIO since
there are (fortunately) no facilities for interrupting a thread to
handle an async signal and then have that thread resume its work.
That's why this is widely simulated as (you and I) described: by having
a thread dispatch a blocking I/O call and sending an event to
interested parties (registered event listeners) as the I/O completes.
There are interesting synchronisation and threading issues involved in
this because the thread that executes the callback routine is (almost
always) a different one than the one that registered its interest in
the event.
As a veteran writer of MT-code and teacher of classes on that topic it
is my personal opinion that we would be much better off with a
good/clean system for non-blocking I/O and handling async (Unix) signals
than with an interface that exposes the full "power" of async signals
for everyone to use as they see fit. In my classes (e.g. Programming
multi-threaded C with Posix Threads) I try to steer my students towards
using one of the available patterns for handling signals because
designing and building a solid signal handler that works in all
circumstances is beyond most programmers (including me).
Dan Sugalski
[Stuff I've snipped]
I was going to refute this point by point, but I was getting cranky
and the responses were less than helpful. So let's start over.
First, to get it out of the way, I don't have to convince you of
anything. You have to convince me. For better or worse I'm
responsible for the design and its ultimately my decision. If you
don't want async IO, it's time to make a very, very good case.
(Trotting out tired and massively overused aphorisms isn't a good way
to do that)
Now, that having been said, there are some things we *aren't* doing.
We're not executing parrot bytecode code at interrupt time. Interrupt
handlers put events into parrot's event queue for processing later.
We're *not* forcing anyone to use async IO. I fully expect that 99%
of the code emitted by compilers will be synchronous. That's fine.
Having an async core doesn't in any way preclude synchronous IO.
We're not going to block on a file read or write. Ever. (Well, except
when systems lie and say data's ready and it isn't and we block
because we believe them) If we do we can't process signals, handle
socket IO, process timer events, or handle GUI events.
Here's what we *are* doing.
We are going to do all I/O under the hood asynchronously. Completed
async IO puts an event in the event queue.
We are going to have a single unified event queue. All IO, timer,
signal, and UI events go in it. All of 'em. We aren't going to do any
of this "we have N different places to look for data" nonsense, where
N is greater than three.
Apps are never required to deal with anything asynchronously if they
don't want. If you don't want signals, block them. The events for
them won't get delivered, or will be delivered and discarded. Want to
explicitly drain the queue only when your app wants it? Fine. You can
do that too.
The event queue generally gets drained (unless otherwise blocked)
when waiting for an event (if you've not made signals plain events),
when waiting for an IO request to complete, or when you sit and
explicitly drain the queue.
On some platforms this will mean we'll need to spawn off a thread for
a filehandle, if the platform has no or really lousy native async
facilities. Not great, but better than nothing, because even faking
it this way has the potential for performance boosts. (Ignoring the
potential speculative reads we might issue asynchronously ourselves)
I've written async IO code to good native async facilities, seen
async code to native facilities that blow the doors off any
synchronous code I've ever seen, and written fake async IO with
threads. It all has a throughput win. Go hit google or citeseer if
you want to find some.
Bottom line is that parrot's IO will be asynchronous, with most code
using a synchronous interface on top of it. That code'll get the
benefits of true base asynchrony (being responsive to events while
waiting) without having to do anything explicitly asynchronous. Code
that wants to take the next step can.
Layering a synchronous interface on an async core is simple, but
layering an async interface on a sync core is not. Moreover, parrot
code *can't* truly layer an async interface on a sync core, because
that layering, for it to work properly, has to be done at the lower
levels of the interpreter. Which is where we're going to do it there,
and one of the reasons we're doing it in general.
Michal Wallace
Hey all,
I've been thinking about the "compiling python to
parrot" concept. Right now it looks like the
approach is to start from scratch, but I'm
wondering if it might make more sense to
leverage python itself, at least for now?
Python has a compiler module (written in python
and standard with the distribution) that can
take a python parse tree and produce python
byte code. It basically just walks the tree
and has a method for each python structure.
The actual parser is written in c, but there's
a drop-in replacement (or at least a partial one)
written in python described here:
http://codespeak.net/moin/pypy/moin.cgi/BytecodeCompiler
Would it make sense to use this to boostrap the
python parrot compiler? I was thinking about taking
a shot next week at replacing compiler.pycodegen.CodeGenerator
with something that produced IMCC.
The thing is, I don't have a lot of experience when
it comes to compilers, but I do know a whole lot about
python. :) If this approach makes sense, is there
someone with IMCC experience who'd be willing to do
some virtual pair programming with me and spike out
a prototype?
Benjamin Goldberg
[snip]
> Here's what we *are* doing.
>
> We are going to do all I/O under the hood asynchronously. Completed
> async IO puts an event in the event queue.
>
> We are going to have a single unified event queue. All IO, timer,
> signal, and UI events go in it. All of 'em. We aren't going to do any
> of this "we have N different places to look for data" nonsense, where
> N is greater than three.
Don't forget, flock(), fcntl(), ioctl(), wait()/waitpid(), all the SysV
stuff, all the (get|set|end)(net|gr|pw)(ent|.id|name?) functions,
sendmsg, recvmsg, ... :)
I've occasionally wanted to do IO while waiting for an exclusive lock on
a filehandle, but I don't think it's possible without threads.
The closest you could come would be to intersperse the IO calls with
calls to flock(...,LOCK_EX|LOCK_NB). Sadly, this runs the risk of
"never" acquiring the lock, due to other processes always having it. If
you lock without the NB, then waiters will generally get the lock in
fifo order (well, it's at the whim of the OS, but most OSs try to
schedule unlocking in a way that avoid resource starvation).
Brad Schick
I am new to parrot, but I've had plenty of experience with (non-unix)
I/O systems. Having a totally async core in Parrot is pretty clearly
the right thing. It makes sense for Parrot to expose a clean async
model that hides the ugly details of each platform (like Unix Signals).
Although the implementation may end up hairy on some platforms, Parrot
seems like the correct place to bear this burden.
This makes life simpler for everything built atop Parrot that needs AIO
for files or whatever. And while you can debate how common this need
will be, it seems almost guaranteed that the need will exist and that
people will use it if given a clean system.
But just because Parrot should have AIO and is also forced to support
signals, I don't think it follows that AIO must be built on signals. If
an async core is to be a key aspect of Parrot, it should be built on a
reliable foundation. Personally I don't have enough experience with
Unix signals to make a strong argument here...
-Brad
K Stol
----- Original Message -----
From: "Michal Wallace" <mic...@sabren.com>
To: <perl6-i...@perl.org>
Sent: Wednesday, July 23, 2003 4:48 PM
Subject: approaching python
>
> Hey all,
>
> I've been thinking about the "compiling python to
> parrot" concept. Right now it looks like the
> approach is to start from scratch, but I'm
> wondering if it might make more sense to
> leverage python itself, at least for now?
>
> Python has a compiler module (written in python
> and standard with the distribution) that can
> take a python parse tree and produce python
> byte code. It basically just walks the tree
> and has a method for each python structure.
>
> The actual parser is written in c, but there's
> a drop-in replacement (or at least a partial one)
> written in python described here:
>
> http://codespeak.net/moin/pypy/moin.cgi/BytecodeCompiler
>
> Would it make sense to use this to boostrap the
> python parrot compiler? I was thinking about taking
> a shot next week at replacing compiler.pycodegen.CodeGenerator
> with something that produced IMCC.
Correct me if I'm wrong, but there's no bootstrap problem, just a matter of
reimplementing the code generator.
>
> The thing is, I don't have a lot of experience when
> it comes to compilers, but I do know a whole lot about
> python. :) If this approach makes sense, is there
> someone with IMCC experience who'd be willing to do
> some virtual pair programming with me and spike out
> a prototype?
>
If I understood correctly, the codegenerator is isolated to one module,
right? I don't know Python, and I've a little experience with IMC, but it
seems to me only a new code generator module should be written. So, if for
example the Python code generator (generating python byte code) has a method
"gen_assignment(...)" for generating python byte code doing an assignment,
then the new module method (generating IMC) should do the same, but with IMC
instructions. As noted earlier, I've got no experience in Python whatsoever,
but it seems to me it's just a matter of generating the right IMC
instructions for each construct, so when walking the parse tree, calling
"gen_imc_*" methods.
I'm no compiler guru (basic overview + a little experience + much interest
:-), but I find the material very interesting and would gladly help.
Klaas-Jan
Luke Palmer
> > The thing is, I don't have a lot of experience when it comes to
> > compilers, but I do know a whole lot about python. :) If this
> > approach makes sense, is there someone with IMCC experience who'd
> > be willing to do some virtual pair programming with me and spike
> > out a prototype?
> >
>
> If I understood correctly, the codegenerator is isolated to one
> module, right? I don't know Python, and I've a little experience
> with IMC, but it seems to me only a new code generator module should
> be written. So, if for example the Python code generator (generating
> python byte code) has a method "gen_assignment(...)" for generating
> python byte code doing an assignment, then the new module method
> (generating IMC) should do the same, but with IMC instructions. As
> noted earlier, I've got no experience in Python whatsoever, but it
> seems to me it's just a matter of generating the right IMC
> instructions for each construct, so when walking the parse tree,
> calling "gen_imc_*" methods.
Well... sortof. It's definitely going to take writing a whole new
code generator module; it's not just a matter of getting the right
instructions. Python's interpreter is stack-based, while Parrot's is
register-based, which are two very different kinds of data.
I think it would be good design to have the python binary parse for
us, but it's not likely practical. Python has eval, so unless we want
to link with python, we should probably write our own parser.[1]
I know a little python, and it looks like writing a parser for it
should not be too difficult.
> I'm no compiler guru (basic overview + a little experience + much
> interest :-), but I find the material very interesting and would
> gladly help.
I'm pretty good at compilers, and helping out would certainly teach me
more python. So, I'm up for it.
> Klaas-Jan
Luke
K Stol
Oh sorry, that's actually what I meant, a complete new code generator
module.
My idea was to write the code generator as a tree traversal. So the source
is parsed, and a parse tree is built, after which this tree can be traversed
for code generation (maybe some type checking in between).
> I think it would be good design to have the python binary parse for
> us, but it's not likely practical. Python has eval, so unless we want
> to link with python, we should probably write our own parser.[1]
>
I thought eval should be implemented as compiling during runtime? Parrot has
instructions for that.
Christian Renz
Just use Perl 6 to write it :). Ooops... looks like a chicken-and-egg
problem...
Greetings,
Christian
Michal Wallace
On 24 Jul 2003, Luke Palmer wrote:
> Klass-Jan Stol writes:
> > module, right? I don't know Python, and I've a little experience
> > with IMC, but it seems to me only a new code generator module should
...[snip]
> Well... sortof. It's definitely going to take writing a whole new
> code generator module; it's not just a matter of getting the right
> instructions. Python's interpreter is stack-based, while Parrot's is
> register-based, which are two very different kinds of data.
Right. The implementation would be completely yanked out, but
if we keep the code generator's interface, then that's the only
file we have to touch.
The other idea i had was a little crazy, but might make
life a lot easier: if perl 6 can use pmcs directly,
why not just compile it down to perl and let the perl
compiler handle all the register stuff? [I honestly
have no idea which way would be easier, since I don't
realy know much about IMCC]
Also, there's an older project called rattlesnake that
never really got off the ground. I don't know if any
code was produced, but the point was to make a register
based python vm. Maybe someone from that camp could help
us out.
> I think it would be good design to have the python binary parse for
> us, but it's not likely practical. Python has eval, so unless we
> want to link with python, we should probably write our own
> parser.[1]
But there's already a parser written in python (in the
pypython project), and it makes trees that work with the
code generator. :) So I'm saying, use those tools, and
just run the compiler from python until it's good enough
to compile itself.
That just seems like the quickest way to get python
working. (Of course, then there's all the C-based
python modules, but that's another story)
Michal Wallace
> I'm not familiar with the Python bytecode spec (to be a little more
> accurate, I'm completely clueless about it), but perhaps something
> similar can be done? Also, another thing to consider is that it
> might be easier to translate python bytecode directly to imcc rather
> than to generate imcc straight from the parse tree itself... It's
> just a thought, anyways.
That's Dan's approach, from what I've read on his blog.
As long as we eventually get a python parser running
as parrot bytecode, it doesn't really matter how it
gets there.
It's just that I personally am a lot more comfortable
with the higher level concepts. I'd rather work with
visit_if() than a string of python bytecodes. My
intuition is that you could do a lot better job
with optimization that way too, but I could be
way off there.
Joseph F. Ryan
>Klass-Jan Stol writes:
>
>>>The thing is, I don't have a lot of experience when it comes to
>>>compilers, but I do know a whole lot about python. :) If this
>>>approach makes sense, is there someone with IMCC experience who'd
>>>be willing to do some virtual pair programming with me and spike
>>>out a prototype?
>>>
>>If I understood correctly, the codegenerator is isolated to one
>>module, right? I don't know Python, and I've a little experience
>>with IMC, but it seems to me only a new code generator module should
>>be written. So, if for example the Python code generator (generating
>>python byte code) has a method "gen_assignment(...)" for generating
>>python byte code doing an assignment, then the new module method
>>(generating IMC) should do the same, but with IMC instructions. As
>>noted earlier, I've got no experience in Python whatsoever, but it
>>seems to me it's just a matter of generating the right IMC
>>instructions for each construct, so when walking the parse tree,
>>calling "gen_imc_*" methods.
>
>Well... sortof. It's definitely going to take writing a whole new
>code generator module; it's not just a matter of getting the right
>instructions. Python's interpreter is stack-based, while Parrot's is
>register-based, which are two very different kinds of data.
Not necessarily. I've had a good deal of success translating
jvm->imcc (which is also a stack->register conversion) simply by
simulating the stack during the translation. Since the JVM spec is
pretty specific on what is on the stack both before and after an op,
this was pretty easy to do.
I'm not familiar with the Python bytecode spec (to be a little more
accurate, I'm completely clueless about it), but perhaps something
similar can be done? Also, another thing to consider is that it
might be easier to translate python bytecode directly to imcc rather
than to generate imcc straight from the parse tree itself... It's
just a thought, anyways.
- Joe
--
This message was sent using 3wmail.
Your fast free POP3 mail client at www.3wmail.com
Damien Neil
> First, to get it out of the way, I don't have to convince you of
> anything. You have to convince me. For better or worse I'm
> responsible for the design and its ultimately my decision. If you
> don't want async IO, it's time to make a very, very good case.
I hope that I haven't given the impression that I feel otherwise.
You're the designer, and Parrot is your baby. I'm just expressing
my opinion; you are of course completely free to disagree with me.
Let me restate my position, since I think it's getting lost in the
general confusion:
I'd be happy if Parrot contained no support at all for interrupts,
in particular the traditional interrupt-based delivery of Unix
signals. I think that support for interrupts will come at a cost,
and I'd prefer not to have to pay that cost.
I've expounded at length in an earlier message on why I think
interrupts in application-level code is generally a bad idea. I
won't bother repeating myself here; I don't think I said anything
particularly controversial there.
I'm not arguing against non-blocking IO, event loops, a unified
event queue, or internally using the aio_*() API on Unix. I think
that all of these things are Nifty(tm) and I highly approve of all
of them.
I /am/ arguing against exposing the aio_*() API (or its equivalent)
to code running atop the Parrot VM, on the grounds that it uses
interrupts as a part of the API. I'd rather just have non-blocking
IO calls and a good event queue.
On a somewhat related note, I'm dubious about the performance gains
that code using interrupt-driven IO will see as opposed to code using
event-loop driven IO. I /think/ you're telling me that I'm wrong,
and that interrupt-driven IO does indeed have performance benefits;
it's possible that you're actually telling me that event-loop driven
code with non-blocking IO has performance benefits as compared to
threaded code with blocking IO. If it's the latter, then we are
in violent agreement. :>
- Damien
Benjamin Goldberg
Except that you'd have to reimplement the code generator *in python* to
produce parrot bytecode.
Then run it once (on the python parser and code generator themselves)
using the python executable, *then* you can run them using parrot.
Sounds like bootstrapping to me. :P
K Stol
Sounds like it's easy to solve the problem ;-)
It could be that my mind is somewhere different than the other people
discussing this.
I thought "Python->Parrot" is just a compiler that translates Python code to
ParrotVM (in practice this will generate IMC), and it *doesn't matter* in
what language this compiler is written in. So, if there is already a Python
parser, constructing a parse tree, I'd say, let's reconstruct the code
generator (indeed, only 1 file added/replaced), and if that has to be in
Parrot, so be it.
One more thing. The "compile at run-time" operation, for translating the
eval instruction needs a compiler, which will eventually be just a sub call
(if I understood the docs on that well). Then, it doesn't really matter if
that call is to a 'normal' sub, or if it's throught NCI, right?
Klaas-Jan
K Stol
----- Original Message -----
From: "Michal Wallace" <mic...@sabren.com>
To: "Luke Palmer" <fibo...@babylonia.flatirons.org>
Cc: <k_s...@hotmail.com>; <perl6-i...@perl.org>
Sent: Thursday, July 24, 2003 12:01 PM
Subject: Re: approaching python
>
> On 24 Jul 2003, Luke Palmer wrote:
> > Klass-Jan Stol writes:
> > > module, right? I don't know Python, and I've a little experience
> > > with IMC, but it seems to me only a new code generator module should
>
> ...[snip]
>
> > Well... sortof. It's definitely going to take writing a whole new
> > code generator module; it's not just a matter of getting the right
> > instructions. Python's interpreter is stack-based, while Parrot's is
> > register-based, which are two very different kinds of data.
>
> Right. The implementation would be completely yanked out, but
> if we keep the code generator's interface, then that's the only
> file we have to touch.
>
> The other idea i had was a little crazy, but might make
> life a lot easier: if perl 6 can use pmcs directly,
> why not just compile it down to perl and let the perl
> compiler handle all the register stuff? [I honestly
> have no idea which way would be easier, since I don't
> realy know much about IMCC]
The register stuff, I presume, is register allocation and the like? When
targeting IMCC,
you can use an infinite amount of registers. Just keep a counter in the code
generator,
each time a new register is needed, just increment the counter and add a
"${S|N|I|P}" in front of it (example: $P1). IMCC does register allocation
and spilling.
So this is not really difficult.
It may be that this is not as efficient, because it adds an extra layer of
translation:
Python->Perl6->IMCC->PASM[1]
instead of
Python->IMCC->PASM
[1]. IMCC->PASM integrated in Parrot exec.
Klaas-Jan
Benjamin Goldberg
Nono, the problem isn't that python uses *more* registers than
<whatever>, but rather, that it doesn't use registers at all. Instead,
it uses a stack. So, for example, python's add instruction might get
translated into the following pir or pasm code:
restore P0
restore P1
clone P0, P0
add P0, P1
save P0
Needless to say, this is not efficient, due to how slow parrot's push
and pop operations are.
Hmm... If imcc is smart enough, (or perhaps I should say, when the flow
control is simple/clear enough) it should be able to see when a value is
pushed onto the stack, and later popped off, when there are enough
registers free that we could have stored it in a spare register instead
of on the stack. That is, a sort of register *un*spilling.
> It may be that this is not as efficient, because it adds an extra layer
> of translation:
>
> Python->Perl6->IMCC->PASM[1]
? Why would we translate Python into Perl6 code?
Also, don't you mean IMCC->PBC ? AFAIK, the parrot assembly language is
a mostly vestigal creation whose purpose was to allow us to write code
for parrot while imcc was being developed. Now that imcc exists&works,
we no longer need nor use a seperate assembly language.
> instead of
>
> Python->IMCC->PASM
>
> [1]. IMCC->PASM integrated in Parrot exec.
>
> Klaas-Jan
> >
> > Also, there's an older project called rattlesnake that
> > never really got off the ground. I don't know if any
> > code was produced, but the point was to make a register
> > based python vm. Maybe someone from that camp could help
> > us out.
> >
> > > I think it would be good design to have the python binary parse for
> > > us, but it's not likely practical. Python has eval, so unless we
> > > want to link with python, we should probably write our own
> > > parser.[1]
> >
> > But there's already a parser written in python (in the
> > pypython project), and it makes trees that work with the
> > code generator. :) So I'm saying, use those tools, and
> > just run the compiler from python until it's good enough
> > to compile itself.
> >
> > That just seems like the quickest way to get python
> > working. (Of course, then there's all the C-based
> > python modules, but that's another story)
Well, we're already writing a parrot <-> perl5/XS compatibility later...
if necessary, we could probably (if the python/C interface has
sufficient encapsulation of python objects) write a compatibility layer
for parrot <-> python/C, if there's sufficient demand for it. But such
a thing would have to wait until we get python running on parrot in the
first place :)
Joseph Ryan
>K Stol wrote:
>
>>The register stuff, I presume, is register allocation and the like? When
>>targeting IMCC, you can use an infinite amount of registers. Just keep a
>>counter in the code generator, each time a new register is needed, just
>>increment the counter and add a "${S|N|I|P}" in front of it (example:
>>$P1). IMCC does register allocation and spilling.
>>So this is not really difficult.
>>
>
>Nono, the problem isn't that python uses *more* registers than
><whatever>, but rather, that it doesn't use registers at all. Instead,
>it uses a stack. So, for example, python's add instruction might get
>translated into the following pir or pasm code:
>
> restore P0
> restore P1
> clone P0, P0
> add P0, P1
> save P0
>
>Needless to say, this is not efficient, due to how slow parrot's push
>and pop operations are.
>
Well, thats because you're trying to make a register machine act like a
stack machine. It would be more efficient to translate add as:
$P2 = $P0 + $P1
That is to say, map stack positions to registers by simulating the
stack while walking each op during translation time, rather than
during runtime. So, in this case, the code that translates the add
instruction might look something like:
translate_add_op {
pop variable1 off of simulated stack
pop variable2 off of simulated stack
create new temp pmc
push new temp_pmc onto simulated stack
print temp_pmc's name
print " = "
print variable1's name
print " + "
print variable2's name
}
So, after this code, our simulated stack is depleted by two items
(the operands), and then replenished by one (the result); this
makes it act exactly like the real stack, except that we are
manipulating the registers rather than the values.
>Hmm... If imcc is smart enough, (or perhaps I should say, when the flow
>control is simple/clear enough) it should be able to see when a value is
>pushed onto the stack, and later popped off, when there are enough
>registers free that we could have stored it in a spare register instead
>of on the stack. That is, a sort of register *un*spilling.
>
Doesn't IMCC already do this?
- Joe
Vladimir Lipskiy
> <whatever>, but rather, that it doesn't use registers at all. Instead,
> it uses a stack. So, for example, python's add instruction might get
Nobody said Python used more registers than <whatever>. Michal
just worried if it would be problematic to deal with the register stuff.
That's why he was hesitating whether to translate Python into IMCC
or into Perl6 that could have handled all the register stuff by itself.
And Klaas-Jan was trying to convince him that the register stuff
wasn't such a beast with IMCC.
K Stol
From: "Joseph Ryan" <ryan...@osu.edu>
To: "Benjamin Goldberg" <ben.go...@hotpop.com>
Cc: <perl6-i...@perl.org>
Sent: Sunday, July 27, 2003 7:56 PM
Subject: Re: approaching python
> Benjamin Goldberg wrote:
>
> >K Stol wrote:
> >
> >>The register stuff, I presume, is register allocation and the like? When
> >>targeting IMCC, you can use an infinite amount of registers. Just keep a
> >>counter in the code generator, each time a new register is needed, just
> >>increment the counter and add a "${S|N|I|P}" in front of it (example:
> >>$P1). IMCC does register allocation and spilling.
> >>So this is not really difficult.
> >>
> >
> >Nono, the problem isn't that python uses *more* registers than
> ><whatever>, but rather, that it doesn't use registers at all. Instead,
> >it uses a stack. So, for example, python's add instruction might get
> >translated into the following pir or pasm code:
> >
> > restore P0
> > restore P1
> > clone P0, P0
> > add P0, P1
> > save P0
> >
> >Needless to say, this is not efficient, due to how slow parrot's push
> >and pop operations are.
this is not what I meant. I didn't want the code generator to generate stack
based instructions, but to generate the most efficient instructions to get
something done, which in the case of Parrot, are register based
instructions.
----- Original Message -----
From: "Vladimir Lipskiy" <fors...@kaluga.ru>
To: "perl6-internals" <perl6-i...@perl.org>; "Benjamin Goldberg"
<ben.go...@hotpop.com>
Sent: Monday, July 28, 2003 1:47 AM
Subject: Re: approaching python
Yep!
Ok. let's review this for a moment, because I began to doubt myself.
At this moment, we have the original Python parser, written in Python.
There's also a code generator (CG) which produces python bytecode.
So, when a new CG is written in Python, that produces IMC instead, then we
have a compiler that translates python->IMC, and this compiler is completely
written in Python, right?
The eventual target is: a compiler that translates Python to IMC, written in
IMC (this could be compiled by Parrot to a pbc file, for efficiency)
When we use the Python->IMC compiler (written in Python) to compiler itself,
then we get a Python->IMC compiler written (generated by the Python->IMC
compiler) in IMC.
Maybe it's not possible to have the eval instruction right away, in the new
Python->IMC code generator. As long as this command is not used in the code
for this new code generator, this can just be skipped, and it may be
implemented in IMC by hand, and inserted into the new code generator
(Python->IMC, written in IMC)
Klaas-jan
Luke Palmer
> Maybe it's not possible to have the eval instruction right away, in the new
> Python->IMC code generator. As long as this command is not used in the code
> for this new code generator, this can just be skipped, and it may be
> implemented in IMC by hand, and inserted into the new code generator
> (Python->IMC, written in IMC)
Well, is the Python parser written in Python? If so, then we have a
bootstrapping problem... but since there is already a good bootstrap
program (the python executable), there's no problem at all.
Luke
> Klaas-jan
K Stol
----- Original Message -----
From: "Luke Palmer" <fibo...@babylonia.flatirons.org>
To: <k_s...@hotmail.com>
<ben.go...@hotpop.com>
Sent: Monday, July 28, 2003 4:46 AM
Subject: Re: approaching python
I'm sorry, but I can't see this boostrapping problem. Do I overlook
something?
Could you explain?
It isn't necessary to implement the complete language, is it? As long as the
features needed to write the compiler are implemented, it's no problem (i.e.
halfbootstrapping).
> Luke
>
> > Klaas-jan
>
Luke Palmer
> From: "Luke Palmer" <fibo...@babylonia.flatirons.org>
> To: <k_s...@hotmail.com>
> Cc: <fors...@kaluga.ru>; <perl6-i...@perl.org>;
> <ben.go...@hotpop.com>
> Sent: Monday, July 28, 2003 4:46 AM
> Subject: Re: approaching python
>
>
> > Klass-jan wrote:
> > > Maybe it's not possible to have the eval instruction right away, in the
> new
> > > Python->IMC code generator. As long as this command is not used in the
> code
> > > for this new code generator, this can just be skipped, and it may be
> > > implemented in IMC by hand, and inserted into the new code generator
> > > (Python->IMC, written in IMC)
> >
> > Well, is the Python parser written in Python? If so, then we have a
> > bootstrapping problem... but since there is already a good bootstrap
> > program (the python executable), there's no problem at all.
> >
> I'm sorry, but I can't see this boostrapping problem. Do I overlook
> something?
> Could you explain?
I said "problem" rhetorically. There's no problem, because we have a
bootstrap already.
> It isn't necessary to implement the complete language, is it? As long as the
> features needed to write the compiler are implemented, it's no problem (i.e.
> halfbootstrapping).
Depends on how much of Python is written in Python. The more, the
better in this case. Everything that is written in Python we can
support by just writing a code generator.
There are inevitably big parts of python written in C that aren't
written in Python (OS-interfaces, for instance), and those we need to
rewrite or build a compatibility layer. I see a compatibility layer
as a huge job, so we could just do those parts by hand... for now.
Luke
Michal Wallace
Well, No, and yes. :)
No, "the" python parser is written in c,
but there is "a" supposedly drop-in replacement
for it (that I've not yet tried) written in python as part
of the pypython project:
http://codespeak.net/moin/pypy/moin.cgi/BytecodeCompiler
So you are correct, in theory there is no big problem here.
Incidentally, python legend amk emailed me last friday
and pointed out his work getting a small python script
to compile down to parrot 0.0.2:
http://www.amk.ca/conceit/parrot.html
It uses this approach of walking the python parse tree
(as opposed to translating the python bytecode), so my
plan is to try and update it this week for IMCC. Assuming
that works out, the python compiler should be just a matter
of writing the rest of these 50-something "visit" methods
and then dealing with the C-stuff.
I think it would be nice to have that python API wrapper
so C-module authors could support both versions with the
same API...
Michal Wallace
> Benjamin Goldberg wrote:
>
> >[...] the problem isn't that python uses *more* registers than
> ><whatever>, but rather, that it doesn't use registers at all. Instead,
> >it uses a stack. So, for example, python's add instruction might get
> >translated into the following pir or pasm code:
> >
> > restore P0
> > restore P1
> > clone P0, P0
> > add P0, P1
> > save P0
> >
> >Needless to say, this is not efficient, due to how slow parrot's push
> >and pop operations are.
>
> Well, thats because you're trying to make a register machine act like a
> stack machine. It would be more efficient to translate add as:
>
> $P2 = $P0 + $P1
Having looked into how this actually works in python,
I don't think that we actually have to worry about the
stack issue at all. Python's stack problems seem to be
confined to the bytecode level, so if you're translating
the bytecode directly, you'd need to either simulate the
stack or work around it somehow, but, as far as I can tell,
it should be no problem at all to generate register-aware
code from the parse tree and not worry about the stack
at all.