Should I target Parrot?
Tom Locke
I am currently embarking on a project to create a new language. Right now
I'm in the process of selecting the platform that's going to give me the
best starting point.
My first choice is whether to go for a VM, or a C-Python style
implementation. Right now I'm leaning towards VM.
The language will be dynamically typed, and I understand that Parrot is a
better fit for such languages than the JVM or the CLR.
I'm sure you're all good Parrot advocates in here, so... If I post some
requirements I have to support my language features, would you folks care to
talk me into targetting Parrot. Is this the right group for such a
discussion?
Thanks,
Tom.
Michal Wallace
> I am currently embarking on a project to create a new language. Right now
> I'm in the process of selecting the platform that's going to give me the
> best starting point.
>
> My first choice is whether to go for a VM, or a C-Python style
> implementation. Right now I'm leaning towards VM.
Hey Tom,
I'm not sure what you mean by the difference here. CPython
does have a VM, it's just a stack-based one.
> The language will be dynamically typed, and I understand that Parrot
> is a better fit for such languages than the JVM or the CLR.
>
> I'm sure you're all good Parrot advocates in here, so... If I post
> some requirements I have to support my language features, would you
> folks care to talk me into targetting Parrot. Is this the right
> group for such a discussion?
Do you have a website for the language?
I've been working on a python->parrot compiler and managed
to get quite a bit working. If your language is strongly
O-O, you may need to dive into the parrot guts and submit
some patches if you want to move quickly (as classes and
objects aren't quite ready yet) but for a more functional
or imprerative style language you should be able to get
pretty far already.
If you've got some seriously weird features, who knows? Ask
and see, or try it out. :)
But: I would say, in general, if you target parrot, and you
make your language compatible with other languages, you're
more likely to get people to try your language for a small
chunk of code here and there, as people will be able to
use libraries they're familiar with your syntax. It'll help
people shorten their learning curve, in other words. (But
that's just my prediction)
Sincerely,
Michal J Wallace
Sabren Enterprises, Inc.
-------------------------------------
contact: mic...@sabren.com
hosting: http://www.cornerhost.com/
my site: http://www.withoutane.com/
--------------------------------------
Kenneth Graves
Date: Wed, 20 Aug 2003 11:58:05 -0400
I'm sure you're all good Parrot advocates in here, so... If I post some
requirements I have to support my language features, would you folks care to
talk me into targetting Parrot. Is this the right group for such a
discussion?
Sure. Keep in mind that Parrot is a moving target at the moment.
For any given feature, you might hear:
1) That's supported.
2) That's in the works.
3) Perl6 is going to do something similar, so Parrot had better support
it eventually.
or, very unlikely,
4) Parrot won't do that.
--kag
Tom Locke
> > My first choice is whether to go for a VM, or a C-Python style
> > implementation. Right now I'm leaning towards VM.
>
> I'm not sure what you mean by the difference here. CPython
> does have a VM, it's just a stack-based one.
OK so I'm learning already :)
I thought CPython was more of a conventional interpreter, and the .pyc files
were pretty much 'pre-parsed' equivalents of the original source file. So
CPython has more of a traditional VM structure, with a byte-code intruction
set and all?
What do you know about other scripting languages, e.g. Rhino JavaScript. Is
everyone choosing a VM approach these days? My background has been in
compiled, statically typed languages so I'm a bit behind the curve in the
scripting world.
I'm coming to the conclusion that building on top of an existing VM is the
only smart way to go in order to leverage a ton of great work done already
by others.
> Do you have a website for the language?
Not at this point.
> But: I would say, in general, if you target parrot, and you
> make your language compatible with other languages, you're
> more likely to get people to try your language for a small
> chunk of code here and there, as people will be able to
> use libraries they're familiar with your syntax.
This is a strong reason for my interest in Parrot. I think Microsoft are
definately going the right direction with the CLR - a common managed
environment but many languages, with great facilities to share libraries and
for cross-language iterop.
The trouble is that languages differ in deep ways that make the reality fall
short of the utopia. That's why Parrot makes sense to me - a kind of CLR for
highly dynamic languages. And if one day it can play ball nicely with both
the CLR and the JVM, so much the better (and one less thing for language
creators to wrestle with :)
Thanks for your input, I'm going to post separately about some requirements
I have.
Tom.
Tom Locke
My big requirement is for lightweight microthreads (hopefully *very*
lightweight - I'm considering one scheduler that can handle *millions* of
threads on a single machine). Oh and I will need them to be serializable. I
can live with co-operative scheduling, perhaps running 'regular' threads
over the top do provide a layer of preemptive scheduling.
I just read some stuff about Rhino JavaScript now having serializable
continuations - that's just the ticket, although I haven't looked into the
performance yet and I don't yet know if Rhino can provide a platform for a
new language.
I understand Parrot uses continuations (serializable), and also supports
(will support?) native threads.
What about multi-methods? Can/Will Parrot do dynamic dispatch based on the
types of more than one parameter? This may not turn out to be a must-have
for me.
I think I'll leave it at that for now - as you can see I'm in the early
stages of this one.
The only other thing is licensing. Did I see that Parrot is GPL? In these
days of dynamic linking and late binding, it's hard to be sure about the
boundaries of the GPL viral effect. What are the implications for the
license of my language?
Oh and I have to ask your opinions - will Guido get a pie in the face next
year?
Thanks
Tom.
Dan Sugalski
> Hi Michal
>
> > > My first choice is whether to go for a VM, or a C-Python style
> > > implementation. Right now I'm leaning towards VM.
> >
> > I'm not sure what you mean by the difference here. CPython
> > does have a VM, it's just a stack-based one.
>
> OK so I'm learning already :)
>
> I thought CPython was more of a conventional interpreter, and the .pyc files
> were pretty much 'pre-parsed' equivalents of the original source file. So
> CPython has more of a traditional VM structure, with a byte-code intruction
> set and all?
Yep.
> What do you know about other scripting languages, e.g. Rhino JavaScript. Is
> everyone choosing a VM approach these days? My background has been in
> compiled, statically typed languages so I'm a bit behind the curve in the
> scripting world.
It's more an issue of distance to be jumped from the compiler to the
target. With a relatively low level language such as C, the conceptual
distance to the hardware's pretty small. With higher-level languages the
distance to the hardware's much larger and the difficulty in bridging that
gap doesn't seem to grow linearly with the size of the gap, so it's a lot
easier to build a middle layer in there, which is what VMs are, after all.
A VM also makes portability easier, which is nice.
Dan
Dan Sugalski
> OK, here's what I'm hoping Parrot can provide for the language I'm building.
>
> My big requirement is for lightweight microthreads (hopefully *very*
> lightweight - I'm considering one scheduler that can handle *millions* of
> threads on a single machine). Oh and I will need them to be serializable. I
> can live with co-operative scheduling, perhaps running 'regular' threads
> over the top do provide a layer of preemptive scheduling.
Parrot's going to live on top of the system thread library, so you may run
into some issues that way. Not all systems can handle lots of
threads--many of them (including Linux) have very low limits relative to
what most microthread-based systems expect.
> I understand Parrot uses continuations (serializable), and also supports
> (will support?) native threads.
I'm still not 100% sure about serializable continuations, though I expect
so. (They may be rather large when serialized, though) Definitely native
threads.
> What about multi-methods? Can/Will Parrot do dynamic dispatch based on the
> types of more than one parameter? This may not turn out to be a must-have
> for me.
Yep. It's a requirement for a number of languages, including perl 6.
> The only other thing is licensing. Did I see that Parrot is GPL? In these
> days of dynamic linking and late binding, it's hard to be sure about the
> boundaries of the GPL viral effect. What are the implications for the
> license of my language?
Combination GPL and AL, your choice. Parrot's license isn't supposed to
leak out, but I still need to clarify that in the licensing file.
Dan
Dan Sugalski
> > > My big requirement is for lightweight microthreads...
> >
> > Parrot's going to live on top of the system thread library, so you may run
> > into some issues that way. Not all systems can handle lots of
> > threads--many of them (including Linux) have very low limits relative to
> > what most microthread-based systems expect.
>
> But Parrot has continuations. Doesn't this gives me (cooperative)
> microthreads? (with a little work on my part).
Sure. And with a little work you could have one parrot interpreter per
microthread, have a zillion (well, memory permitting as the interpreters
are a bit porky) interpreters and switch between them yourself.
You could also play interesting games with the event system and a timer,
now that I think about it.
> > I'm still not 100% sure about serializable continuations, though I expect
> > so.
>
> Weather or not I go for Parrot, I would very much encourage you in this
> direction. The world needs:
>
> * Long lived processes (long could mean years!)
> * ACID transactional processes
> * Mobile processes
The world could also use a pony. And a lollypop. :) Seriously, while I'd
want to, whether it happens depends on a number of engineering and
personnel factors--while nice, it's not as important as other things.
> > (They may be rather large when serialized, though)
>
> With a deep stack, this is inevitable right? Or are you saying that Parrot
> would be worse than other systems in this respect?
Probably worse, since to serialize a continuation properly means
serializing everything the continuation would need to reconstitute itself,
including any and all code it has a handle on, lexical pads, used global
variables, and so forth. I'd bet about 90% of what an interpreter had
loaded would need serializing in the conservative case.
Dan
Nicholas Clark
> Oh and I have to ask your opinions - will Guido get a pie in the face next
> year?
No, unless Dan gets good or lucky with his aim. 10 paces is a long way.
However, given the sterling work of Daniel and Leo on the JIT, I expect
Dan will be the one doing the throwing. And even if the Python people
cry foul on the JIT I suspect that the computed-goto-predereference core
will scream faster than anything they currently have.
(Mmmm. I wonder if they're implementing their own CGP core)
Nicholas Clark
Nathanael Kuipers
>the computed-goto-predereference core
Zarniwoop! Can you explain what this means?
Thanks,
Nathanael Kuipers
Leopold Toetsch
> On Thu, Aug 21, 2003 at 11:03:58AM -0400, Tom Locke wrote:
>> Oh and I have to ask your opinions - will Guido get a pie in the face next
>> year?
> No, unless Dan gets good or lucky with his aim. 10 paces is a long way.
He will have to practice his skills.
> (Mmmm. I wonder if they're implementing their own CGP core)
With stack pointer relative addressing ...
CGP or JIT are at the same speed for PMC based code. It all depends on
the code that HLL emits and what the parrot optimizer can do for it.
> Nicholas Clark
leo
Tom Locke
>
> Parrot's going to live on top of the system thread library, so you may run
> into some issues that way. Not all systems can handle lots of
> threads--many of them (including Linux) have very low limits relative to
> what most microthread-based systems expect.
But Parrot has continuations. Doesn't this gives me (cooperative)
microthreads? (with a little work on my part).
> I'm still not 100% sure about serializable continuations, though I expect
> so.
Weather or not I go for Parrot, I would very much encourage you in this
direction. The world needs:
* Long lived processes (long could mean years!)
* ACID transactional processes
* Mobile processes
> (They may be rather large when serialized, though)
With a deep stack, this is inevitable right? Or are you saying that Parrot
would be worse than other systems in this respect?
Thanks for you advice.
Tom.
Tom Locke
> > But Parrot has continuations. Doesn't this gives me (cooperative)
> > microthreads? (with a little work on my part).
>
So these would be real cheap right? Time and space overheads similar to
regular procedure calls?
> The world could also use a pony. And a lollypop. :)
Um, I think this is off topic - I better check the charter :)
> ...whether it happens depends on a number of engineering and
> personnel factors--while nice, it's not as important as other things.
For me this is must-have, so I may be in the mood to contribute... Do you
have any rough handle on the size of the task? Would the learning curve be
painful for a newcomer?
To return to the general thread of this, er, thread, which was not
specifically about, um, threads... Let's say I have decided to build on top
of a VM, is Parrot the right VM?
Given that I'm not deeply into all this as of right now, I'm thinking
probably yes, simply because Parrot seems to be the only VM project that is:
1) Built for dynamic scripting languages from the ground up.
and
2) Built to support a range of such languages (and interop between them)
Any VM that cannot claim (2) seems scary as a target for a new language.
Agree? True there are many languages for the JVM, but if you look at Java
bytecode (no function pointers!), it's pretty clear that performace is
directly proportional to conformance to Java's object model.
Fire away!
Tom.
Benjamin Goldberg
Tom Locke wrote:
>
> OK, here's what I'm hoping Parrot can provide for the language I'm building.
>
> My big requirement is for lightweight microthreads (hopefully *very*
> lightweight - I'm considering one scheduler that can handle *millions*
> of threads on a single machine).
Hmm, I can envision a microthread implementation wherein each
microthread is nothing more than a Continuation PMC.
Add an interpreter->microthreads PMC* pointer. (This could be a global
stored in interpreter->perl_stash->stash_hash, but that wouldn't be as
fast). Then, add the following ops:
inline op init_microthreads() {
if( !interpreter->microthreads )
interpreter->microthreads =
new_pmc(interpreter, enum_class_PerlArray);
goto NEXT();
}
=item B<mthread_create>(in PMC)
Create a new microthread, which will start with the Continuation in $1.
=cut
inline op mthread_create(in PMC) {
VTABLE_push(interpreter, interpreter->microthreads, $1);
goto NEXT();
}
=item B<mthread_create>(inconst INT)
Create a new Continuation, with the address set to $1, and
=cut
inline op mthread_create(inconst INT) {
opcode_t *dest = PTR2OPCODE_T(CUR_OPCODE + $1));
PMC * p = new_ret_continuation_pmc(interpreter, dest)
VTABLE_push(interpreter, interpreter->microthreads, p);
goto NEXT();
}
=item B<mthread_yieldcc>()
Create a new Continuation (which resumes after this op), schedule
it to activate later, and then switch to the next microthread.
=cut
inline op mthread_yieldcc() {
opcode_t *dest = expr NEXT();
PMC * p = new_ret_continuation_pmc(interpreter, dest);
VTABLE_push(interpreter, interpreter->microthreads, $1);
p = VTABLE_shift(interpreter, interpreter->microthreads);
dest = (opcode_t *)VTABLE_invoke(interpreter, p, dest);
goto ADDRESS(dest);
}
=item B<mthread_yield>(in PMC)
Schedule the Continuation in $1, then switch to the next microthread.
=cut
inline op mthread_yield(in PMC) {
opcode_t *dest = expr NEXT();
PMC * p;
VTABLE_push(interpreter, interpreter->microthreads, $1);
p = VTABLE_shift(interpreter, interpreter->microthreads);
dest = (opcode_t *)VTABLE_invoke(interpreter, p, dest);
goto ADDRESS(dest);
}
=item B<mthread_yield>(inconst INT)
Create a new Continuation, which resumes at address $1, then switch
to the next microthread.
=cut
inline op mthread_yield(inconst INT) {
opcode_t *dest = PTR2OPCODE_T(CUR_OPCODE + $1);
PMC * p = new_ret_continuation_pmc(interpreter, dest);
VTABLE_push(interpreter, interpreter->microthreads, p);
p = VTABLE_shift(interpreter, interpreter->microthreads);
dest = (opcode_t *)VTABLE_invoke(interpreter, p, dest);
goto ADDRESS(dest);
}
inline op mthread_fork() {
PMC * p = new_ret_continuation_pmc(interpreter, expr NEXT());
VTABLE_push(interpreter, interpreter->microthreads, p);
goto NEXT()
}
> Oh and I will need them to be serializable.
I hope you aren't planning on serializing just a single isolated
microthread... that wouldn't work well with what I've got in mind due to
how much stuff comes along when you serialize a continuation -- you'd
get almost the whole interpreter serialized.
If you want, instead, to serialize interpreter->microthreads, however...
well, you'd *still* get almost the whole interpreter serialized, but
you're getting more bang for your buck :)
> I can live with co-operative scheduling, perhaps running
> 'regular' threads over the top do provide a layer of preemptive
> scheduling.
Remember, each thread has it's own interpreter. This means that one
would have to share ->microthreads amongst multiple interpreters.
Besides the very minor nuisance of locking for ->push and ->shift,
there's the very big headach of figuring out what happens when you
invoke a continuation created in one interpreter in another interpreter.
An easier solution might be to create a timer event which, when handled,
does a yieldcc.
[PS: All code in this message is UNTESTED. If you'd *like* to test, be
my guest! :)]
--
$a=24;split//,240513;s/\B/ => /for@@=qw(ac ab bc ba cb ca
);{push(@b,$a),($a-=6)^=1 for 2..$a/6x--$|;print "$@[$a%6
]\n";((6<=($a-=6))?$a+=$_[$a%6]-$a%6:($a=pop @b))&&redo;}
Michal Wallace
(not sure who you're quoting here... dan I think)
> > > But Parrot has continuations. Doesn't this gives me (cooperative)
> > > microthreads? (with a little work on my part).
> >
> > Sure...
>
> So these would be real cheap right? Time and space overheads similar to
> regular procedure calls?
Yes. This isn't working 100% (at least I couldn't get it working
without tweaking the opcodes a bit) but you can see an example
of some microthreads at http://pirate.tangentcode.com/ ) ...
Microthreads is kind of a loose word for it because it's just
a list of generators that increment at each tick.
The bytecode I'm generating there is really really bad, so it
runs pretty slow, but it definitely works. You might try playing
around with that.
Michal Wallace
> I hope you aren't planning on serializing just a single isolated
> microthread... that wouldn't work well with what I've got in mind due to
> how much stuff comes along when you serialize a continuation -- you'd
> get almost the whole interpreter serialized.
>
> If you want, instead, to serialize interpreter->microthreads, however...
> well, you'd *still* get almost the whole interpreter serialized, but
> you're getting more bang for your buck :)
Well how else are we going to implement squeak? :)
Leopold Toetsch
> inline op mthread_create(inconst INT) {
> opcode_t *dest = PTR2OPCODE_T(CUR_OPCODE + $1));
> PMC * p = new_ret_continuation_pmc(interpreter, dest)
Please note that the ret_continuation is intended for returning only, not
for executing code on. (it just has pointers to the original interpreter
context that get swapped in on invoke() - no COW copies like the normal
Continuation class)
leo
Benjamin Goldberg
Oops. I was writing this mostly by looking at core.ops, and I saw this
and thought that it was just a shortcut for creating a Continuation (one
line instead of two). Foolish me.
Benjamin Goldberg
Michal Wallace wrote:
>
> On Thu, 21 Aug 2003, Benjamin Goldberg wrote:
>
> > I hope you aren't planning on serializing just a single isolated
> > microthread... that wouldn't work well with what I've got in mind due
> > to how much stuff comes along when you serialize a continuation --
> > you'd get almost the whole interpreter serialized.
> >
> > If you want, instead, to serialize interpreter->microthreads,
> > however...
> > well, you'd *still* get almost the whole interpreter serialized, but
> > you're getting more bang for your buck :)
>
> Well how else are we going to implement squeak? :)
Squeak uses synchronous communication channels, right? Writing blocks
until there's a reader, and reading blocks until there's a writer.
I just thought of a way of implementing such things for microthreads :)
Something like the following:
enum {
enum_state_none = 0,
enum_state_ready = 1,
enum_state_readers = 2,
enum_state_writers = 4,
};
pmclass SynchronousChannel {
void init() {
SELF->cache.int_val = enum_state_none;
PMC_data(SELF) = pmc_new(INTERP, enum_class_PerlArray);
PObj_custom_mark_SET(SELF);
}
void mark() {
pobject_lives(INTERP, PMC_data(SELF));
}
void push_pmc(PMC *the_val) {
if( SELF->cache.int_val & enum_state_ready )
internal_exception(???, "Illegal state\n");
SELF->cache.int_val |= enum_state_ready
VTABLE_push(INTERP, PMC_data(SELF), the_val);
}
PMC * shift_pmc() {
if( !(SELF->cache_int_val & enum_state_ready) )
internal_exception(???, "Illegal_state");
SELF->cache_int_val &= ~enum_state_ready
return VTABLE_pop(INTERP, PMC_data(SELF));
}
void* invoke(void* next) {
PMC * data = (PMC*)PMC_data(SELF);
PMC * cont;
switch( SELF->cache.int_val ) {
case enum_state_ready:
case enum_state_ready|enum_state_writers:
case enum_state_ready|enum_state_readers:
case enum_state_none:
case enum_state_readers:
cont = pmc_new(INTERP, enum_class_Continuation);
VTABLE_set_integer_native(INTERP, cont, (INTVAL)next);
break;
}
switch( SELF->cache.int_val ) {
/* ready means we're trying to write data */
case enum_state_ready:
case enum_state_ready|enum_state_writers:
/* if there are no readers, suspend ourself */
SELF->cache.int_val = enum_state_writers;
VTABLE_push_pmc(INTERP, data, cont);
break;
case enum_state_ready|enum_state_readers:
/* otherwise, stay awake, and wake up */
/* (and switch to) a reader. */
VTABLE_push_pmc(INTERP, interpreter->microthreads, cont);
cont = VTABLE_shift(INTERP, data);
if( VTABLE_get_integer(INTERP, data) == 1 )
/* Remove _readers flag */
SELF->cache.int_val = enum_state_ready;
return VTABLE_invoke(INTERP, cont);
/* not-ready means we're trying to read data */
case enum_state_writers:
/* if data has been written, we stay awake, */
/* and we wake up the writer who wrote. */
SELF->cache.int_val = enum_state_ready;
VTABLE_push_pmc(INTERP, interpreter->microthreads,
VTABLE_shift(INTERP, data));
if( VTABLE_get_integer(INTERP, data) > 1 )
SELF->cache.int_val |= enum_state_writers;
return next;
case enum_state_none:
SELF->cache.int_val = enum_state_readers;
/* fallthrough */
case enum_state_readers:
/* If no data has been written, we suspend ourself */
VTABLE_push_pmc(INTERP, data, cont);
break;
case enum_state_readers|enum_state_writers:
case enum_state_readers|enum_state_writers|enum_state_ready:
internal_exception(???, "Illegal state");
break;
default:
internal_exception(???, "Illegaller state");
break;
}
if(!VTABLE_get_integer(INTERP, interpreter->microthreads))
internal_exception(???, "Deadlock!");
cont = VTABLE_shift(INTERP, interpreter->microthreads);
return VTABLE_invoke(INTERP, cont, next);
}
}
Writing to a channel is done with:
push $Pchannel, $Pdata
invoke $Pchannel
Reading from a channel is done with:
invoke $Pchannel
shift $Pchannel, $Pdata
We can detect when the whole system is deadlocked, since ->microthreads
becomes empty.
A partial deadlock, though, is undetectable, since there's no way to
know whether or not one of the 'live' threads might happen to write to
one of the channels that the deadlocked threads are reading from, or
might happen to read from a channel that a deadlocked thread is writing
to.
I believe that it would require solving the halting problem to be able
to detect that a partial deadlock will remain locked forever.
If full deadlock does occur, then we can probably find out what went
wrong, by looking through all the syncchannel objects, and inducing the
all of the Continuation pmcs they contain to barf up stacktraces.
Michal Wallace
> Michal Wallace wrote:
> >
> > On Thu, 21 Aug 2003, Benjamin Goldberg wrote:
> >
> > > If you want, instead, to serialize interpreter->microthreads,
> > > however... well, you'd *still* get almost the whole interpreter
> > > serialized, but you're getting more bang for your buck :)
> >
> > Well how else are we going to implement squeak? :)
>
> Squeak uses synchronous communication channels, right? Writing blocks
> until there's a reader, and reading blocks until there's a writer.
Beats me. I was kind of being silly. But squeak does save the state of
the entire interpreter in an image file. :)