A proposition for streamlining Perl 6 development
Yuval Kogman
that Perl 6 (pugs, parrot, everything) is losing too much momentum
lately. I think we need to seriously rethink some of the
implementation plan.
The required points of emphasis which I think are slipping out of
our fingers are: code reuse, KISS (on the implementation level).
On the positive side I would like to say that despite my criciticism
I think the Perl 6 community has accomplished a great deal in the
last 5 years, and an especially great deal in the last year (thanks
to pugs serving as a catalyst). Good work everyone!
In a nutshell:
Split the big task that is "Implementing Perl 6" into smaller
chunks detailed herein.
Why is this a problem::
Perl 6 is a very very very rich language. It's got many overlapping
features (a bajillion loops, operators, yadda), many special cases
(Roles, Classes, Traits - they are specializations of a similar
thing), a HUGE library (access to most of the standard C library,
extended math routines, routines for the builtin types, routines for
higher level IO than the standard C library, etc etc etc).
How can it be fixed:
Split Perl 6 into layers stack, instead of mashing it all up.
Perl 6 Core - a simple, concise core language in the spirit of
Scala, Haskell, and all the "cool" languages today. Let's face
it - they have been the most innovative languages of the last 20
years or so in terms of quality, robustness, performance,
extensibility, conciseness. If we copy this we get a solid
foundation. This backend is optionally type checked, contains a
very bare bones object oriented system (prototype + multimethod
seems to me like the most generic approach), etc.
Perl 6 Extended - Using good domain specific language support
(extensible grammer, string macros and core language AST macros)
we implement all the syntax that makes Perl 6 an accessible
language on top of the Perl 6 core. This contains the grammar
for Perl 6 as defined by the synopses.
Perl 6 Prelude - the builtin routines for handling internal data
types, etc. These must be completely portable, and are part of
the internal spec.
Perl 6 Standard Library - contains chunks of stdc, POSIX/Win32
API, generalizations of IO, etc etc. This is mostly portable,
but e.g. fork() will be missing on platform X, that will be
missing on platform Y, etc.
Perl 6 VM support - the prelude compilation ideas I tossed
around several months ago relate to this - *ANY* high level
function can get it's own opcode or low level replacement
function if it's provided. This approach is just a simplified
generalization of the opcode/internal routine pattern, in a way
that can be exploited for performance, rapid development of
backends, testing, having a reference implementation of the
prelude and library, etc. (I will reply to this post with
refernces from the past).
Perl 6 Parrot support - implementing Perl 6 Core, then Perl 6
Extended, then perl 6 Prelude (mostly for performance reasons)
in a way that canonical with the VM support.
-- on the compiler front we can also componentize --
Parser - can parse Perl core, and have it's grammar updated from
callbacks.
Compiler - compiles Perl core to PIL
Interpreter - interprets PIL - slow, but complete
Linker - loads PIL interface files / modules and grammar
extension modules, assists the compiler in disamgiuating, allows
refactoring the parser extensions, and provides necessary info
for the type checker.
Typechecker
These should be implemented in Perl 6 eventually, but not at first.
Pugs is a good start, but requires lots of separation and decoupling
for this to be realized.
Perl 6 Core macros are just functions that accept ASTs as input, and
are invoked at compile time. They are interpreter by the PIL
interpreter if all else fails.
Using these tools Perl 6 Extended can be bootstrapped. At the next
phase we need:
Generic VM Emitter - a "Base class" for implementing VM emitters
that assist in matching functions to opcodes or VM specific
routines.
Parrot core - porting the PIL interpreter, porting bootstrap
code to allow the linker to link compiled modules, bridging
between PIL and PIR up to a point where PIL macros can be
written in PIR, and the compiler can use the PIR backend to run
compile-time PIL.
Note that whenever I say "Parrot" in my opinion you can also shove
"javascript", "llvm", "neko", ".net", "jvm" and whatever else in
there - I think they are equally important (if not more - after all
they are live, production ready targets).
This plan with pugs in mind:
If Audrey is willing, I think a correct new direction for pugs is to
try and separate the parts even more - the prelude is a mess right
now, many of it's part are duplicated across the backends, the
standard library that is mashed into the prelude, and into pugs core
itself, etc.
After this step I think the multiple backend approach should be
refactored heavily to share as much code as possible, in accordance
with the plans I posted (confusingly termed Circular Prelude stuff
back then, see also VM support listed above).
The biggest missing component of the compiler is: self hosting
grammar, "real" macro support all the way down, a linker (the
groundwork for this has improved with the new lexical importing code
lately), and type checking.
The Perl 5 compatibility layer:
an additional deprecated library that emulates the Perl 5
standard library which wasn't absorbed into Perl 6
the perl 5 to perl 6 converter
Integration with Ponie to enhance code reuse
Additional criticism:
Right now the biggest problem in Perl 6 land is project management.
We, as a community, are facing a task which is sliced into chunks
that are just way too big in my opinion. People burn out. People
have to do things more related to day jobs (that's why my pugs
hacking is on hold, for example), people think things half way, etc.
If we have a layered approach we can concentrate on providing
something that is more balanced
... Phew.
Ciao!
--
() Yuval Kogman <nothi...@woobling.org> 0xEBD27418 perl hacker &
/\ kung foo master: /me spreads pj3Ar using 0wnage: neeyah!!!!!!!!!!!
Yuval Kogman
being taken by pugs, now that PIR targetting is being worked out
(finally) - i just think it needs to be more explicit and in tune
with the @Larry.
Also, the way pugs is refactoring implies nothing on refactoring and
layering Perl 6's design, which I think is also important.
--
() Yuval Kogman <nothi...@woobling.org> 0xEBD27418 perl hacker &
/\ kung foo master: /me does not drink tibetian laxative tea: neeyah!
Chromatic
> Right now the biggest problem in Perl 6 land is project management.
I disagree, but even if it were true, I don't think the solution is to add
more project management and design to partition the process into even more
subprojects of nebulous definition and dubious benefit.
If you *want* Perl 6/Scheme running on Spidermonkey, that's cool. I just
don't see an army of volunteers magically appearing to make it work, not in
the least because it's Yet Another Rewrite From Scratch.
-- c
Yuval Kogman
> On Tuesday 07 February 2006 13:28, Yuval Kogman wrote:
>
> > Right now the biggest problem in Perl 6 land is project management.
>
> I disagree, but even if it were true, I don't think the solution is to add
> more project management and design to partition the process into even more
> subprojects of nebulous definition and dubious benefit.
Separation to subprojects was not what I meant.
I think the best way to describe it is to take concepts which I
think are correct for writing clean code:
* separation of concerns
* layering
* prefer generic components to specific ones
and to try and apply them to Perl 6's design, which is a bit too big
for any implementation effort to complete as one whole unit.
If we have more steps and clearer milestones for whatever is between
parrot and the syntax/feature level design implementation will be
easier. One way to start creating these milestones is to try to
refactor the design of the language.
> If you *want* Perl 6/Scheme running on Spidermonkey, that's cool. I just
> don't see an army of volunteers magically appearing to make it work, not in
> the least because it's Yet Another Rewrite From Scratch.
That was entirely not my point =(
De-facto we have people running PIL on javascript. It works more
than parrot does. The design of Perl 6 itself should be agnostic to
where people are developing backends IRL.
--
() Yuval Kogman <nothi...@woobling.org> 0xEBD27418 perl hacker &
/\ kung foo master: /me wields bonsai kittens: neeyah!!!!!!!!!!!!!!!!
Allison Randal
> Apologies if this is insulting to anyone, but personally I think
> that Perl 6 (pugs, parrot, everything) is losing too much momentum
> lately. I think we need to seriously rethink some of the
> implementation plan.
I understand your frustration. I even sympathize, as I had to work
through this same frustration a few years ago. But, micromanagement
is not the answer to lost momentum. It actually makes things worse,
as people throw their effort into defining the problem more and more
clearly, instead of throwing their effort into producing shippable code.
If it makes you feel any better, pretty much all projects suffer a
loss of momentum after the first year.
Parrot, on the other hand, has noticeably gained momentum the past 6
months or so. AFAICT, this is largely due to the fact that we're
close enough to finished that we can see the light at the end of the
tunnel, and because Pugs reminded us to hold on to our sense of fun.
Blessings,
Allison
Chromatic
> If we have more steps and clearer milestones for whatever is between
> parrot and the syntax/feature level design implementation will be
> easier.
Parrot has had such milestones for well over a year.
> De-facto we have people running PIL on javascript.
> It works more than parrot does.
No, it works *differently* from Parrot, just as an LR parser works differently
from an LR parser.
Don't make the mistake of thinking "Wow, it took Parrot X months to get a
working PGE, while the Pugs version only took Y weeks", especially because
the Pugs version had the benefit of looking at *already designed, debugged,
and tested* Parrot code.
> The design of Perl 6 itself should be agnostic to where people are
> developing backends IRL.
That's a nice goal in the sense of diversity, but I remain unconvinced of its
utility in speeding up the implementation. Every abstraction comes at a
price. The recent velocity of Pugs toward the goal of running on N multiple
backends rather than one backend seems to argue that education is still
cheaper than ignorance.
-- c
PS - Yes, that *is* a Greek-English pun. Language interoperability is a good
thing.
Stevan Little
> On Feb 7, 2006, at 13:28, Yuval Kogman wrote:
>
> > Apologies if this is insulting to anyone, but personally I think
> > that Perl 6 (pugs, parrot, everything) is losing too much momentum
> > lately. I think we need to seriously rethink some of the
> > implementation plan.
>
> I understand your frustration. I even sympathize, as I had to work
> through this same frustration a few years ago. But, micromanagement
> is not the answer to lost momentum.
I really don't think that Yuval is talking about micromangement. He is
talking about refactoring. I think we can all agree that:
- Small methods are good
- Monolithic God objects are bad
Decomposing the problem into smaller and smaller problems until the
problems become manageable for a small team of volunteers to work on
and understand.
> It actually makes things worse,
> as people throw their effort into defining the problem more and more
> clearly, instead of throwing their effort into producing shippable code.
I am not sure if that is Yuval's point either, in fact I think his
point is that without defining the problem a little clearer it will be
very difficult to actually produce shippable code.
> If it makes you feel any better, pretty much all projects suffer a
> loss of momentum after the first year.
Well I dont know about Yuval, but that depresses me somewhat :(
> Parrot, on the other hand, has noticeably gained momentum the past 6
> months or so. AFAICT, this is largely due to the fact that we're
> close enough to finished that we can see the light at the end of the
> tunnel, and because Pugs reminded us to hold on to our sense of fun.
Now I am not as involved in Parrot as I am in Pugs so I might be way
off base here, but from my point of view Parrot still has a long way
to go before it runs Perl 6 code. Part of that is because the bridge
between PIR/PMCs and Perl 6 just does not exist yet (either in code,
or even conceptually). Having PGE parse Perl 6 code only gives us an
AST, it does not give us running code. And even if we have a nicely
massaged AST, running Perl 6 is not a matter of just walking the tree
and evaluating it like it is in Perl 5 (of course, I am simplifying
quite a bit here). We found (a few months ago) in Pugs that this model
just isn't robust enough, and Perl 6 is going to need a more
sophisticated "runtime" environment to support many of it's features.
This "runtime" (or as we have been calling it in Pugs the "Object
Space") will need to exist on top of Parrot too since it is far to
Perl 6 specific to be implemented into the Parrot core.
This is the kind of stuff that Yuval is talking about. The missing
bits that need to exist in the nether-region between perl6-language
and perl6-internals.
We are building from the bottom-up (Parrot) and the top-down (Perl 6 -
the language) and it seems (at least to many of us on the Pugs
project) that there is a big hole somewhere in the middle.
Now, I am perfectly willing to admit that I am totally wrong and eat
every single one of my words if you can show me the missing conceptual
bridge that I am talking about. And please, no hand-waving as that
does not produce shippable code.
Respectfully,
Stevan
David K Storrs
On Feb 7, 2006, at 5:33 PM, Allison Randal wrote:
> Parrot, on the other hand, has noticeably gained momentum the past
> 6 months or so. AFAICT, this is largely due to the fact that we're
> close enough to finished that we can see the light at the end of
> the tunnel, and because Pugs reminded us to hold on to our sense of
> fun.
First off, I'm just a lurker on the lists and I don't spend tuits
hacking Perl6, Parrot, or pugs. Also, I'm definitely behind on the
SOTA as far as P6 goes so, hopefully, nothing that I say below is
outright wrong. If it is, apologies.
I know that, over the last year or so, I've been feeling fairly
exhausted with Perl6--as all large projects do, it takes a long time
and, while it's going on, it's hard to see the progress that is being
made. I follow the lists, but I haven't done any Parrot or pugs
hacking, so I didn't really have a practical sense of where things
stood. It seemed like a never ending treadmill, and I've been
reading the lists in a more and more casual way as tuits and energy
flag.
Well, after this exchange I decided to check out the state of things,
and I was very pleasantly surprised. IIRC, $Larry said that he would
define the language in terms of the chapters from the Camel book,
with the Synopses being the definitive version of the AES triad for a
particular chapter. So, here are the Synopses that are written and
in the repository at https://svn.perl.org/perl6/doc/trunk/design/syn/
# S01.pod
# S02.pod
# S03.pod
# S04.pod
# S05.pod
# S06.pod
# S07.pod Not actually here because formats have been removed, but
I'm including it in the "complete" section.
# S09.pod
# S10.pod
# S11.pod
# S12.pod
# S13.pod
# S17.pod Not complete, just lists topics to cover
# S29.pod Not complete, just a pointer to
So, if @Larry precisely followed the Camel, here's what's left
(sorted by my opinion of its likelihood of being written):
Synopsis
Topic
Will it be written? (Just IMO)
------------ -------
-------------------------------------
S14.pod Tied variables ?
S25.pod Portable
Perl Maybe--probably
just a tweak of P5 version
S19.pod The
CLI
Probably not
S20.pod The Debugger
Probably not
S22.pod
CPAN
Probably not
S26.pod
POD
Probably not
S24.pod Common Practices No
S27.pod Perl
Culture No
S08.pod
References Yes
S15.pod
Unicode Yes
S16.pod
IPC
Yes
S18.pod
Compiling Yes
S21.pod Internals and Externals Yes
S23.pod
Security Yes
S28.pod Reference; Special Names Yes
S30.pod Reference; The Standard Perl Library Yes
S31.pod Reference; Pragmatic Modules Yes
S32.pod Reference; Standard Modules Yes
S33.pod Reference; Diagnostic Messages Yes
My reasoning:
- S14: I'm not clear on whether tied variables have gone the way of
formats.
- S25: "Portable Perl", it's unclear how much the elements addressed
therein would change from Perl5 to Perl6, so it might not be
necessary to rewrite this beyond a few tweaks to the P5 version (like
deleting the section on XS).
- The ones labelled "Probably not" are (arguably) not properly part
of the language but part of the toolkit around it. As such, they
don't really need to be included in the language spec. YMMV.
- S24: There can't really be any "Common Practices" until some time
after 6.0.0 is released, so that's a No.
- S25: Perl Culture is definitely not part of the language design.
- S08: More and more elements are being given first-class status and
auto-referencing behaviors. Once something is first-class and can
smartly manage its own (de)referencing, there is no real need for
user-level operators to do it. However, we still need defined
semantics for how it all works under the hood, so we need a Synopsis.
- S15, S16, S18, S21, S23: I thought these were a pretty clear call
for "gotta have a spec".
- S28, S30-33: The Reference Synopses are "simply" compilations of
information that is designed elsewhere so producing them is more a
matter of good writing than of actual language design. This is good
because, difficult as it is, good writing is much easier than good
language design.
So, to bring it down to brass tacks: there are 5 big chunks (S15,
S16, S18, S21, S23) that remain to be be written, a 6th (S08) that
needs to be written but will probably be fairly short, and 5 (S28,
S30-33) that need to be compiled out of the mass of emails flying
through the lists. I know that substantial progress has been in
defining the semantics of all of these topics, and I have the
impression that it's mostly a question of wrapping up the last 5-20%
of each one, compiling all the data, and writing the Synopsis.
I'd say that qualifies as light at the end of the tunnel indeed!
--Dks
David K Storrs
On Feb 7, 2006, at 6:51 PM, David K Storrs wrote:
>
> I'd say that qualifies as light at the end of the tunnel indeed!
Forgot to say...all of this was was predicated on the idea that the
code can't really be written until the spec is done. Once the spec
is complete (even if not totally frozen), it becomes much easier to
produce the code. Also, simply having the complete spec is a pretty
major milestone that would be worth a lot of "spirit uplifting" points.
--Dks
Stevan Little
> On Tuesday 07 February 2006 14:17, Yuval Kogman wrote:
> > De-facto we have people running PIL on javascript.
> > It works more than parrot does.
>
> No, it works *differently* from Parrot, just as an LR parser works differently
> from an LR parser.
>
> Don't make the mistake of thinking "Wow, it took Parrot X months to get a
> working PGE, while the Pugs version only took Y weeks", especially because
> the Pugs version had the benefit of looking at *already designed, debugged,
> and tested* Parrot code.
The Pugs project and the Parrot project have had very different goals
actually (at least Pugs did from the early days). Pugs aimed to be
able to evaluate Perl 6 code, as a way of testing the language
features and design. It did not really attempt (until the PIL work
began) to provide a VM for Perl 6 to run on. And even the PIL work
began as a way to strip Perl 6 down to a more managable core calculus
which was easier to interpret, the multiple backends seemed to grow
out of that as a side-effect.
So I guess what i am saying is that I agree with you, comparing Pugs
development to Parrot development does not make sense. However, I
think we arrive at that conclusion from different angles.
It seems to me that Pugs has taken a top-down (more language centric)
approach, and Parrot has taken a more bottom-up (runtime/VM centric
approach), and in my eyes, there is a big gapping hole in the middle
(see my response to Allison's post for details about the "big gapping
hole").
Much of what Yuval is proposing is ways to fill that hole and to
decompose and refactor the current Perl 6 development process so that
we can have a real production Perl 6 to play with that much sooner.
But also have a Perl 6 that some PhD canidate can re-write the
type-checker for his thesis project or that some volunteer hobbiest
can re-implement the core in FORTH or some open source hacker can hack
the circular prelude to make the Schwartzian transformation that much
quicker and efficient.
IMHO breaking down the project into smaller more digestable chunks
carries as much risk of failure as putting all the eggs into single
Parrot nest.
At the very least, this is a debate worth having, especially since we
have all been waiting very patiently for so many years now.
Once again...
Respectfully,
Stevan
Chromatic
> The Pugs project and the Parrot project have had very different goals
> actually (at least Pugs did from the early days). Pugs aimed to be
> able to evaluate Perl 6 code, as a way of testing the language
> features and design. It did not really attempt (until the PIL work
> began) to provide a VM for Perl 6 to run on.
In my mind, that's the most valuable thing Pugs could do.
> And even the PIL work began as a way to strip Perl 6 down to a more
> managable core calculus which was easier to interpret, the multiple backends
> seemed to grow out of that as a side-effect.
But they're not free to support.
Now I'm not arguing that the existence of multiple backends takes effort away
from a single unified backend. This is open development. People work on
what they want to work on.
Still, finding the greatest common factor of features between LLVM, Scheme,
Spidermonkey, classic Pugs, Parrot, the CLR, the JVM, Perl 5, and whatever
other VM is out there means pushing a lot of things up the implementation
stack.
> Much of what Yuval is proposing is ways to fill that hole and to
> decompose and refactor the current Perl 6 development process so that
> we can have a real production Perl 6 to play with that much sooner.
I agree that that's his goal. I disagree on its appropriateness.
There are people who can write a bootstrapping compiler from the top down in
such a way that normal people can write the user-level primitives in that
language. I've met those people. I'm not one of them. There are precious
few of them for any language, much less Perl 6.
It's not fast. It's not free. It's not clear that they'll suddenly appear to
do this work if there's a comprehensive, intelligible rework of the Perl 6
plan.
I could be wrong and if Yuval writes the plan and it works, great! I'm happy
to be wrong.
> But also have a Perl 6 that some PhD canidate can re-write the
> type-checker for his thesis project or that some volunteer hobbiest
> can re-implement the core in FORTH or some open source hacker can hack
> the circular prelude to make the Schwartzian transformation that much
> quicker and efficient.
Again, I can see the theoretical benefit to that, but it's still not free.
The well-worn adage is "Good, fast, or cheap -- pick two." Perl 6 development
right now is cheap but hopefully good. Reducing the goodness might make it
faster. Reducing the cheapness might too. I think the real problem is
somewhere in there.
> IMHO breaking down the project into smaller more digestable chunks
> carries as much risk of failure as putting all the eggs into single
> Parrot nest.
Exactly how is Yuval's proposal making the chunks more digestible? There's
sort of a dearth of Scheme, CLOS, Haskell, and Scala experts in Perl 6
development right now. Where are they going to come from to write all this
stuff?
-- c
Audrey Tang
> If Audrey is willing, I think a correct new direction for pugs is to
> try and separate the parts even more - the prelude is a mess right
> now, many of it's part are duplicated across the backends, the
> standard library that is mashed into the prelude, and into pugs core
> itself, etc.
Er, of course I'm willing, that was exactly we've been moving toward
in the recent weeks. :-)
Though an explicit Standard Library design -- as compared to Perl5's which was
grown out gradually by the porters and CPAN folks -- is tricky, and I'm not
yet ready for that, lacking a practical understanding of how module interfaces
and roles can be applied to this diverse design space.
So I will be focusing on Prelude (the part of the language that always gets
loaded by default) refactoring as well as providing an OO core calculus that can
support this, and take advantage of the target VM's vast library instead of
writing them in Perl 6, at least up until 6.2831 (the primary target
VM is Perl 5,
then Parrot, then JavaScript.)
But if you'd like to work on the standard library -- well, you have a
commit bit. :-)
Audrey
Allison Randal
>
> Now I am not as involved in Parrot as I am in Pugs so I might be way
> off base here, but from my point of view Parrot still has a long way
> to go before it runs Perl 6 code. Part of that is because the bridge
> between PIR/PMCs and Perl 6 just does not exist yet (either in code,
> or even conceptually). Having PGE parse Perl 6 code only gives us an
> AST, it does not give us running code. And even if we have a nicely
> massaged AST, running Perl 6 is not a matter of just walking the tree
> and evaluating it like it is in Perl 5 (of course, I am simplifying
> quite a bit here). We found (a few months ago) in Pugs that this model
> just isn't robust enough, and Perl 6 is going to need a more
> sophisticated "runtime" environment to support many of it's features.
> This "runtime" (or as we have been calling it in Pugs the "Object
> Space") will need to exist on top of Parrot too since it is far to
> Perl 6 specific to be implemented into the Parrot core.
>
> This is the kind of stuff that Yuval is talking about. The missing
> bits that need to exist in the nether-region between perl6-language
> and perl6-internals.
>
> We are building from the bottom-up (Parrot) and the top-down (Perl 6 -
> the language) and it seems (at least to many of us on the Pugs
> project) that there is a big hole somewhere in the middle.
You imply here that obstacles to implementing Pugs are necessarily
obstacles to implementing Perl 6. That's not entirely accurate. The
bootstrapping Perl 6-on-Perl 6 architecture does require a high
degree of abstraction. The choice of architecture means there's a
greater gap to fill between the abstraction and the core
implementation. This was my original objection to Pugs, but I changed
my mind. You all have demonstrated incredible skill and energy over
the past year, and I'm confident you will figure out a way to do it.
But, there is another route, and we're working on it at the same
time. From the Parrot perspective, PGE parses the source, its output
is translated to an AST (or a couple of intermediate ASTs), and that
is translated either to PIR, or directly to bytecode. I'm working on
a prototype of this now in Punie, specifically so we can try out the
whole path from source code to bytecode.
Perl 6 will get implemented.
Allison
Stevan Little
>
> > The Pugs project and the Parrot project have had very different goals
> > actually (at least Pugs did from the early days). Pugs aimed to be
> > able to evaluate Perl 6 code, as a way of testing the language
> > features and design. It did not really attempt (until the PIL work
> > began) to provide a VM for Perl 6 to run on.
>
> In my mind, that's the most valuable thing Pugs could do.
Well, a few months ago I would have disagreed, but now I agree with
you, by taking this down the VM level (which is that the PIL^N/PIL2
runcore is focusing on) is good "research" for eventually connecting
this all to Parrot.
I am glad we agree here.
>
> > And even the PIL work began as a way to strip Perl 6 down to a more
> > managable core calculus which was easier to interpret, the multiple backends
> > seemed to grow out of that as a side-effect.
>
> But they're not free to support.
Well yes that is very true, but that was a learning process. It helped
uncover some of the deficencies in the first PIL implementation (most
notable the lack of OO support). It also lead to the development of
the Object Space sub-project which is aiming to clarify how we get
from Perl 6 to something that is executable in an environment which
supports all the features designed.
These are both things which the Parrot project and the Perl 6 design
project did not address from what I can see. Only after going down
some highly experimental paths did this reveal itself.
So while I agree, they are not free to support, I would argue that
they are R&D prototypes and so (to some degree) disposable, and the
benefits they have brought in terms of insight into Perl 6 "the
runtime" (not the language, and not the VM, but somewhere in between)
are very vaulable.
> Now I'm not arguing that the existence of multiple backends takes effort away
> from a single unified backend. This is open development. People work on
> what they want to work on.
Exactly Yuval's point. People want something interesting enough to
pique their interest, but small enough to digest for weekend/nighttime
hacking sessions. If Perl 6 was broken down in such a way as he
proposes, maybe it would attrack more people? or maybe it won't.
Neither I or you knowt that, we can only guess.
> Still, finding the greatest common factor of features between LLVM, Scheme,
> Spidermonkey, classic Pugs, Parrot, the CLR, the JVM, Perl 5, and whatever
> other VM is out there means pushing a lot of things up the implementation
> stack.
Sure that's one way to look at it, but it does not need to be that
way. Reducing Perl 6 down to a core calculus like PIL actually makes
it easier to target any backend you want. And the new PIL^N/PIL2
runcore will make it even easier since all that will be required will
be that you create a PIL^N runtime, all the
metamodel/container/boxed-type prelude will either be written in PIL^N
or in Perl 6. Then the Perl 6 -> PIL2 part can be written using
PGE/TGE/Parsec/whatever.
IMHO this design direction (which makes multiple backends almost
trivial) makes for a better more modular and decoupled design in
general, which is surely a good thing for all projects involved
including Parrot.
> > Much of what Yuval is proposing is ways to fill that hole and to
> > decompose and refactor the current Perl 6 development process so that
> > we can have a real production Perl 6 to play with that much sooner.
>
> I agree that that's his goal. I disagree on its appropriateness.
What is inappropriate about it? He is questioning the current
direction of an open source project which has be regarded as many to
be mearly vaporware. Sure you and I know that Perl 6 is chugging right
along and making great strides, but until Pugs many people considered
Perl 6 to be a joke at best, and total vaporware at worst.
I think Yuval has every right to question the direction, and to make
suggestions as to how he thinks it can be improved upon. He has put in
time on the project, maybe not as much as others, but enough that I
think he has a right to speak up if he wants. What is so wrong with
that?
> There are people who can write a bootstrapping compiler from the top down in
> such a way that normal people can write the user-level primitives in that
> language. I've met those people. I'm not one of them. There are precious
> few of them for any language, much less Perl 6.
Hmm, quite true, but I think that is mostly because the texts on the
subject are so dense and there is a severe lack of hackable projects
out there that people can contribute too that don't involve some
esoteric language meant to explore some equally esoteric concept.
However, that said, the idea of bootstrapping compilers is
progressively getting more mainstream. Many of the recent languages
which have sprung up for the CLR are moving towards bootstrappability.
The new version of Scala is written in Scala. So maybe, as more and
more people do it, they will find simple ways to accomplish that which
you seem to think is so difficult. And maybe that someone will be
Yuval. You never know.
> It's not fast. It's not free. It's not clear that they'll suddenly appear to
> do this work if there's a comprehensive, intelligible rework of the Perl 6
> plan.
Your right, but you never know until you try ;)
> I could be wrong and if Yuval writes the plan and it works, great! I'm happy
> to be wrong.
>
> > But also have a Perl 6 that some PhD canidate can re-write the
> > type-checker for his thesis project or that some volunteer hobbiest
> > can re-implement the core in FORTH or some open source hacker can hack
> > the circular prelude to make the Schwartzian transformation that much
> > quicker and efficient.
>
> Again, I can see the theoretical benefit to that, but it's still not free.
Well if it is a side effect of a modular and clean design, they it is free :)
> The well-worn adage is "Good, fast, or cheap -- pick two." Perl 6 development
> right now is cheap but hopefully good. Reducing the goodness might make it
> faster. Reducing the cheapness might too. I think the real problem is
> somewhere in there.
Okay, I think maybe there has been too much emphasis in this
converstation based on speed of development. My concern (and I know
Yuval's as well) is also with correctness, in the
theoretical/mathmatical sense. Better to get it right first, then all
things will follow in time.
> > IMHO breaking down the project into smaller more digestable chunks
> > carries as much risk of failure as putting all the eggs into single
> > Parrot nest.
>
> Exactly how is Yuval's proposal making the chunks more digestible?
Well modularizing it makes more digestable, thats just a given. These
chunks may seem a little more complex, but IMHO they are much easier
to think/reason about as seperate parts then they are as peices of a
large monolothic project.
> There's sort of a dearth of Scheme, CLOS, Haskell, and Scala experts in Perl 6
> development right now.
Well, some are experts, some are (like me) people who want to have the
cool features of Scheme/CLOS/Haskell/Scala so they are reading as much
info on them as possible to try and apply that to Perl 6.
> Where are they going to come from to write all this stuff?
Well Pugs has either created them, or attracted them already :)
Stevan
Stevan Little
> >
> > Now I am not as involved in Parrot as I am in Pugs so I might be way
> > off base here, but from my point of view Parrot still has a long way
> > to go before it runs Perl 6 code. Part of that is because the bridge
> > between PIR/PMCs and Perl 6 just does not exist yet (either in code,
> > or even conceptually). Having PGE parse Perl 6 code only gives us an
> > AST, it does not give us running code. And even if we have a nicely
> > massaged AST, running Perl 6 is not a matter of just walking the tree
> > and evaluating it like it is in Perl 5 (of course, I am simplifying
> > quite a bit here). We found (a few months ago) in Pugs that this model
> > just isn't robust enough, and Perl 6 is going to need a more
> > sophisticated "runtime" environment to support many of it's features.
> > This "runtime" (or as we have been calling it in Pugs the "Object
> > Space") will need to exist on top of Parrot too since it is far to
> > Perl 6 specific to be implemented into the Parrot core.
> >
> > This is the kind of stuff that Yuval is talking about. The missing
> > bits that need to exist in the nether-region between perl6-language
> > and perl6-internals.
> >
> > We are building from the bottom-up (Parrot) and the top-down (Perl 6 -
> > the language) and it seems (at least to many of us on the Pugs
> > project) that there is a big hole somewhere in the middle.
>
> You imply here that obstacles to implementing Pugs are necessarily
> obstacles to implementing Perl 6. That's not entirely accurate.
Well, the first obstacle I see to implementing Perl 6 can be fixed
with the object space work, and I do not see the Perl 6 Object space
work as being Pugs specific at all. From work I and others have done
on the meta-model as well as the container types, it seems clear that
we need a very robust Perl 6 runtime environment. And currently Parrot
does not provide enough of that environment. This is not to say that
Parrot *cannot*, only that it does not currently. And in my opinion,
Parrot shouldn't cater this much to Perl 6 anyway. Parrot's object
model is sufficently generic to support the object model of most of
the current crop of dynamic languages, but that will not be enough for
Perl 6. You just can't compile all the runtime dynamism into PIR and
PMCs, you will need a runtime environment (an object space) to support
it.
The next obstacle to implementing Perl 6 I see is the
type-checker/inferencer, this is not the job of Parrot, or of PGE. It
a the job for a type inferencer, of which I don't see work on one
currently outside of Pugs and Yuval's Blondie work.
Then there is the prelude. Why write Perl 6's built-ins in PIR when
you can write it in Perl 6? Assuming the Perl 6 codegen is good enough
of course. And modern compiler and optimization technology has been
doing those things since the late-80s (there are many studies of how
compiled Ada code was faster and better than expert hand coded
asssembler, there are just some things a computer can do better than a
person).
I think Pugs and Parrot/PGE share many more obstacles than you might think.
> But, there is another route, and we're working on it at the same
> time. From the Parrot perspective, PGE parses the source, its output
> is translated to an AST (or a couple of intermediate ASTs), and that
> is translated either to PIR, or directly to bytecode.
But this is my point, this won't be enough to support all that Perl 6
is to be. PIR & PMCs simply are not enough to have full metaclass
support, roles (at compile/class composition time, and runtime),
traits, etc. And lets not forget that (to quote Larry in S02 i think)
"Perl 6 is an OO engine". Which means that container types like
Scalars, Arrays and Hashes are objects now too. These things map
nicely to some of the current PMCs, but they are not boxed inside the
object metamodel, and until they are they are not extendable and
usable in the way the language design prescribes.
The Pugs project started out with an AST which was then evaluated,
which is similar to your AST translated to PIR, and we just found it
wasn't enough. Perl 6 is just simply to dynamic a language for that.
> I'm working on a prototype of this now in Punie, specifically so we can try out the
> whole path from source code to bytecode.
I am familiar with your Punie work as I read your use.perl journal and
the Perl 6 meeting notes regularly. But IIRC Punie is a compiler for
Perl 1 is it not? Perl 1 is a very very very long way from Perl 6.
> Perl 6 will get implemented.
Oh, of that I have no doubt. Never did, and neither does Yuval (if I
may speak for him while he is asleep :). But all that we are trying to
do here is shake out some cobwebs, a little spring cleaning if you
will.
Stevan
Allison Randal
>
>> Perl 6 will get implemented.
>
> Oh, of that I have no doubt. Never did, and neither does Yuval (if I
> may speak for him while he is asleep :). But all that we are trying to
> do here is shake out some cobwebs, a little spring cleaning if you
> will.
Excellent. I wish you much fun! :)
Allison
Yuval Kogman
> On 2/8/06, Yuval Kogman <nothi...@woobling.org> wrote:
> > If Audrey is willing, I think a correct new direction for pugs is to
> > try and separate the parts even more - the prelude is a mess right
> > now, many of it's part are duplicated across the backends, the
> > standard library that is mashed into the prelude, and into pugs core
> > itself, etc.
>
> Er, of course I'm willing, that was exactly we've been moving toward
> in the recent weeks. :-)
>
> Though an explicit Standard Library design -- as compared to Perl5's which was
> grown out gradually by the porters and CPAN folks -- is tricky, and I'm not
> yet ready for that, lacking a practical understanding of how module interfaces
> and roles can be applied to this diverse design space.
By standard library is i don't mean core modules - it's Perl 6's
perlfunc + some really critical pieces.
> So I will be focusing on Prelude (the part of the language that always gets
> loaded by default) refactoring as well as providing an OO core calculus that can
> support this, and take advantage of the target VM's vast library instead of
> writing them in Perl 6, at least up until 6.2831 (the primary target
> VM is Perl 5,
> then Parrot, then JavaScript.)
Aye =)
--
() Yuval Kogman <nothi...@woobling.org> 0xEBD27418 perl hacker &
/\ kung foo master: /me wields bonsai kittens: neeyah!!!!!!!!!!!!!!!!
Yuval Kogman
stevan seems to be getting the wrong impression.
Perl 6 has some hard to answer questions. The questions the
community has answered so far are:
* How the VM will work/look
* What the syntax/feature requirements are
If we ignore pugs for a second.
These are though questions to answer, and I have no criticism
whatsoever that they took long to answer.
Development speed is *NOT* what this post was about.
What I do think is that there is something in the middle of these
two big questions, and they are:
* How will the Perl 6 compiler be designed (parts, etc)
* What are the definitions of some of the standard features
mentioned in the Synopses ( S29 is a good start, but we need
more than the standard perlfunc)
If we let Perl 6 continue to grow organically we will have answers
to these questions, but we will likely invest lots of effort in
trial and error.
I think that some of these questions can be answered based on some
up front design, thinking, and decision making, thus helping us
direct our trial and error efforts towards a more defined goal.
Furthermore, I think this has important implications on the quality
of the implementation of the Perl 6 compiler and standard library,
and that we should start worrying about that too.
The second (much larger) part of the post contains a sort of draft,
if you will, of what I think can be a good start towards trying to
answer these questions.
Thanks
--
() Yuval Kogman <nothi...@woobling.org> 0xEBD27418 perl hacker &
/\ kung foo master: /methinks long and hard, and runs away: neeyah!!!
Audrey Tang
> What I do think is that there is something in the middle of these
> two big questions, and they are:
>
> * How will the Perl 6 compiler be designed (parts, etc)
That... was what Pugs Apocrypha was meant to contain, with PA02 being a
design overview, and PA03 onward documenting the various parts with
their interfaces.
But English is not my forte (by far), and the last time you and I worked
toward it, it resulted in a jargon-filled document largely inaccessible
to a casual participant. So that needs to be fixed.
I'll be in Tel Aviv in 5 days (thank $deity), and I'd be very willing to
work with you on this before the Hackathon starts.
Audrey
Chromatic
> Does this imply that we should think up this process?
Go ahead.
> If I propose a concrete plan for the implementation of Perl 6 in a
> layered fashion it will probably be even more overlooked.
>
> I have no authority, and this is not something I can do on my own.
If other people find it valuable, that's all the authority you need.
The last time someone tried to set forth a complete specification in a
particular order was the Perl 6 documentation project. That didn't work
then. I have doubts that specifying a complete compiler and toolchain
without at least some trial and error will work, but I could be wrong.
Maybe the right place to start is to gather a list of all of the questions you
need to have answered and all of the features people want, and then try to
unify them into a Perl 6-ish whole.
-- c
Yuval Kogman
> On Tuesday 07 February 2006 23:55, Yuval Kogman wrote:
>
> > Does this imply that we should think up this process?
>
> Go ahead.
We'll start at the Israel hackathon, with a little preamble.
> The last time someone tried to set forth a complete specification in a
> particular order was the Perl 6 documentation project. That didn't work
> then.
I doubt it'll work now, either... Here is my reply on #perl6 to your
discussion on Perl 6:
My reply:
http://colabti.de/irclogger/irclogger_log/perl6?date=2006-02-09,Thu&sel=11
Your discussion:
http://colabti.de/irclogger/irclogger_log/perl6?date=2006-02-08,Wed&sel=348#l558
> I have doubts that specifying a complete compiler and toolchain
> without at least some trial and error will work, but I could be wrong.
Trial and error is always required, and a very good tool for
innovation in the hands of the community. I don't think Big Bang
design is ever good, but I also believe a rough plan with easy to
understand milestones (as opposed to a daunting cliff) are required,
even if not formal or even on paper.
Also, I am trying to formulate a plan that will help us write most
of the parts in Perl 6, *NOT* Haskell, because I, like you, despite
my love for Haskell, think it's just too inaccessible.
What I'd like is to optimize the modularization such that Pugs
serves as a bootstrap parser/interpreter/compiler - a good solid
tool to help us write:
The Perl 6 standard library in Perl 6 (with stubs for IO, system
calls, and other things that cannot be defined in a "pure"
language)
The Perl 6 compiler toolchain in Perl 6 (the linker, compiler,
emitters, and an interpreter, too).
And then eventually refactor the current state of pugs into a
haskell runtime, and and possibly a historical parser/compiler that
we can use to compare things to.
The way things are headed now, we are just shy of being able to
write good tools in Perl 6 using pugs - it's too slow, the object
model is not finalized, the grammar is not extensible, etc etc.
These are many things that are mentioned in the synopses but not
described in enough detail, and if we want the other parts to be in
Perl 6 we need these done in haskell first, and then rewritten. If
we get them later, we'll have to write the other parts in haskell
too.
> Maybe the right place to start is to gather a list of all of the questions you
> need to have answered and all of the features people want, and then try to
> unify them into a Perl 6-ish whole.
Yes, that's an excellent start, and in fact, I think this is what
Audrey plans on starting with at the prehackathon, when she arrives
in Israel and works with Gaal.
Unfortunately for myself, I will be unable to follow this discussion
as of ~14:00 GMT, today (Feb 9th) as I'm going to visit my grand
parents in Austria, and try not to die while snowboarding.
--
() Yuval Kogman <nothi...@woobling.org> 0xEBD27418 perl hacker &
/\ kung foo master: /me tips over a cow: neeyah!!!!!!!!!!!!!!!!!!!!!!
ilya
open source project and I understand that no one is getting paid to do
this, at least most. But for the last 4/5 years, due to the
anticipation of Perl 6, the development of any major perl 5 features
have pretty much stalled as well. That is why the state of Perl is so
poor right now, as compared to Java and/or Ruby. I'm not saying that
perl 6 is a bad idea, etc..., it's definitely a great project, but...
Although I think that the market is ready for a new language, I don't
know if perl 6 is it. Ruby is great, but still not what the market
needs. For masses to move to perl 6 as opposed to Java/Ruby, there
needs to be a compeling reason for doing so. Perl 6 has so many
changes, that even a perl guru would have to retrain pretty much from
scratch.
What I see happening is people moving more and more towards Java. Most
enterprises are adapting it as a standard development environment and a
lot of open source folks are looking into Ruby these days. Definitely
some excitement going on with the RoR framework. Perl was always very
strong, but certain hard-line "opinions" (I won't mentioned anyone
here) from folks, are killing it. I respect everyone in the perl
community and have participated for quite a few years now, but I
completely agree with the original post, momentum is dying, and not
only in the perl 6, which I think most folks to this day don't truly
know what it will eventually be, but in Perl 5 as well. I'm yet to see
anything exciting come out of either of these communities in the last
few years.
That being said, I truly hope that the project succeeds, if not Perl 6,
at least Parrot. It can possibly compete with the JVM's of the worlds
in a dynamically typed language.
Also, I didn't post this for the flames to begin, though I don't think
this disclaimer will stop anyone:-) Just my honest opinion and what
I'm seeing in the industry.
Ilya Sterin
> --WTbgq2twYBxfsYA6
> Content-Type: application/pgp-signature
> Content-Transfer-Encoding: base64
> X-Google-AttachSize: 190