the value of maintaining a working install target to the parrot project

0 views
Skip to first unread message

Mike Mattie

unread,
Mar 18, 2007, 3:18:08 AM3/18/07
to perl6-i...@perl.org
Hello,

-> 1. Introduction

I am a programmer interested in the parrot vm. In my spare time I have
attempted several times to 'get into' parrot. This time I have
progressed much further, and in doing so I have developed a perspective
that is informed enough for me to hazard some input on the project.

I would like to get a clean parrot install on my machine so I can
hack on it. I have quickly reached the limits of what can be done
without submitting changes to parrot itself.

In short parrot needs work on the install target, and I need to
justify the value of the target to the project before proposing any
changes.

I will first present myself as a use case, and then proceed to the
main point.

-> 2. My use case

My first step was to try and install parrot. It was not enough for me
to simply hack in a "check-out" of the parrot tree. When a project
interests me I want to first take small steps.

The first step is always to install the project, and get a "-V" dump
and "hello world" to run. If there are example sources I like to run
those and then look at the code.

I was able to reach the "-V" dump and examples/ stage fairly
easily. The largest problem I ran into was ensuring that all of the
necessary pbc/pir/pasm files where there. I will expand on that issue
in a later mail.

The next step is "hello world" on the language I was interested in. I
would use that "itch" to get a beach-head on PIR, and more importantly
the real showcase of parrot's impressive features would naturally be
the the languages or apps/tools that have been built with parrot.

With hyperbole something like "built scheme in 100 lines with parrot",
or "best etags implementation to date with PGE". The killer app/demo,
think Ruby before/after rails.

At this point things became very difficult, there are a few problems
that all stem from a "check-out" centric work-flow common to all
developers. It works in the check-out but the install is broken.

Caveat:

I have been using linux for a long time. my use case is the opposite
of a more casual user. I took it one small step at a time *for
parrot*, many more will go right to the languages parrot
provides. Assembly was one of the first languages I learned, but many
these days are not even familiar with C.

This is the concise and relevant portions of my use case that I can
present without overloading the subject line.

-> 3. Value of a install target

Maintaining a working install target will greatly benefit the
development of parrot.

I see three major use cases for parrot: The parrot hacker, the
language hacker, and the end-user (languages/apps).

Some time ago it was noticed that the tree of languages was large with
alot of stalled projects. The parrot hackers recognized this and
addressed it by becoming more responsive to the language developers.

I gathered this from a slide presentation, I believe it was: "Parrot:
Present and Future" by Chip Salzenburg ( the link is dead, cannot
confirm ).

This is a critical point. Without the languages running on-top of
parrot, the parrot project would slow to a crawl, or possibly
stall.

When you get down to it, in a human sense writing something without
users is lonely and boring. Once the itch is scratched the code is
forgotten until it breaks.

The parrot hackers recognized this and addressed it. However the
language hackers are in the same position now as the parrot hackers.

parrot hacker

| (improving)

language hacker

| (broken)

language user

I looked at several of the languages in the parrot tree and saw alot
of test-harness like stuff. When I went to run pheme with: (+ 1 1)
I found pheme broken completely with:

Couldn't find file 'compilers/tge/TGE/Rule.pbc'

[Pheme does work in the parrot tree, remember this is with the
install tree. Also I am not picking on pheme, it's just what I want to
run first.]

Assuming that I want to build a language of my own for whatever
reason, I want to use parrot both as a infrastructure and a deployment
platform. The first to harness the power of parrot, the second to get
users, start some feedback/community/fun-with-peers.

Parrot cannot fully serve the language hackers without serving the
language user as well with a installable parrot. Closing the circle
will accelerate the project by serving the key motives that will
attract brain-share.

Turn the use-case digram above upside-down and and the value of a
install target is instantly apparent.

language user

|

language hacker

|

parrot hacker

recap:

The parrot hackers depend on feedback from language hackers, and in
doing so make sure that parrot works for the in-tree languages.

Language hackers need feedback too, and without a installable parrot
only the macho peers, or "really good friends" will jump all of the
hurdles that a broken parrot install imposes on getting "hello world"
to run. Without a usable install target making a distro build is a
huge burden, and just not practical right now.

-> 3.1 reasonable expectations for a parrot install target

First parrot is still in a developmental stage without a locked-down
ABI/API. Trying to create a ABI/API at this point would be a disaster
IMHO.

What is a reasonable expectation ?

I think that if a parrot hacker gets up and does a slide-show with the
highlights of parrot, that viewers should be able to install parrot
using their distro, and then run the examples given.

Instead of trying to untangle the impossible problem of an API/ABI
re-formulate the problem in this way:

"what cool features do we want to show off"

The problem is instantly graspable and tractable. As far as
compatibility goes it is already punted. Parrot is known to be in
development, and a distro can put a "API/ABI in development" warning
in the install.

-> 3.2 requirements of a parrot install target.

1. install all of the necessary files into the install image.

Parrot is badly broken here. below is a list of files I added in my
install image via ebuild (gentoo build system)

Some of these files are obviously experiments like STM.pbc but
things like PGE/{Text,Util}.pbc are used all over the place.

A distro packager does not know the difference between experiments
and dependencies without investing in the knowledge of a parrot
contributer.

+runtime/parrot/library/CGI/QueryHash.pbc [main]
+runtime/parrot/library/Data/Replace.pbc [main]
+runtime/parrot/library/Digest/MD5.pbc [main]
+runtime/parrot/library/JSON.pbc [main]
+runtime/parrot/library/MIME/Base64.pbc [main]
+runtime/parrot/library/Node.pbc [main]
+runtime/parrot/library/PAST-pm.pbc [main]
+runtime/parrot/library/PAST.pbc [main]
+runtime/parrot/library/PAST/Code.pbc [main]
+runtime/parrot/library/PAST/Exp.pbc [main]
+runtime/parrot/library/PAST/Op.pbc [main]
+runtime/parrot/library/PAST/Stmt.pbc [main]
+runtime/parrot/library/PAST/Stmts.pbc [main]
+runtime/parrot/library/PAST/Sub.pbc [main]
+runtime/parrot/library/PAST/Val.pbc [main]
+runtime/parrot/library/PAST/Var.pbc [main]
+runtime/parrot/library/PGE/Dumper.pbc [main]
+runtime/parrot/library/PGE/Glob.pbc [main]
+runtime/parrot/library/PGE/P6Grammar.pbc [main]
+runtime/parrot/library/PGE/Text.pbc [main]
+runtime/parrot/library/PGE/Util.pbc [main]
+runtime/parrot/library/Parrot/Capture_PIR.pbc [main]
+runtime/parrot/library/Parrot/Coroutine.pbc [main]
+runtime/parrot/library/Parrot/HLLCompiler.pbc [main]
+runtime/parrot/library/STM.pbc [main]
+runtime/parrot/library/String/Utils.pbc [main]
+runtime/parrot/library/TGE.pbc [main]
+runtime/parrot/library/YAML/Parser/Syck.pbc [main]
+runtime/parrot/library/libpcre.pbc [main]
+runtime/parrot/library/postgres.pbc [main]

2. equivalent behavior when running in a install tree, and a build tree.

The paths requested by higher-level components are broken, ie:
hard-coded to the layout of the check-out tree. lots of
load_bytecode/include foo.{pir,pasm,pbc}.

The extensions are fairly easy to deal with, just drop the extensions
and DTRT. I will address that in a separate mail.

directory components in paths is a bit more tricky. The loader needs
to be exercised in the same way in both the install-tree and the
build-tree. I have a proposal for this in development, but it is a
little bit more invasive than getting rid of hard-coded extensions.

3. install targets/lists for the languages.

It is not strictly necessary to do a --enable-foo for every language
to enable this, at least for gentoo. For pheme I was able to do this
with the gentoo ebuild compilation phase.

<snip>
if use "pheme" ; then
cd languages
emake pheme || "pheme compile failed"
fi
</snip>

any decent distro build system will not have a problem doing more than
one make invocation. With building covered my work-around for the install
is next:

<snip>
if use "pheme" ; then
einfo "installing pheme language"

dodir /usr/$(get_libdir)/parrot/languages

dodir /usr/$(get_libdir)/parrot/languages/pheme
insinto /usr/$(get_libdir)/parrot/languages/pheme
doins ${S}/languages/pheme/pheme.pbc

dodir /usr/$(get_libdir)/parrot/languages/pheme/lib
insinto /usr/$(get_libdir)/parrot/languages/pheme/lib

doins ${S}/languages/pheme/lib/pheme_grammar_gen.pir

for pbc in $( find ${S}/languages/pheme/lib -name '*.pbc' -print ) ; do
# einfo "installing pbc file $pbc"
doins $pbc

done
fi
</snip>

Two major issues here:

3.1. where do languages go ?

This is a distro issue, I will ponder more on it , but ignore it for now.

3.2. What to install ?

This is the showstopper. that find command is hideous. I don't even
need an actual install target that each language maintains. At
this point a flat-file with one-per-line listing of the runtime files
would be just fine.

The big thing to notice here is that I am not talking about breaking
the languages out of the tree. I think that would greatly harm the
parrot project. It is sufficient to have a make target for building,
and a list of files to install, or another make target for installing
if that is deemed necessary.

Full recap:

A parrot install target will benefit the project by providing the
platform needed to drive feedback to language hackers. This will
trickle-down to the parrot hackers accelerating the development
process and bringing in more brain-share and project visibility.

the ABI/API is punted, instead of trying to build a comprehensive
exported API at this stage, instead ask what cool things parrot
hackers would like to show off, and put that in the install list.

Only make build targets and install lists are minimally necessary.
Languages should remain in-tree. Adding a source/built-object to
the install list should be data-driven and dead simple for language
hackers (per-language MANIFEST ?)

Parrot,languages, and apps need to behave the same in both the
install tree, and the build-tree.

This could be dropped if my later proposal is determined
unsuitable/undesirable, however that will lead to some heavy-handed
release testing/policy etc. It's usually better if something just
breaks in both trees then waiting until tarballs are rolled to find
out what's broken in the install. There are enough day-after release
releases already.

status:

I am working [ spare-time ] on a gentoo ebuild (un-official , I am
_not_ a gentoo dev ), and I am also working up a couple of proposals
for deltas to the parrot tree addressing some of the issues that I
have raised.

P.S: finally !

My thanks for considering these thoughts. Anything seen as criticism
should be considered constructive, as I am very impressed by parrot
and wish to see it flourish.

cheers,
Mike Mattie (coder...@gmail.com)


signature.asc

Allison Randal

unread,
Mar 21, 2007, 9:35:31 PM3/21/07
to Mike Mattie, perl6-i...@perl.org
Thanks, Mike, and welcome to the bird house. :)

Mike Mattie wrote:
>
> In short parrot needs work on the install target, and I need to
> justify the value of the target to the project before proposing any
> changes.

The install process is generally known to need work, and critical to the
1.0 release, so we welcome your patches and suggestions.

> -> 2. My use case

> -> 3. Value of a install target

Agreed that the average users (i.e. the greatest number of people using
Parrot) are not the compiler writers, but the language users, and on the
importance of a working install target. So, I'll skip down to:

> I looked at several of the languages in the parrot tree and saw alot
> of test-harness like stuff. When I went to run pheme with: (+ 1 1)
> I found pheme broken completely with:
>
> Couldn't find file 'compilers/tge/TGE/Rule.pbc'

Part of this problem is that Parrot doesn't currently have the
equivalent of @INC or 'use lib'. We shouldn't be specifying a literal
file path to a module here, but a simple name, and have Parrot search a
(modifiable) list of directories.

> What is a reasonable expectation ?
>
> I think that if a parrot hacker gets up and does a slide-show with the
> highlights of parrot, that viewers should be able to install parrot
> using their distro, and then run the examples given.
>
> Instead of trying to untangle the impossible problem of an API/ABI
> re-formulate the problem in this way:
>
> "what cool features do we want to show off"

That's a good fundamental goal.


> -> 3.2 requirements of a parrot install target.
>
> 1. install all of the necessary files into the install image.
>
> Parrot is badly broken here. below is a list of files I added in my
> install image via ebuild (gentoo build system)
>
> Some of these files are obviously experiments like STM.pbc but
> things like PGE/{Text,Util}.pbc are used all over the place.
>
> A distro packager does not know the difference between experiments
> and dependencies without investing in the knowledge of a parrot
> contributer.

I expect that some files will be included in the "core" install image,
and others will be loaded as separate packages. Looking over your list,
I agree that most of those modules are necessary to doing anything
useful with Parrot, and so should be included in the core. Most (perhaps
all) languages/ won't be, so I'd have to run something like (debian
example):

$ aptitude install parrot-python

to get the python language implementation on Parrot.

(Note that this means anyone can release their own language
implementation for Parrot as a separate package, without getting it
added to the core.)

What we need is better meta-information for the install process and
package builders on what is and isn't necessary for a core install. And,
a strategy for installing extras in the basic install process.

> 2. equivalent behavior when running in a install tree, and a build tree.
>
> The paths requested by higher-level components are broken, ie:
> hard-coded to the layout of the check-out tree. lots of
> load_bytecode/include foo.{pir,pasm,pbc}.

I'm happy enough with this, but it does assume that all the .pir, .pasm,
or .pbc files are stored in the same directory (there's been some debate
about that, and about the corresponding .tg and .pg files).

> directory components in paths is a bit more tricky. The loader needs
> to be exercised in the same way in both the install-tree and the
> build-tree. I have a proposal for this in development, but it is a
> little bit more invasive than getting rid of hard-coded extensions.

Agreed, and connect to earlier comments about @INC/'use lib'.

> 3. install targets/lists for the languages.

[...]


> 3.1. where do languages go ?
>
> This is a distro issue, I will ponder more on it , but ignore it for now.

Yeah, the exact directory structure isn't critical, but figuring out how
to handle varying directory structures for various distros is
(configurable search paths again).

> 3.2. What to install ?
>
> This is the showstopper. that find command is hideous. I don't even
> need an actual install target that each language maintains. At
> this point a flat-file with one-per-line listing of the runtime files
> would be just fine.
>
> The big thing to notice here is that I am not talking about breaking
> the languages out of the tree. I think that would greatly harm the
> parrot project. It is sufficient to have a make target for building,
> and a list of files to install, or another make target for installing
> if that is deemed necessary.

I suspect languages will need to each have their own install target,
with the appropriate information about which files need to be installed.
Especially when we get into languages that aren't part of the Parrot
repository. And being able to ship a language separate from the whole
Parrot install is an important goal, in much the same way that you can
ship a tool that expects a C compiler without including the C compiler
in your tarball.

It may be that we can handle this with a simple YAML file in each
languages/ directory.


> Only make build targets and install lists are minimally necessary.
> Languages should remain in-tree. Adding a source/built-object to
> the install list should be data-driven and dead simple for language
> hackers (per-language MANIFEST ?)

MANIFEST is also possible, though it usually indicates which files are
necessary to build, not which files should be installed (e.g. it
includes all the tests and source files, which won't be installed).


> Parrot,languages, and apps need to behave the same in both the
> install tree, and the build-tree.

Absolutely. We can't entirely get around the fact that they are
different environments (for example, the install will be lacking a
number of files only needed for the build process), but we can minimize
the differences as much as possible.

We might setup a framework for running tests against the installed
Parrot too (not for general users to run, but for maintainers to run).


> I am working [ spare-time ] on a gentoo ebuild (un-official , I am
> _not_ a gentoo dev ), and I am also working up a couple of proposals
> for deltas to the parrot tree addressing some of the issues that I
> have raised.

Thanks for the thorough and carefully considered analysis of the
problem. I look forward to reviewing the specific change
proposals/suggestions that grow out of it.

Allison

Reply all
Reply to author
Forward
0 new messages