PGE: Code Blocks
Matt Diephouse
mentioned on IRC that this issue came up at YAPC, but didn't seem to
indicate that there was any plan. As we transition to using PGE for
Tcl parsing, it'd be nice to have code blocks.
Specifically, it'd be nice to be able to specify a compiler to use
along with a code block. In the immediate future, it'd be nice if we
could use PIR for these. That would allow us to throw exceptions to
catch parse errors.
--
matt diephouse
http://matt.diephouse.com
Patrick R. Michaud
> What is the plan for integrating code blocks into PGE? Will Coleda
> mentioned on IRC that this issue came up at YAPC, but didn't seem to
> indicate that there was any plan. As we transition to using PGE for
> Tcl parsing, it'd be nice to have code blocks.
Oh, there is definitely a plan (several, in fact), however, getting
code blocks to work properly requires some coordination between
PGE and the underlying compiler language. In short, when PGE's
parser encounters a code block, it needs to hand off control to
the target language's compiler to parse to the end of the
code block and receive back from that compiler the length of
the block parsed.
Or, we try to do something along the lines of Text::Balanced and
find the closing brace by doing simple delimiter counting, but
Perl 6 brings in several new delimiters that have to be taken into
account in the balancing act.
> Specifically, it'd be nice to be able to specify a compiler to use
> along with a code block. In the immediate future, it'd be nice if we
> could use PIR for these. That would allow us to throw exceptions to
> catch parse errors.
For PIR there is already a possibility -- simply define a subrule
that executes the PIR code desired. For example, one can do:
.sub throw
.param pmc mob # match object
# execute whatever code you want here
# ...
# to return "success" to the matching engine
$P0 = getattribute mob, "PGE::Match\x0$:from"
$P1 = getattribute mob, "PGE::Match\x0$:pos"
assign $P1, $P0
.return (mob)
.end
and then the <throw> subrule will execute the PIR subroutine.
I'm currently working on the parameterized subrule syntax from
A05/S05, so that one would be able to do:
<throw:Don't do that!>
and the string "Don't do that!" would be passed as a parameter
to the "throw" subrule (where it could be printed, parsed,
etc).
Pm
Larry Wall
: In short, when PGE's
: parser encounters a code block, it needs to hand off control to
: the target language's compiler to parse to the end of the
: code block and receive back from that compiler the length of
: the block parsed.
That is the preferred way.
: Or, we try to do something along the lines of Text::Balanced and
: find the closing brace by doing simple delimiter counting...
Perl 5 would have done it with two passes. Which is why Perl 6 is
specced to always do one-pass parsing instead. :-)
One wrinkle of one-pass is that the parser must somehow know where
to quit. It's fine if you know you want a block and can call a rule
that automatically terminates on '}', or if your rule reliably gets an
error at the point it should stop. But in the general case you might
want to be able to pass in a set of terminating delimiters that stop
the outermost parse even if it looks like it should otherwise continue.
: ...Perl 6 brings in several new delimiters that have to be taken into
: account in the balancing act.
These should generally be recognizable from Unicode properties, but
that's not a good reason to take the balanced approach. And as soon
as people start defining their own circumfix operators that violate
the Unicode properties, all bets are off. Even a user-defined quote
is going to cause grief unless you know the / of xxx/.../ is an opener.
For languages that cannot do one-pass parsing, it would be saner in
the long run for rules to delimit such code with delimiters that
are unlikely to occur in the target language. Double curlies, or
here docs, or some such. That's hacky, but counting brackets is
also hacky. Any time you write two different parsers for the same
language, it's a new set of bugs just waiting to happen.
Larry
Patrick R. Michaud
>
> For languages that cannot do one-pass parsing, it would be saner in
> the long run for rules to delimit such code with delimiters that
> are unlikely to occur in the target language. Double curlies, or
> here docs, or some such. That's hacky, but counting brackets is
> also hacky.
Any chance we could identify such a set of delimiters and standardize
them within the rules language, or at least within PGE?
In the general case, Parrot's "compile" opcode doesn't (yet?) have
the interface we'd really need or want in order to do the "parse
a program up to a closing delimiter and return the result" sort
of thing that we'd need. And if a target language is more of
an interpreter and not a compiler, then it might need a flag that
indicates "just parse this, don't execute it yet". Maybe we
need a "parse" opcode, or a standard parse function-call interface
that language parsers use to interface with PGE.
It could also be that one might wish to avoid parsing and/or
compiling a code block until it's actually encountered during
execution of the rule.
Perl 6 rightfully grabs { ... } for its code blocks, and it's reasonable
to expect the rules engine to have some intimate knowledge of the
Perl 6 parser (and vice versa). But from a more general "tool"
perspective it seems like it'd be nice to have a set of delimiters
available for codeblocks that the rules parser could use without
having to communicate with another parser to handle it. Even if
the chosen delimiters aren't a 100% solution, if they manage to cover
a wide swath of the most common language syntaxes I think it'd be
a win for most language and tool developers.
Technically, one could conceivably use the string-argument form of
subrules (mentioned in A05) to achieve this:
\d+ <tcl: ...tcl-code-here...>
but we'd have to have some mechanism to escape any '>' characters
in the string argument. And that could get pretty nasty, and we
haven't really spec'd out the string delimiters here yet (do all
backslash-escapes get processed)?
In the more general case it seems like it'd be really nice to
have a generalized delimiter available, such as double-curlies,
angle+double-curlies, or something that could be extracted without
resorting to a language-specific parser to find the end of the code
block:
rx :code('tcl') / \d+ {{ ...codeblock source... }} /
rx :code('tcl') / \d+ <{{ ...codeblock source... }}> /
rx / \d+ <tcl{{ ...codeblock source... }}> /
Some other ideas are at the end of this message.
It might also be worth mentioning that the format of the opening
delimiter isn't terribly important -- it's just a closing delimiter token
that PGE or a rules engine needs to scan for; and hopefully that closing
delimiter is something that is unlikely to ever appear in a code block
for a target language (or can be easily worked around when it does).
Anyway, if this is really chasing down a dead-end, just say so and
we'll go with the other approach. But I feel like it simplifies things
a lot (both for PGE and for compiler authors) if we can have a
generalized code block delimiter available.
Pm
Some other random syntax possibilities--using chars not yet defined
for angles:
<| ... code block ... |>
<* ... code block ... *>
<` ... code block ... `>
<^ ... code block ... ^>
<~ ... code block ... ~>
Thus far I think I like {{...}}, <{{...}}>, or <`...`> the best.
Pm
Larry Wall
: On Tue, Jul 05, 2005 at 08:51:39AM -0700, Larry Wall wrote:
: >
: > For languages that cannot do one-pass parsing, it would be saner in
: > the long run for rules to delimit such code with delimiters that
: > are unlikely to occur in the target language. Double curlies, or
: > here docs, or some such. That's hacky, but counting brackets is
: > also hacky.
:
: Any chance we could identify such a set of delimiters and standardize
: them within the rules language, or at least within PGE?
Sure. For the moment it's still our language to do however we like.
: In the general case, Parrot's "compile" opcode doesn't (yet?) have
: the interface we'd really need or want in order to do the "parse
: a program up to a closing delimiter and return the result" sort
: of thing that we'd need. And if a target language is more of
: an interpreter and not a compiler, then it might need a flag that
: indicates "just parse this, don't execute it yet". Maybe we
: need a "parse" opcode, or a standard parse function-call interface
: that language parsers use to interface with PGE.
Some languages could have difficulty with that, I suppose.
: It could also be that one might wish to avoid parsing and/or
: compiling a code block until it's actually encountered during
: execution of the rule.
Hmm. That's probably more in the province of { FooLang.eval("...") },
assuming FooLang doesn't have its own eval.
: Perl 6 rightfully grabs { ... } for its code blocks, and it's reasonable
: to expect the rules engine to have some intimate knowledge of the
: Perl 6 parser (and vice versa). But from a more general "tool"
: perspective it seems like it'd be nice to have a set of delimiters
: available for codeblocks that the rules parser could use without
: having to communicate with another parser to handle it. Even if
: the chosen delimiters aren't a 100% solution, if they manage to cover
: a wide swath of the most common language syntaxes I think it'd be
: a win for most language and tool developers.
:
: Technically, one could conceivably use the string-argument form of
: subrules (mentioned in A05) to achieve this:
:
: \d+ <tcl: ...tcl-code-here...>
:
: but we'd have to have some mechanism to escape any '>' characters
: in the string argument. And that could get pretty nasty, and we
: haven't really spec'd out the string delimiters here yet (do all
: backslash-escapes get processed)?
Yes, avoiding that kind of ambiguity is precisely why the Perl call
variant requires parens: <foo(...)>. It would be yucky to reintroduce
it on behalf of other languages. But, hey... :-)
: In the more general case it seems like it'd be really nice to
: have a generalized delimiter available, such as double-curlies,
: angle+double-curlies, or something that could be extracted without
: resorting to a language-specific parser to find the end of the code
: block:
:
: rx :code('tcl') / \d+ {{ ...codeblock source... }} /
: rx :code('tcl') / \d+ <{{ ...codeblock source... }}> /
: rx / \d+ <tcl{{ ...codeblock source... }}> /
:
: Some other ideas are at the end of this message.
I think I'd rather see a :lang('tcl') option since :c is taken and
:l isn't. But mostly people will want to put
use rule :lang<tcl>;
or some such at the beginning of the file, since all the rule actions
are likely to be in the same language.
: It might also be worth mentioning that the format of the opening
: delimiter isn't terribly important -- it's just a closing delimiter token
: that PGE or a rules engine needs to scan for; and hopefully that closing
: delimiter is something that is unlikely to ever appear in a code block
: for a target language (or can be easily worked around when it does).
But from a human point of view it would be nice if it feels "nesty".
: Anyway, if this is really chasing down a dead-end, just say so and
: we'll go with the other approach. But I feel like it simplifies things
: a lot (both for PGE and for compiler authors) if we can have a
: generalized code block delimiter available.
We need to pursue both options. Languages with left-to-right parsers
aren't going to want to put {{...}} everywhere.
: Pm
:
: Some other random syntax possibilities--using chars not yet defined
: for angles:
:
: <| ... code block ... |>
: <* ... code block ... *>
: <` ... code block ... `>
: <^ ... code block ... ^>
: <~ ... code block ... ~>
:
: Thus far I think I like {{...}}, <{{...}}>, or <`...`> the best.
I'd lean more toward the infinitely extensible PODly technique:
{...}
{{...}}
{{{...}}}
{{{{...}}}}
...
In any case, we probably want the outermost delimiters to be curlies
so that we can use these anywhere we allow Perl closures, such as
\d**{{ (range 1 5) }}. Though perhaps Lisp is a bad example of a
bad language. :-)
Larry
Patrick R. Michaud
> I think I'd rather see a :lang('tcl') option since :c is taken and
> :l isn't. But mostly people will want to put
> use rule :lang<tcl>;
> or some such at the beginning of the file, since all the rule actions
> are likely to be in the same language.
Fair enough. :-)
> : Anyway, if this is really chasing down a dead-end, just say so and
> : we'll go with the other approach. But I feel like it simplifies things
> : a lot (both for PGE and for compiler authors) if we can have a
> : generalized code block delimiter available.
>
> We need to pursue both options. Languages with left-to-right parsers
> aren't going to want to put {{...}} everywhere.
Agreed; I was looking for an additional option as opposed to a
replacement.
> I'd lean more toward the infinitely extensible PODly technique:
>
> {...}
> {{...}}
> {{{...}}}
> {{{{...}}}}
This is very fine with me, except I'm guessing that the {...} form
(single curly brace) probably needs to call the language parser to
find the end and not simply stop on the first closing curly brace
encountered. But the others seem to be just fine, and I'll write
PGE that way for the time being.
Pm