Simple::Lexer optionally shouldn't call end_input
6 views
Skip to first unread message
Peter Stuifzand
Apr 13, 2012, 11:43:58 AM4/13/12
to marpa-...@googlegroups.com
Hi all,
I will start with my point and give an example below. It would be nice to have another interface to MarpaX::Simple::Lexer. It has a recognize() method that calls $rec->end_input after it recognized all tokens in the buffer. It would be nice if we could call 'end_input' ourselves when we don't receive more input. The change is small, but the interface is really different. Maybe it would be possible to add another method that doesn't call the end_input method.
And now for the example. I was working on a simple Lisp-like language. And I thought it would be fun to add a REPL (read-eval-print-loop) where you can write and evaluate expressions. One of the nice things is of a REPL is that you can write a partial expression, press enter, and write some more on the next line. I'll show an example:
> (label f (lambda (x y)<ENTER>
: (+ x y)))<ENTER>
#<code>
>
The parser knows that the expression isn't complete after the first <ENTER> and we print the ":" instead of the ">". The Marpa recognizer supports this. We call read(...) a few times and check if the parser thinks it's done. If it is we can eval the parse tree, if it's not we can call read a few more times with the tokens from the next lines. Currently this doesn't work with MarpaX::Simple::Lexer because after the first call to 'recognize' it calls end_input.
--
Peter
I will start with my point and give an example below. It would be nice to have another interface to MarpaX::Simple::Lexer. It has a recognize() method that calls $rec->end_input after it recognized all tokens in the buffer. It would be nice if we could call 'end_input' ourselves when we don't receive more input. The change is small, but the interface is really different. Maybe it would be possible to add another method that doesn't call the end_input method.
And now for the example. I was working on a simple Lisp-like language. And I thought it would be fun to add a REPL (read-eval-print-loop) where you can write and evaluate expressions. One of the nice things is of a REPL is that you can write a partial expression, press enter, and write some more on the next line. I'll show an example:
> (label f (lambda (x y)<ENTER>
: (+ x y)))<ENTER>
#<code>
>
The parser knows that the expression isn't complete after the first <ENTER> and we print the ":" instead of the ">". The Marpa recognizer supports this. We call read(...) a few times and check if the parser thinks it's done. If it is we can eval the parse tree, if it's not we can call read a few more times with the tokens from the next lines. Currently this doesn't work with MarpaX::Simple::Lexer because after the first call to 'recognize' it calls end_input.
--
Peter
Ruslan Zakirov
Apr 13, 2012, 11:58:51 AM4/13/12
to marpa-...@googlegroups.com
Hi,
I'm ok with this. It was on list of things to think about before CPAN
release. Would send a pull request with docs update?
How are you going check that you're still in the middle of a statement?
> --
> Peter
--
Best regards, Ruslan.
Peter Stuifzand
Apr 13, 2012, 12:16:17 PM4/13/12
to marpa-...@googlegroups.com
I have a grammar that will parse one expression. That way it knows what one expression is and when it's finished with it. While feeding the Lexer I check the exhausted() method of the Recognizer before I prompt and read more from the input.
--
Peter
--
Peter
Ruslan Zakirov
Apr 13, 2012, 4:00:38 PM4/13/12
to marpa-...@googlegroups.com
On Fri, Apr 13, 2012 at 20:16, Peter Stuifzand
<peter.s...@gmail.com> wrote:
> I have a grammar that will parse one expression. That way it knows what one
> expression is and when it's finished with it. While feeding the Lexer I
> check the exhausted() method of the Recognizer before I prompt and read more
> from the input.
<peter.s...@gmail.com> wrote:
> I have a grammar that will parse one expression. That way it knows what one
> expression is and when it's finished with it. While feeding the Lexer I
> check the exhausted() method of the Recognizer before I prompt and read more
> from the input.
Right.
--
Best regards, Ruslan.
Peter Stuifzand
Apr 14, 2012, 6:46:20 AM4/14/12
to marpa-...@googlegroups.com
I posted a pull request on GitHub. Maybe my patch is a bit naive, but it works in the example I've given above.
I really like this feature, so let me know what you think.
--
Peter
On Friday, April 13, 2012 10:00:38 PM UTC+2, Ruslan Zakirov wrote:
I really like this feature, so let me know what you think.
--
Peter
On Friday, April 13, 2012 10:00:38 PM UTC+2, Ruslan Zakirov wrote:
Reply all
Reply to author
Forward
0 new messages