Simple::Lexer optionally shouldn't call end_input

Showing 1-5 of 5 messages
Simple::Lexer optionally shouldn't call end_input Peter Stuifzand 4/13/12 8:43 AM
Hi all,

I will start with my point and give an example below. It would be nice to have another interface to MarpaX::Simple::Lexer. It has a recognize() method that calls $rec->end_input after it recognized all tokens in the buffer. It would be nice if we could call 'end_input' ourselves when we don't receive more input. The change is small, but the interface is really different. Maybe it would be possible to add another method that doesn't call the end_input method.

And now for the example. I was working on a simple Lisp-like language. And I thought it would be fun to add a REPL (read-eval-print-loop) where you can write and evaluate expressions. One of the nice things is of a REPL is that you can write a partial expression, press enter, and write some more on the next line. I'll show an example:

 > (label f (lambda (x y)<ENTER>
 : (+ x y)))<ENTER>
 #<code>
 >

The parser knows that the expression isn't complete after the first <ENTER> and we print the ":" instead of the ">". The Marpa recognizer supports this. We call read(...) a few times and check if the parser thinks it's done. If it is we can eval the parse tree, if it's not we can call read a few more times with the tokens from the next lines. Currently this doesn't work with MarpaX::Simple::Lexer because after the first call to 'recognize' it calls end_input.

--
Peter
Re: Simple::Lexer optionally shouldn't call end_input Ruslan Zakirov 4/13/12 8:58 AM
Hi,

I'm ok with this. It was on list of things to think about before CPAN
release. Would send a pull request with docs update?

How are you going check that you're still in the middle of a statement?

> --
> Peter

--
Best regards, Ruslan.

Re: Simple::Lexer optionally shouldn't call end_input Peter Stuifzand 4/13/12 9:16 AM
I have a grammar that will parse one expression. That way it knows what one expression is and when it's finished with it. While feeding the Lexer I check the exhausted() method of the Recognizer before I prompt and read more from the input.

--
Peter
Re: Simple::Lexer optionally shouldn't call end_input Ruslan Zakirov 4/13/12 1:00 PM
On Fri, Apr 13, 2012 at 20:16, Peter Stuifzand
<peter.s...@gmail.com> wrote:
> I have a grammar that will parse one expression. That way it knows what one
> expression is and when it's finished with it. While feeding the Lexer I
> check the exhausted() method of the Recognizer before I prompt and read more
> from the input.

Right.

--
Best regards, Ruslan.

Re: Simple::Lexer optionally shouldn't call end_input Peter Stuifzand 4/14/12 3:46 AM
I posted a pull request on GitHub. Maybe my patch is a bit naive, but it works in the example I've given above.
I really like this feature, so let me know what you think.
--
Peter

On Friday, April 13, 2012 10:00:38 PM UTC+2, Ruslan Zakirov wrote:
On Fri, Apr 13, 2012 at 20:16, Peter Stuifzand wrote:
> I have a grammar that will parse one expression. That way it knows what one
> expression is and when it's finished with it. While feeding the Lexer I
> check the exhausted() method of the Recognizer before I prompt and read more
> from the input.

Right.


>
> --
> Peter


>
>
>
> On Fri, Apr 13, 2012 at 17:58, Ruslan Zakirov wrote:
>>
>> Hi,
>>

--
Best regards, Ruslan.