MarpaX::Simple::Lexer dies with "exhausted"

[Peter Stuifzand]

Hi,

I'm not sure why this testcase dies with the message "exhausted".
I created a simple testcase on gist.

My guess is that it $skips++ past the end of the buffer in this part and should call end_input. Can anyone confirm?


--
Peter Stuifzand
http://peterstuifzand.nl

[Ruslan Zakirov]

Hi,

I can reproduce, fixed and pushed. Please test.

Note that there are more changes in the repo then just this fix.

Fix is not complete as it leaves buffer with last token(s) and I'm not
sure yet how deal here.

In most cases it would be ok to skip last token in the buffer and
return. In this situation you
can continue parsing input as buffer is stored in lexer. This is
helpful if you have input with
repeated blocks you parse one by one.

If parser is ambiguos then several tokens may start in the same
position, end in different
positions and result in successful exhaustion. In this case you have
several "correct" parse
trees that end in different positions.

Most probably I will implement cutting buffer with the shortest match.

--
Best regards, Ruslan.

[Peter Stuifzand]

Thanks for the fix. It works.

I also added another simple case that doesn't work. In a .t file. See the pull request on GitHub.

--
Peter Stuifzand | peterstuifzand.nl | @pstuifzand