Two very simple questions

[Thibaud]

Hi all!

I am beginning to use LEPL. It seems very promising but I have two problems. I guess the answer is easy or already written somewhere, but I searched for some time now and I did not found yet :(

Firstly, this very simple code:

with TraceVariables():

from lepl import *

PERCENT = Token('%')

TEXT = Token('[^\n]*')

comment = PERCENT & TEXT

print(comment.parse('% ThisIsAComment'))

outputs:

Traceback (most recent call last):

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/stream/maxdepth.py", line 55, in _matcher

    (result2, stream2) = yield generator

StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "<stdin>", line 6, in <module>

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/core/config.py", line 858, in parse

    return self.get_parse()(input_, **kargs)

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/core/parser.py", line 257, in single

    return next(raw(arg, **kargs))[0]

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/core/parser.py", line 161, in trampoline

    value = stack[-1].generator.throw(value)

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/stream/maxdepth.py", line 61, in _matcher

    raise FullFirstMatchException(stream1)

lepl.stream.maxdepth.FullFirstMatchException: The match failed in <string> at '% ThisIsAComment' (line 1, character 1).

I do not really see why it does not work... I did some tests with only '%' and it is correctly detected.

The second (and last) point ; this simple code:

test = Token(String())

test.parse('"irt"')

produces:

Traceback (most recent call last):

  File "<stdin>", line 3, in <module>

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/core/config.py", line 858, in parse

    return self.get_parse()(input_, **kargs)

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/core/config.py", line 815, in get_parse

    return make_single(self.get_match())

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/core/config.py", line 723, in get_match

    return self._raw_parser()

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/core/config.py", line 675, in _raw_parser

    make_raw_parser(self, stream_factory, config)

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/core/parser.py", line 220, in make_raw_parser

    matcher = rewriter(matcher)

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/lexer/rewriters.py", line 125, in __call__

    return self.lexer(graph, tokens, self.alphabet, self.discard)

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/lexer/lexer.py", line 74, in __init__

    token.compile(alphabet)

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/lexer/matchers.py", line 135, in compile

    self.regexp = self.__to_regexp(self.regexp, self.alphabet)

  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/LEPL-5.0.0-py3.2.egg/lepl/lexer/matchers.py", line 155, in __to_regexp

    'a regular expression: {0}', rewrite))

lepl.lexer.support.LexerError: A Token was specified with a matcher, but the matcher could not be converted to a regular expression: And(NfaRegexp, Transform, NfaRegexp)

It is as if the regexp behind String() was not recognized by LELP itself. Do I miss something?

Thank you very much if you can help me or give me some clues :)

[andrew cooke]

On Fri, Sep 09, 2011 at 02:43:51PM -0700, Thibaud wrote:
> Hi all!
>
> I am beginning to use LEPL. It seems very promising but I have two problems.
> I guess the answer is easy or already written somewhere, but I searched for
> some time now and I did not found yet :(
>
> Firstly, this very simple code:
>
> with TraceVariables():
> from lepl import *
> PERCENT = Token('%')
> TEXT = Token('[^\n]*')
> comment = PERCENT & TEXT
> print(comment.parse('% ThisIsAComment'))

hi! this is because tokens match as much as possible. so TEXT takes
everything and comment fails (because nothing was left for PERCENT).

here's one solution:

>>> with TraceVariables():
... PERCENT = Token('%')
... TEXT = Token('[^%\n]*')
... comment = PERCENT & TEXT
...
>>> comment.parse('% ThisIsAComment')
PERCENT = ['%'] stream = ' ThisIsAComment'
TEXT = [' ThisIsAComment'] stream = <EOS>
comment = ['%', ' ThisIsAComment'] stream = <EOS>
['%', ' ThisIsAComment']

but really it's much better to not use tokens at all:

>>> with TraceVariables():
... percent = Literal('%')
... text = AnyBut('\n')[:,...]
... comment= percent & text
...
>>> comment.parse('% ThisIsAComment')
percent = ['%'] stream = ' ThisIsAComment'
text = [' ThisIsAComment'] stream = ''
comment = ['%', ' ThisIsAComment'] stream = ''
['%', ' ThisIsAComment']

tokens are an "advanced" feature that you shouldn't use unless you need to.

> The second (and last) point ; this simple code:
>
> test = Token(String())
> test.parse('"irt"')

[...]

> lepl.lexer.support.LexerError: A Token was specified with a matcher, but the
> matcher could not be converted to a regular expression: And(NfaRegexp,
> Transform, NfaRegexp)
>
> It is as if the regexp behind String() was not recognized by LELP itself. Do
> I miss something?

again, this is because you're using tokens. tokens work with regular
expressions. not everything can be converted to a regular expression. for
example, in this case, lepl doesn't know how to convert the String() matcher.

if you want a token that matches a simple string you can use

>>> test = Token('"[^"]*"')
>>> test.parse('"irt"')
['"irt"']

but that's nothing like as flexible as String() (it's not as flexible because
it's a regular expression, and regular expressions are limited in what they
can do).

again, the simplest solution is to simply not use tokens:

>>> test = String()
>>> test.parse('"irt"')
['irt']

hope that helps,
andrew

[Thibaud]

Thank you very much for these explanations. It helped me a lot. Now you told me, I remembered this fact is stated in the tutorial, but I was unable to notice it was my problem :)