LEPL and Line Numbers

[astromme]

Hi there,

I'm working on getting line numbers put into my AST for further processing, but I seem to have a case where the matching fails when I try and add line number information. Consider this example where V1 works and V2 doesn't. Any ideas?

#!/usr/bin/env python

import re

from lepl import *

class Block(List): pass

def with_line(node):

    def wrapper(results, stream_in, stream_out):

        return node([results, s_delta(stream_in)[1], s_delta(stream_out)[1]])

    return wrapper

identifier = Token('[a-zA-Z][a-zA-Z0-9_]*') > List

symbol = Token('[^0-9a-zA-Z \t\r\n]')

block = (~symbol('{') & (identifier | symbol)[0:] ** with_line(Block) & ~symbol('}')) # V1

block = (~symbol('{') & (identifier | symbol)[0:] & ~symbol('}')) ** with_line(Block) # V2

parser = block.get_parse()

print parser('{\n Andrew \n}')[0]

Andrew

[andrew cooke]

Hi,

I haven't looked at this in detail, but if you look at my previous answer at https://groups.google.com/d/msg/lepl/0nsdsTH_bXI/rG5PR57sk1wJ you'll see that the function I use has a "*kargs" because results, stream_in and stream_out aren't the only parameters available.  Since you don't have that, that code is going to fail.

Andrew

[Andrew Stromme]

Hi,

I realize my previous message was a little cryptic "it don't work" without any explanation. Thanks for the *kargs hint, but I think I still have something that I'm not understanding. Given the input "{\n Andrew \n}" with V1 I get the valid parse 

Block

 +- list

 |   `- List

 |       `- 'Andrew'

 +- 2

 `- 3

however when I use V2 I get a matcher fail

lepl.stream.maxdepth.FullFirstMatchException: The match failed in <string> at '' (line 3, character 2).

I'm finding it hard to figure out how adding a post-match function is changing what gets matched? I'm using parens to keep my ordering correct, I think.

Thanks,

Andrew

Andrew

--
You received this message because you are subscribed to the Google Groups "lepl" group.

To view this discussion on the web visit https://groups.google.com/d/msg/lepl/-/qhLlkko94bIJ.

To post to this group, send email to le...@googlegroups.com.
To unsubscribe from this group, send email to lepl+uns...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/lepl?hl=en.

[andrew cooke]

ok, but if i understood right then your code had a bug, which i explained. do
you have a version that doesn't have the bug? or is that not a bug? because
it seems pointless trying to make code work right if you know it isn't going
to work. maybe the bug is somehow making V1 work when it should "really"
fail... i don't know, but it seems crazy to not fix the bug and try again.
if you have tried again, and it still doesn't work, can you post the new code?
thanks, andrew

[Andrew Stromme]

I'm pretty sure that the only things that KArgs can apply are results, stream_in and stream_out. See around line 252 in matchers/derived.py

    def fun(stream_in, matcher):

        '''

        Apply args as **kargs.

        '''

        (results, stream_out) = matcher()

        kargs = {'results': results,

                 'stream_in': stream_in,

                 'stream_out': stream_out}

        if raw:

            return function(**kargs)

        else:

            return ([function(**kargs)], stream_out)

regardless, with or without a *kargs param the code functions/breaks in the same way.

Andrew

[andrew cooke]

OK (I think that may be a bug - the new stream handling was supposed to expand
the available set of values).

Anyway, I ran your code, and what is happening in the second case is that you
are trying to find the position of an exhausted stream. I realise that it
should do something more useful, like return one past the end of the file, or
-1 or something (it might do so if you're not using tokens), and I'll fix it
for the next release (which is still a lot of work away). But for now a
simple fix would be:

class Block(List): pass

def with_line(node):
def wrapper(results, stream_in, stream_out):

print('inside')
a = s_delta(stream_in)[1]
try:
b = s_delta(stream_out)[1]
except StopIteration:
b = 'eof'
return node([results, a, b])
return wrapper

identifier = Token('[a-zA-Z][a-zA-Z0-9_]*') > List
symbol = Token('[^0-9a-zA-Z \t\r\n]')

#block = (~symbol('{') & (identifier | symbol)[0:] ** with_line(Block) &

~symbol('}')) # V1
block = (~symbol('{') & (identifier | symbol)[0:] & ~symbol('}')) **
with_line(Block) # V2

parser = block.get_parse()

print(parser('{\n Andrew \n}')[0])

cheers,
andrew

> > > > at https://groups.google.com/d/msg/lepl/0nsdsTH_bXI/rG5PR57sk1wJyou'll

[Andrew Stromme]

That works great, thanks a lot.

Andrew