Matching tokens
Malthe Borch
succesfully tokenized an input using the lexer into *objects* such as:
result = [Word("abc"), Word("def"), Number(1235)]
This would be from a string input such as: "abc def 1235"
Is there anything in Lepl which can operate on such objects instead of
characters? E.g.
matcher = Word[1:] & Optional(Number)
matcher.parse(result)
This is probably pretty far from the right usage, but I hope it's
reasonably clear.
\malthe
andrew cooke
Your model of how Lepl works isn't exactly right (objects aren't created -
it's mor elike tagged tuples), but your example code is more or less right.
Does something like that not work?
w = Token(Word())
n = Token(Number())
matcher = w[1:] & Optional(n)
I haven't tried it (am at work!) but that kind of thing should be possible.
Andrew
> --
> You received this message because you are subscribed to the Google Groups "lepl" group.
> To post to this group, send email to le...@googlegroups.com.
> To unsubscribe from this group, send email to lepl+uns...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/lepl?hl=en.
>
Malthe Borch
>
> Your model of how Lepl works isn't exactly right (objects aren't created -
> it's mor elike tagged tuples), but your example code is more or less right.
> Does something like that not work?
>
> w = Token(Word())
> n = Token(Number())
> matcher = w[1:] & Optional(n)
>
> I haven't tried it (am at work!) but that kind of thing should be possible.
In my example, ``Word`` and ``Number`` are like part of the alphabet.
The matcher would no longer care about characters, but instead these
other primitives.
It's mentioned in ./lepl/regexp/core.py that:
"It can use a variety of alphabets -
it is not restricted to strings. It could,
for example, match lists of integers,
or sequences of tokens."
That's exactly what I want to do –– I want to match sequences of
opaque objects that match with ``isinstance``, object equality or
similar assertion.
I know that the picoparse library can work in this way, i.e. expect
any sort of sequential input and match based on equality.
Looking through the alphabet code, it seems like I have to equip my
objects with an ordering and ranges, both of which don't apply. I
think what I'm looking for might be called "monads", but it also might
not.
Thanks,
\malthe
andrew cooke
Interesting.
Unfortunately, that's now how the lexer works. I need to think a lot more
before I can say whether it could work like that. It's possible that would be
better than the current design.
Separately, from that, however, it is true that Lepl will work with "arbitray"
objects, however, you do need to define an alphabet, and you would also need
to define an "isinstance" matcher (this is trivial). I don't think defining
an alphabet would be too hard - you say that ordering "doesn't apply", but
that doesn't mean you can't just add one anyway. - it doesn't have to "mean"
anything.
I don't know anything about picoparse. I will go look at it soon(ish).
Unfortunately, life is very busy right now, and I what you propose to do,
while it may work, is not something anyone has done before. I'll try support
you if you want to keep going, but it's only fair to say that my time is
limited at the moment.
Also, as this all starts to sink in, I am not sure what you are suggesting is
any better than the way Lelp works. It's possible that you are simply taking
how to do thigns with one tool and applying that to another, instead of seeing
how the other tool can solve the same problems using slightly different
concepts. I hope that doesn't sound rude!
Andrew
andrew cooke
Actually, I'm not sure this is correct. I think you may not even need an
alphabet. There isn't an alphabet for integers, as far as I know. The
alphabet is only used for regexp support. So if you don't use regexps then
Lepl should simply parse a list of objects like it would parse anything else.
To make a matcher that matches an item using isinstance you'd want:
@function_matcher_factory
def Instance(type_):
def matcher(support, stream):
if stream and isinstance(stream[0], type_):
return (stream[0], stream[1:])
Andrew
Malthe Borch
from lepl import function_matcher_factory
@function_matcher_factory()
def Instance(type_):
def matcher(support, stream):
if stream and isinstance(stream[0], type_):
return (stream[0], stream[1:])
return matcher
class Base(object):
def __radd__(self, other):
return other + [self]
class A(Base): pass
... etc ...
To support my use-case of being able to report on the next expected
element, I use the following (sample) error class:
class Error(object):
def __init__(self, name):
self.name = name
def __call__(self, stream_in=None, stream_out=None, results=None):
result = self.name, tuple(stream_in), tuple(stream_out)
del stream_out[:]
return result
For the record, my test matcher:
matcher = Instance(A) & \
Optional(Instance(C) & (Instance(D) % (Any() ** Error("D")))) & \
Optional(Instance(C) & (Instance(B) % (Any() ** Error("B"))))
matcher.config.no_full_first_match()
If one of my results are clear of any error instances, then it's a
match; otherwise, I'll report on the deepest match from my selection
of error instances.
\malthe
--
Malthe Borch
Technical Advisor
UNICEF Uganda
+256 (0) 703 945 965
andrew cooke
> Works like a charm;
Awesome :o)
> I made these adjustments:
>
> from lepl import function_matcher_factory
>
> @function_matcher_factory()
> def Instance(type_):
> def matcher(support, stream):
> if stream and isinstance(stream[0], type_):
> return (stream[0], stream[1:])
> return matcher
>
> class Base(object):
> def __radd__(self, other):
> return other + [self]
Not sure what the __radd__ is for here, but if you're trying to duplicate the
behaviour of Lepl's matcher operators then OperatorMixin
http://www.acooke.org/lepl/api/lepl.matchers.operators.OperatorMixin-class.html
or OperatorMatcher
http://www.acooke.org/lepl/api/lepl.matchers.support.OperatorMatcher-class.html
might be useful.
> class A(Base): pass
> ... etc ...
>
> To support my use-case of being able to report on the next expected
> element, I use the following (sample) error class:
>
> class Error(object):
> def __init__(self, name):
> self.name = name
>
> def __call__(self, stream_in=None, stream_out=None, results=None):
> result = self.name, tuple(stream_in), tuple(stream_out)
> del stream_out[:]
> return result
>
> For the record, my test matcher:
>
> matcher = Instance(A) & \
> Optional(Instance(C) & (Instance(D) % (Any() ** Error("D")))) & \
> Optional(Instance(C) & (Instance(B) % (Any() ** Error("B"))))
>
> matcher.config.no_full_first_match()
>
> If one of my results are clear of any error instances, then it's a
> match; otherwise, I'll report on the deepest match from my selection
> of error instances.
I'm glad % works...
Cheers,
Andrew
andrew cooke
Thinking some more, your __radd__ may indicate that I have a bug. I'll add
this as a test and see if there's a problem.
Thanks,
Andrew