retrieving tokens in parsing rules
6 views
Skip to first unread message
dhendriks
May 8, 2008, 6:47:52 AM5/8/08
to ply-hack
In this parsing rule:
def p_abc(self, p):
' abc : DEF '
p[0] = p[1]
p[1] returns the .value of the token. One can get the linenumber with
p.lineno(1). It is possible to get the lexpos and other information as
well. However, it seems impossible to get the actual token. Well, it's
possible, by using p.splice[1], but splice is an 'internal' variable.
Therefore, it may be nice to add a .lextoken(n) method to
YaccProduction. This way it is possible to get the actual LexToken
instance. I don't really know what it should return on non-terminals.
It suppose it could return the YaccSymbol instances, return something
else (like None) or it could raise an exception.
def p_abc(self, p):
' abc : DEF '
p[0] = p[1]
p[1] returns the .value of the token. One can get the linenumber with
p.lineno(1). It is possible to get the lexpos and other information as
well. However, it seems impossible to get the actual token. Well, it's
possible, by using p.splice[1], but splice is an 'internal' variable.
Therefore, it may be nice to add a .lextoken(n) method to
YaccProduction. This way it is possible to get the actual LexToken
instance. I don't really know what it should return on non-terminals.
It suppose it could return the YaccSymbol instances, return something
else (like None) or it could raise an exception.
David Beazley
May 8, 2008, 8:26:47 AM5/8/08
to ply-hack, dhendriks
The token objects used in a lex/yacc are really meant to be an internal implementation detail---not something that you would ever directly work
with. As such, the .value attribute of a symbol (which is returned by p[1], p[2], etc.) is really the only safe thing to use.. I suppose it would be
easy to add a method that returns the token, but I'm not sure I would advise anyone to use it for anything other than debugging. One reason I
would be cautious about this is that I'm planning to make some pretty big implementation changes in a future PLY version. The parser interface
will stay the same, but the underlying representation of tokens/symbols is going to change.
with. As such, the .value attribute of a symbol (which is returned by p[1], p[2], etc.) is really the only safe thing to use.. I suppose it would be
easy to add a method that returns the token, but I'm not sure I would advise anyone to use it for anything other than debugging. One reason I
would be cautious about this is that I'm planning to make some pretty big implementation changes in a future PLY version. The parser interface
will stay the same, but the underlying representation of tokens/symbols is going to change.
Cheers,
Dave
On Thu 08/05/08 6:47 AM , dhendriks d.hen...@tue.nl sent:
dhendriks
May 9, 2008, 7:23:25 AM5/9/08
to ply-hack
On May 8, 2:26 pm, David Beazley <d...@dabeaz.com> wrote:
> The token objects used in a lex/yacc are really meant to be an internal implementation detail---not something that you would ever directly work
> with.
OK, I won't use LexToken instances then.
> The token objects used in a lex/yacc are really meant to be an internal implementation detail---not something that you would ever directly work
> with.
> As such, the .value attribute of a symbol (which is returned by p[1], p[2], etc.) is really the only safe thing to use..
> I suppose it would be
> easy to add a method that returns the token, but I'm not sure I would advise anyone to use it for anything other than debugging. One reason I
> would be cautious about this is that I'm planning to make some pretty big implementation changes in a future PLY version. The parser interface
> will stay the same, but the underlying representation of tokens/symbols is going to change.
another?
Reply all
Reply to author
Forward
0 new messages