Why doesn't this work

[matio]

I have this code:
#
-----------------------------------------------------------------------------
# calc.py
#
# A simple calculator with variables. This is from O'Reilly's
# "Lex and Yacc", p. 63.
#
-----------------------------------------------------------------------------

import sys
sys.path.insert(0,"../..")

if sys.version_info[0] >= 3:
raw_input = input

tokens = (
'NAME','NUMBER','STRING', 'STRDEF'
)

literals = ['=','+','-','*','%','/', '(',')']

# Tokens

t_NAME = r'[a-zA-Z_][a-zA-Z0-9_]*'
t_STRING = r'"[a-zA-Z0-9]*"'
t_STRDEF = 'string'

def t_NUMBER(t):
r'\d+'
t.value = int(t.value)
return t

t_ignore = " \t"

def t_newline(t):
r'\n+'
t.lexer.lineno += t.value.count("\n")

def t_error(t):
print("Illegal character '%s'" % t.value[0])
t.lexer.skip(1)

# Build the lexer
import ply.lex as lex
lex.lex()

# Parsing rules

precedence = (
('left','+','-'),
('left','*','/'),
('right','UMINUS'),
)

# dictionary of names
names = { }

def p_statement_assign(p):
'''statement : NAME "=" expression
| STRDEF NAME "=" STRING'''
if p[2] == "=": names[p[1]] = p[3]
if p[3] == "=": names[p[2]] = p[4]



def p_statement_expr(p):
'statement : expression'
print(p[1])

def p_expression_binop(p):
'''expression : expression '+' expression
| expression '-' expression
| expression '*' expression
| expression '/' expression
| expression '%' expression'''

if p[2] == '+' : p[0] = p[1] + p[3]
elif p[2] == '-': p[0] = p[1] - p[3]
elif p[2] == '*': p[0] = p[1] * p[3]
elif p[2] == '/': p[0] = p[1] / p[3]
elif p[2] == '%': p[0] = p[1] % p[3]

def p_expression_uminus(p):
"expression : '-' expression %prec UMINUS"
p[0] = -p[2]

def p_expression_group(p):
"expression : '(' expression ')'"
p[0] = p[2]

def p_expression_number(p):
"expression : NUMBER"
p[0] = p[1]

def p_expression_name(p):
"expression : NAME"
try:
p[0] = names[p[1]]
except LookupError:
print("Undefined name '%s'" % p[1])
p[0] = 0

def p_error(p):
if p:
print("Syntax error at '%s'" % p.value)
else:
print("Syntax error at EOF")

import ply.yacc as yacc
yacc.yacc()

while 1:
try:
s = raw_input('calc > ')
except EOFError:
break
if not s: continue
yacc.parse(s)

when I do something like:
string a = "h"

it says:
Syntax error at 'a'

why doesn't it work?

[Hendriks, D.]

Hello matio,

> string a = "h"

This is not valid Python syntax. You don't declare variables by type. Instead just assign like this:

a = "h"

Hope this helps.

Best,
Dennis Hendriks

________________________________________
Van: ply-...@googlegroups.com [ply-...@googlegroups.com] namens matio [ma...@hotmail.co.uk]
Verzonden: zondag 11 oktober 2009 10:09
Aan: ply-hack
Onderwerp: Why doesn't this work

[Bruce Frederiksen]

I'm going to go with specifying t_STRING after t_NAME, so that the word "string" is taken as a NAME.  Try putting t_STRING before t_NAME...

-Bruce

[David Beazley]

On Oct 11, 2009, at 7:55 AM, Bruce Frederiksen wrote:

> I'm going to go with specifying t_STRING after t_NAME, so that the
> word "string" is taken as a NAME. Try putting t_STRING before
> t_NAME...
>

[snip]

> # Tokens
>
> t_NAME = r'[a-zA-Z_][a-zA-Z0-9_]*'
> t_STRING = r'"[a-zA-Z0-9]*"'
> t_STRDEF = 'string'

I'm going to agree with this. The t_STRDEF token is being matched by
the t_NAME rule above it which is almost certainly causing this
problem. The original poster should look at http://www.dabeaz.com/ply/ply.html#ply_nn6
and note the part about "reserved words."

This problem comes up so often, I'm almost wondering if I should add
an extra validation check to PLY. It occurs to me that I could
modify lex to look for regexes that are made up of nothing but simple
characters (e.g., 'string' above) and then test them for matches
against the other regular expressions (e.g., t_NAME). If a match is
found, some kind of stern warning message could be issued about it.

As the documentation states, it's really a bad idea to make separate
token rules for reserved words like 'string', 'if', 'else', etc.

Cheers,
Dave

[matio]

"t.type = reserved.get(t.value,'ID')"
what is the 'ID' bit?
Thanks for the help

On Oct 11, 2:11 pm, David Beazley <d...@dabeaz.com> wrote:
> On Oct 11, 2009, at 7:55 AM, Bruce Frederiksen wrote:
>
> > I'm going to go with specifying t_STRING after t_NAME, so that the  
> > word "string" is taken as a NAME.  Try putting t_STRING before  
> > t_NAME...
>
> [snip]
>
> > # Tokens
>
> > t_NAME    = r'[a-zA-Z_][a-zA-Z0-9_]*'
> > t_STRING  = r'"[a-zA-Z0-9]*"'
> > t_STRDEF  = 'string'
>
> I'm going to agree with this.  The t_STRDEF token is being matched by  
> the t_NAME rule above it which is almost certainly causing this  

> problem.   The original poster should look athttp://www.dabeaz.com/ply/ply.html#ply_nn6

[David Beazley]

ID is the default token type to use if get() can't find the name in
the dictionary. In your case, it would be 'NAME'.

-Dave

[matio]

Thanks, when this:

# calc.py
#
# A simple calculator with variables. This is from O'Reilly's
# "Lex and Yacc", p. 63.
#

import sys
sys.path.insert(0,"../..")

if sys.version_info[0] >= 3:
raw_input = input

reserved = {
'string':'STRDEF'
}

tokens = [
'NUMBER','STRING', 'NAME'
] + list(reserved.values())

literals = ['=','+','-','*','%','/', '(',')']

# Tokens

t_STRING = r'"[a-zA-Z0-9]*"'

#t_NAME = r'[a-zA-Z_][a-zA-Z0-9_]*'

def t_NUMBER(t):
r'\d+'
t.value = int(t.value)
return t

def t_NAME(t):
r'[a-zA-Z_][a-zA-Z0-9_]*'
t.value = reserved.get(t.value, 'NAME')

return t

t_ignore = " \t"

def t_newline(t):
r'\n+'
t.lexer.lineno += t.value.count("\n")

def t_error(t):
print("Illegal character '%s'" % t.value[0])
t.lexer.skip(1)

# Build the lexer
import ply.lex as lex
lex.lex()

# Parsing rules

precedence = (
('left','+','-'),
('left','*','/'),
('right','UMINUS'),
)

# dictionary of names
names = { }

def p_statement_assign(p):

'''statement : STRDEF NAME "=" STRING
| NAME "=" expression'''

is run it says:

Syntax error at 'NAME'

[matio]

BTW sorry for the rubbish title :)

[Andrew Dalke]

On Oct 11, 2009, at 7:12 PM, matio wrote:
> def t_NAME(t):
> r'[a-zA-Z_][a-zA-Z0-9_]*'
> t.value = reserved.get(t.value, 'NAME')
> return t

...

> is run it says:
>
> Syntax error at 'NAME'

That's because you changed all NAME tokens to have the value of NAME.
You really want to change the type. Change "t.value" to "t.type" so
the assignment becomes

t.type = reserved.get(t.value, 'NAME')

Andrew
da...@dalkescientific.com

[matio]

thanks, one last question, how would you print out a string containing
characters like "\n"?

[matio]

Also how would you use code like this:

def t_ccode_lbrace(t):
r'\{'
t.lexer.level +=1

def t_ccode_rbrace(t):
r'\}'
t.lexer.level -=1


to implement an 'if', do you have to use 'ast' like in GardenSnake

[A.T.Hofkamp]

matio wrote:
> Also how would you use code like this:
>
> def t_ccode_lbrace(t):
> r'\{'
> t.lexer.level +=1
>
> def t_ccode_rbrace(t):
> r'\}'
> t.lexer.level -=1

It is for counting curly brace levels.

> to implement an 'if', do you have to use 'ast' like in GardenSnake

Please do not assume we know anything but PLY.

I have no idea what you mean with the above question. This is a PLY mailing
list, not a GardenSnake mailing list. Also, I fail to see any relation between
the lexer functions and your question.

(The lexer functions of your question are nearly the first thing in a parser,
and an ast is about the last thing in the parser. In between are at least 2 or
3 other steps that you don't mention at all. Therefore, I cannot make a
relation between the code and your question.)

In general, if you want useful answers, plz give a concrete coherent concise
problem description, instead of a random piece of code and a (in my view)
unrelated random question without any context. Also show what you have done so
far with respect to that problem.
In addition, a title "Why doesn't this work" is too generic. Please use a more
specific title, and change the title of the post when you change the subject.
Many readers will decide whether or not read a post based on the title alone.

From your previous posts I believe that you have little Python experience.
That is not a problem in itself, but asking basic Python questions here is not
the most effective way of learning the language. Instead, I'd like to
recommend you read a few tutorials, and/or subscribe the python-tutor mailing
list.

To a lesser extent, that also holds for asking basic parsing questions here.
Your question above seems to indicate (to me) a lack of understanding basic
parsing terminology. Please make sure you understand what those terms mean and
how they are related when using them, otherwise they will only confuse the
discussion.

Sincerely,
Albert

[matio]

GardenSnake IS bundled as an example & i've already been asked to look
at it from this group

[matio]

Sorry, I didn't know you could change the title

On Oct 15, 8:37 am, "A.T.Hofkamp" <a.t.hofk...@tue.nl> wrote:

[A.T.Hofkamp]

matio wrote:
> GardenSnake IS bundled as an example & i've already been asked to look
> at it from this group

Ok, sorry for the confusion.

It just shows I also don't read everything of the list, and/or know everything
about PLY.

Albert

[Andrew Dalke]

On Oct 14, 2009, at 4:42 PM, matio wrote:
> to implement an 'if', do you have to use 'ast' like in GardenSnake

GardenSnake was something I wrote years ago. It uses the ast because
GardenSnake ends up generating Python byte code and I could leverage
existing Python data structures and byte code generation to help with
that.

Implementing an 'if' can be done in many ways, so there's no way to
answer than without a lot more context.

Based on your postings, you still have a lot to go before you
understand PLY and how it works, You posted a snippet of code (with
t_ccode_lbrace and t_ccode_rbrace) and asked what it does. It comes
from the PLY documentation, and with a description in the paragraph
before the code.

But to understand that description you have to know why lexers (being
regular grammars) can't on their own handle balanced braces. And to
get that requires some theory background that can be hard for people
to pick up on their own.

I cannot help you there. I learned this through university course
work and a lot of work on my own. There are books on the topic, but
then you'll have to learn how to convert from the syntax used in the
book to using PLY, which isn't always direct. Nor do I have advice on
which books to even look at to get a better understanding.

Best regards,

Andrew
da...@dalkescientific.com

[David Beazley]

>
> I cannot help you there. I learned this through university course
> work and a lot of work on my own. There are books on the topic, but
> then you'll have to learn how to convert from the syntax used in the
> book to using PLY, which isn't always direct. Nor do I have advice on
> which books to even look at to get a better understanding.
>

I would agree. I originally developed PLY so I could use it in a
university course I was teaching on compilers--that was a graduate-
level computer science course. I don't think you need to suffer
through the infamous Dragon book to be able to use PLY. However,
you'll definitely want to get some supporting texts. The O'Reilly
"Lex & Yacc" book isn't too bad as a reference. Almost everything in
that book applies to PLY and can be used as a source of ideas. Other
than that, I don't really have many
recommendations for a modern compilers text.

Cheers,
Dave

[Oldrich Jedlicka]

I can recommend a book called Parsing Techniques - A Practical Guide. It
describes grammars and parsing and is very useful if you want to learn what
is under the hood (and why your grammar doesn't work sometimes). It is freely
available at http://www.cs.vu.nl/~dick/PTAPG.html (check the Availability
section for direct download links).

Best regards,
Oldrich.

> Best regards,
>
> Andrew
> da...@dalkescientific.com
>
>
>
>