negative floats and unary minus

[arsf10]

i am having trouble specifying a grammar which can parse negative
integers as negative integer tokens rather than a unary minus
operation in front of a positive integer. i've tried a couple of
things but i end up creating a parser that either doesnt compile or
confuses the subtraction operator with the unary minus operation and
so forth.

for example,

i'd like to have

1. f(-1) parse as f acting on the token -1
2. -1+1 parse as addition of the token -1 and 1
3. -1^1 parse as unary negation times 1^1 (where ^ is exponentiation)
4. 1^-1 to give a parser error
5. 1-1 to parse as the token 1, the operation 'subtraction', and the
token 1
6(optional or parser error). 1--1 to parse as the token 1, the
operation 'subtraction', and the token -1.

i have looked through example grammars and posts but i've never seen
the '-' operation parsed this way.

thanks for any help. i'm pretty obtuse about theory and whether the
following grammar is LL or possible in javacc, so any references to
basic theory would also be appreciated.

[Eric Nickell]

You need to keep in mind the work being done by your tokenizer vs the
work being done by the parser.

Your question mostly concerns the tokenizer, which, in general, finds the
longest string of characters starting at the current position in the
input stream which matches one of your token definitions. I'll use "*" to
mark the position of concern in each of your examples:

1. f(*-1) parse as f acting on the token -1
2. *-1+1 parse as addition of the token -1 and 1
3. *-1^1 parse as unary negation times 1^1 (where ^ is exponentiation)
4. 1^*-1 to give a parser error
5. 1*-1 to parse as the token 1, the operation 'subtraction', and the
token 1
6(optional or parser error). 1-*-1 to parse as the token 1, the

operation 'subtraction', and the token -1.

It's easy to create a parser which finds the token "-1" in these
examples, but you say that in example 5, you want it to match the token
"-" instead.

To do this, you may need to use lexical states so that finding a negative
number is only permitted in certain states. You can try to drive the
state-changing machinery either from the parser (hard in general), or
from the tokenizer (e.g., after finding an integer, change temporarily to
a state which cannot tokenize an integer).

An alternative hack might be to extend your grammar so that
expression ::= expression <NEGATIVE_INTEGER>
was legal everywhere
expression ::= expression "-" expression
was.

Food for thought.

[arsf10]

eric,

from your comments (and sreeni's), i understand that i should be handling
the context of the '-' in the parser and not try to magically have javacc
interpret the '-' as a token in 3 different ways, which is impossible. i
separated the problem into the lexical states of an unambiguous negative
integer, a subtraction operation, and a unary minus operation in the
parser section and i can parse my cases correctly. the function of the two
different parts became obvious when i set both DEBUG_PARSER and
DEBUG_TOKEN_MANAGER to true. i should probably read the documentation
again.

thanks.

[Theodore Norvell]

It's not so much that it is impossible to interpret the - sign in
more than one way in the lexer. You can do that with lexical states.
The problem is that which lexical state the tokenizer is in is
determined by the characters to the left of the current character.
In your case you want
-1^1 to tokenize as -, 1, ^, and 1
but
-1+1 to tokenize as -1, +, and 1
So how to tokenize the "-1" depends on the characters to its right.
I don't know of any way to do that in JavaCC. (You can do it in
lex and presumably JLex.) Maybe it can be done by putting characters
back on the character stream.

The trouble with handling everything at the parsing level is
that you can't distinguish between "- 1" and "-1" (assuming
space is skipped).

Is this a language you are designing, or one that has a design that
it is not too late to influence. If so, you might want to push
for a simpler language design. For example C, C++, and Java do not
have negative literals at all.

Cheers,
Theodore Norvell