Lexer Eating Integer Constant

[Jason Ferguson]

First, I'll admit: this is probably one of the ugliest grammars you will ever see in your life (ironically named SimplePolicy.g). It STARTED as a simple grammar, but I ended up needing more detailed exceptions, so I ended up making multiple exceptions extending RecognitionException. Then a RuntimeException to wrap it so I could get out of the parser methods.

This grammar is supposed to validate a group of one or more IF/THEN/ELSE?/ENDIF statements. Unfortunately, when I try to compare one of the INTEGER_FIELD tokens to a INTEGER token, the INTEGER token has totally disappeared from the TokenInputStream. I've tried sprinkling println()s through to figure out what is going on, with no luck.

Example:

"IF AGE < 21 THEN DOB < TODAY ENDIF" is the string fed to the charstream, but somehow this ends up as "IF AGE < THEN DOB < TODAY ENDIF"

I apologize for the uglyness of this grammar, but if anyone can tell me why this dies I would be greatly appreciative.

grammar SimplePolicy;

options {
  language = Java;
  backtrack = true;
}


@header {
package com.manager.impl;

import com.manager.RecognitionRuntimeException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Arrays;


@lexer::header {
package com.manager.impl;

}

@lexer::members {

}

@parser::members {
 
 private static final Logger log = LoggerFactory.getLogger(SimplePolicyParser.class);
 
 @Override
 protected Object recoverFromMismatchedToken(IntStream input, int ttype, BitSet follow) throws RecognitionException {
   throw new MismatchedTokenException(ttype, input);
 }

  @Override
 public Object recoverFromMismatchedSet(IntStream input, RecognitionException e, BitSet follow) throws RecognitionException {
   throw e;
 }

 @Override
 public String getErrorMessage(RecognitionException e, String[] tokenNames) {
  // wrap in a runtime exception to escape ANTLR's dungeon
  throw new RecognitionRuntimeException(e);
 }

}

@rulecatch {
    catch (RecognitionException e) {
           System.out.println(getErrorHeader(e));
      //System.out.println(getErrorMessage(e,tokenNames));
        throw new RecognitionRuntimeException(e);
    }
}

// evaluate multiple show statements
policyGroupWithShow
  : show (show)* EOF
  | ifStatement+ EOF
  {
    // nope, this isn't legal
    ShowExpectedException ex = new ShowExpectedException();
    ex.line = 1;
    ex.charPositionInLine=0;
    throw ex;
  }
  ;
 
policyGroupWithoutShow
  : ifStatement (ifStatement)* EOF
  | show+ EOF
  {
    // not legal
    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = 1;
    ex.charPositionInLine = 0;
    throw ex;
  }
  ;
 
//policyGroup
//  : show (show)* EOF
//  ;


// evaluate a single SHOW statement
show
  //: ifStatement
  : SHOW STRING FOR ifStatement // good
  // missing for (FOR expected)
  | SHOW expr1a=STRING ifStatement // missing for
  {
    int nextTokenPosition = expr1a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(nextTokenPosition + 1);
   
    ShowWithoutForException ex = new ShowWithoutForException();
    ex.line = expr1a.getLine();
    ex.charPositionInLine = token.getCharPositionInLine();
    throw ex;

  }
  // missing show (SHOW expected)
  | expr1b=STRING FOR ifStatement
  {
    int tokenPosition = expr1b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition);
    ForWithoutShowException ex = new ForWithoutShowException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  ;


ifStatement
  @init {
    log.debug("ifStatement: " + input.toString());
  }
  //: IF logical THEN logical+ (ELSE logical+)? ENDIF
  : IF operation 'THEN' expression+ ENDIF // good
  {
    log.debug("Matched [IF logical THEN expression+ ENDIF]");
  }
  | IF logical THEN expression+ ELSE expression+ ENDIF
  {
    log.debug("Matched [IF logical THEN expression+ ELSE expression+ ENDIF]");
  }
  | logical expr1a=THEN expression* (ELSE expression*)? ENDIF // missing if
  {
    int tokenPosition = expr1a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition);
    MissingIfException ex = new MissingIfException();
    ex.line = token.getLine();
    ex.charPositionInLine = 0;
    throw ex;
  }
  // missing THEN (THEN clause is missing)
  | expr1b=IF logical expression+ (ELSE expression+)? ENDIF
  {
    int tokenPosition = expr1b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
    MissingThenClauseException ex = new MissingThenClauseException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getCharPositionInLine();
    throw ex;
  }
  // missing ELSE or ENDIF (ENDIF Expected)
  | IF logical expr1c=THEN expression+
  {
    String inputText = input.toString();
    if (inputText.indexOf("ELSE") < 0 && inputText.indexOf("ENDIF") < 0) {
      // best I can do is get the line/column from the THEN keyword
        MissingElseOrEndifException ex = new MissingElseOrEndifException();
        ex.line = expr1c.getLine();
        ex.charPositionInLine = expr1c.getCharPositionInLine();
        throw ex;    
    }
  }
  // missing comparison for IF (rule expected)
  | IF expr1d=THEN expression* ENDIF
  {
    int tokenPosition = expr1d.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition - 1);
   
    RuleExpectedException ex = new RuleExpectedException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStopIndex();
    throw ex;
  }
  // missing body of then (unexpected symbol or construction)
  | IF logical a=THEN b=ENDIF
  {
    int tokenPosition1 = a.getTokenIndex();
    int tokenPosition2 = b.getTokenIndex();
    CommonToken tokenA = (CommonToken) input.get(tokenPosition1);
    CommonToken tokenB = (CommonToken) input.get(tokenPosition2);
   
    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    //MissingThenBodyException ex = new MissingThenBodyException();
    if (tokenA.getLine() == tokenB.getLine()) {
      ex.line = tokenA.getLine();
      ex.charPositionInLine = tokenA.getCharPositionInLine();
    } else {
      ex.line = tokenA.getLine() + 1;
      ex.charPositionInLine = 0;
    }
    throw ex;
  }
  // missing body of ELSE (Unexpected symbol or construction)
  | IF logical THEN expression+ ELSE expr3e=ENDIF
  {

    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = expr3e.getLine();
    ex.charPositionInLine = expr3e.getCharPositionInLine();
    throw ex;
  }
  // body of IF missing (Body of IF clause is missing)
  | IF expr3f=ENDIF
  {
    int tokenPosition = expr3f.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition);
   
    MissingIfBodyException ex = new MissingIfBodyException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getCharPositionInLine();
    throw ex;
  }
  | expression // expression should just pass through
  ;

 
expression
  @init {
    log.debug("Expression: " + input.toString());
  }
  : logical
  ;

// deal with AND/OR operator
logical
  @init {
    log.debug("Logical:" + input.toString());
  }
 : operation (AND operation | OR operation)*
 ;

operation
  @init {
    log.debug("Operation:" + input.LT(1) + input.LT(2) + input.LT(3));
  }
  // good rules
  //: ';' ~('/r/n'|EOF)*
  : DATE_FIELD (EQ|NE|LT|LE|GT|GE) (DATE_FIELD|DATE|DATE_CONSTANT)
  {
   
    log.info("Matched STRING_FIELD (EQ|NE) (STRING_FIELD|STRING) field operation: " + input.toString());
  }
  | DATE_FIELD (PLUS|MINUS) (DATE_FIELD|DATE|DATE_CONSTANT|DATE_PERIOD_FIELD|datePeriod)
  {
    log.info("Matched DATE_FIELD (PLUS|MINUS) (DATE_FIELD|DATE|DATE_CONSTANT|DATE_PERIOD_FIELD|datePeriod) field operation: " + input.toString());
  }
  | DATE_PERIOD_FIELD (EQ|NE|LT|LE|GT|GE) (DATE_PERIOD_FIELD|DATE_PERIOD_CONSTANT)
  {
    log.info("Matched DATE_PERIOD_FIELD (EQ|NE|LT|LE|GT|GE) (DATE_PERIOD_FIELD|DATE_PERIOD_CONSTANT) field operation: " + input.toString());
  }
  | DATE_PERIOD_FIELD (PLUS|MINUS) (DATE_PERIOD_FIELD|datePeriod|DATE_FIELD|DATE_CONSTANT|DATE)
  {
    log.info("Matched DATE_PERIOD_FIELD (PLUS|MINUS) (DATE_PERIOD_FIELD|datePeriod|DATE_FIELD|DATE_CONSTANT|DATE) field operation: " + input.toString());
  }
  | STRING_FIELD (EQ|NE) (STRING_FIELD|STRING)
  {
  log.info("Matched STRING_FIELD (EQ|NE) (STRING_FIELD|STRING) field operation: " + input.toString());
  }
  | INTEGER_FIELD (EQ|NE|LT|LE|GT|GE) test=inAllFields
  {
    Boolean result = $inAllFields.result;
    Integer type = $inAllFields.tokenType;
   
    //int testType = test.getType();
    if (type != INTEGER && type != INTEGER_FIELD) {
   
        IncompatibleTypeException ex = new IncompatibleTypeException();
        //ex.line = test.getLine();
        //ex.charPositionInLine = test.getStartIndex();
        throw ex;
    }
  }
  | INTEGER_FIELD (EQ|NE|LT|LE|GT|GE) test1=(INTEGER_FIELD|INTEGER)
  {
    log.info(test1.getText());
    log.info("Matched INTEGER_FIELD (EQ|NE|LT|LE|GT|GE) (INTEGER_FIELD|INTEGER) field operation: " + input.toString());
  }
  | BOOLEAN_FIELD (EQ|NE) (BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
  log.info("Matched BOOLEAN_FIELD (EQ|NE) (BOOLEAN_FIELD|BOOLEAN_CONSTANT) field operation: " + input.toString());
  }
  | COMMENT
  {
  log.info("Matched COMMENT field operation: " + input.toString());
 
  }
  // specify bad rules
  // defined fields with no operation. The expression will be null if datePeriod isn't split off
  | (INTEGER_FIELD|INTEGER) (PLUS|MINUS|EQ|NE|LT|LE|)
  {
    System.out.println("Will it work?");
  }
  | (datePeriod)|e1=(DATE_PERIOD_FIELD|DATE_FIELD|STRING_FIELD|INTEGER_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    RelationalOperatorOrNotExpectedException ex = new RelationalOperatorOrNotExpectedException();
    if (e1 == null) {
        ex.line = 1;
        ex.charPositionInLine = 0;
    } else {
      ex.line = e1.getLine();
      ex.charPositionInLine = e1.getCharPositionInLine();
    }
    throw ex;
  }
  | e2=(INTEGER|DATE|DATE_CONSTANT)
  {
    RuleCannotBeginWithNumberOrDateException ex = new RuleCannotBeginWithNumberOrDateException();
    ex.line = e2.getLine();
    ex.charPositionInLine = e2.getCharPositionInLine();
    throw ex;
  }
  // attempt to compare a date field to a different field type. Incompatable Type exception
  | DATE_FIELD operator1a=(EQ|NE|LT|LE|GT|GE) (DATE_PERIOD_FIELD|datePeriod|STRING_FIELD|STRING|INTEGER_FIELD|INTEGER|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator1a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 2);
   
    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  // attempt to add/sub illegal field types to date (Relational Operator or NOT expected)
  | DATE_FIELD operator1b=(PLUS|MINUS)
      (STRING_FIELD|STRING|INTEGER_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator1b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition+1);
   
    RelationalOperatorOrNotExpectedException ex = new RelationalOperatorOrNotExpectedException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;  
  }
  // attempting to add an int to a date field makes it assume it's a bad date period
  | DATE_FIELD operator1c=(PLUS|MINUS) expr1c=(INTEGER)
  {
   int tokenPosition = operator1c.getTokenIndex();
   CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
   MissingYearsMonthsWeeksDaysException ex = new MissingYearsMonthsWeeksDaysException();
   ex.line = token.getLine();
   ex.charPositionInLine = token.getStartIndex();
   throw ex;
  }
  // attempt to compare a String field to something besides a String field or String constant
  | STRING_FIELD operator2a=(EQ|NE) ~(STRING_FIELD|STRING)
  {
    int tokenPosition = operator2a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | expr2b=STRING_FIELD operator2b=(LT|LE|GT|GE) (STRING_FIELD|STRING)
  {
    int tokenPosition = operator2b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | stringExpr3=STRING
  {
    RuleCannotBeginWithAStringException ex = new RuleCannotBeginWithAStringException();
    ex.line = stringExpr3.getLine();
    ex.charPositionInLine = stringExpr3.getCharPositionInLine();
    throw ex;
  }
  // attempt to compare a number field to something besides a num field or int constant

  | INTEGER_FIELD operator3a=(EQ|NE|LT|LE|GT|GE) expr3a2=(DATE_FIELD|DATE|STRING_FIELD|STRING|DATE_PERIOD_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    Boolean value = $inAllFields.result;
    //System.out.println("Expr text: " + expr3a2.getText());
    int tokenPosition = operator3a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  // attempt to perform a PLUS/MINUS on a number field with something besides a num field or constant int
  | INTEGER_FIELD operator3b=(PLUS|MINUS)
    (DATE_FIELD|DATE|STRING_FIELD|STRING|DATE_PERIOD_FIELD|datePeriod|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator3b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;  
  }
  // attempt to compare date period to something besides a date period field or date period constant
  | DATE_PERIOD_FIELD operator4a=(EQ|NE|LT|LE|GT|GE)
    (INTEGER_FIELD|INTEGER|DATE_FIELD|DATE|DATE_CONSTANT|STRING_FIELD|STRING|BOOLEAN_FIELD|BOOLEAN_CONSTANT|WAIVER_FIELD)
  {
    int tokenPosition = operator4a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | DATE_PERIOD_FIELD operator5a=(PLUS|MINUS) (STRING_FIELD|STRING|INTEGER_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator5a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    RelationalOperatorOrNotExpectedException ex = new RelationalOperatorOrNotExpectedException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | DATE_PERIOD_FIELD operator5b=(PLUS|MINUS) INTEGER
  {
    int tokenPosition = operator5b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    MissingYearsMonthsWeeksDaysException ex = new MissingYearsMonthsWeeksDaysException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | WAIVER_FIELD operator6a=(HAS|EQ|NE) ~(STRING)
  {
    int tokenPosition = operator6a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | WAIVER_FIELD operator6b=(PLUS|MINUS|LT|LE|GT|GE)  STRING
  {
    int tokenPosition = operator6b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  | BOOLEAN_FIELD operator7a=(EQ|NE) ~(BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator7a.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    IncompatibleTypeException ex = new IncompatibleTypeException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;  
  }
  | BOOLEAN_FIELD operator7b=(PLUS|MINUS|LT|LE|GT|GE) (DATE_FIELD|DATE|DATE_CONSTANT|DATE_PERIOD_FIELD|datePeriod|STRING_FIELD|STRING|INTEGER_FIELD|WAIVER_FIELD|BOOLEAN_FIELD|BOOLEAN_CONSTANT)
  {
    int tokenPosition = operator6b.getTokenIndex();
    CommonToken token = (CommonToken) input.get(tokenPosition + 1);
   
    UnexpectedSymbolOrConstructionException ex = new UnexpectedSymbolOrConstructionException();
    ex.line = token.getLine();
    ex.charPositionInLine = token.getStartIndex();
    throw ex;
  }
  ;

// this rule checks that a value is valid. It's not in the validation
// chain
atom
  : datePeriod
  | DATE_FIELD
  | INTEGER_FIELD
  | STRING_FIELD
  | DATE_PERIOD_FIELD
  | WAIVER_FIELD
  | BOOLEAN_FIELD
  | INTEGER
  {
    System.out.println("Matched atom INTEGER");
  }
  | DATE
  | DATE_CONSTANT
  | BOOLEAN_CONSTANT
  | STRING
  //| LPAREN expression RPAREN
  ;

datePeriod
  : (DATE_PERIOD_CONSTANT)+
  ;
 
// "subatomic" rules, meant to feed the atom rule understandable values



fragment DIGIT: ('0'..'9');

DATE_FIELD:('DOB'|'TEST_DATE');
DATE_PERIOD_FIELD:('AFS');
BOOLEAN_FIELD:('CERTIFIED'|'OVERRIDE');
INTEGER_FIELD:('AGE'|'HT'|'WT');
STRING_FIELD:('CURR_LOC'|'LANG_1'|'LANG_2''USER_LEVEL');
WAIVER_FIELD:('WAIVER');
   
DATE: /* empty */ {System.out.println("Matched DATE");};

INTEGER: DIGIT+
//INTEGER:('0'..'9')+
{
  System.out.println("Matched INTEGER: " + $text);
  //$type = INTEGER;
    // dynamically change the type when we match the date regular expression
  if ($text.matches("(19|20|21)[0-9]{2}[0-1]\\d{3}")) {
  //if ($text.matches("(19|20|21)[0-9]{2}[0-1]\\d{3}")) {
    System.out.println("Matched date pattern");
    $type = DATE;
  }
 
}
;
DATE_PERIOD_CONSTANT: ((INTEGER ' ' YEAR)|(INTEGER ' ' MONTH)|(INTEGER ' ' WEEK)|(INTEGER ' ' DAY))
{
  System.out.println("Matched DATE_PERIOD_CONSTANT");
};
DATE_CONSTANT:('TODAY'|'YESTERDAY'|'TOMMOROW');
BOOLEAN_CONSTANT:('TRUE'|'FALSE'|'"Y"'|'"N"');
IF: 'IF';
THEN: 'THEN';
ELSE: 'ELSE';
ENDIF: 'ENDIF';
AND: 'AND';
OR: 'OR';
YEAR: ('YEAR'|'YEARS');
MONTH: ('MONTH'|'MONTHS');
WEEK: ('WEEK'|'WEEKS');
DAY: ('DAY'|'DAYS');

STRING: '"' ID  (' ' ID)* '"' {System.out.println("Matched STRING");};
//  {
//    // strip the quotes once we match this token
//    setText(getText().substring(1, getText().length()-1));
//  }
//  ;

EQ: '=' {System.out.println("Matched EQ");};
NE: '<>';
LT: '<';
LE: '<=';
GT: '>';
GE: '>=';
HAS: 'HAS';
LPAREN: '(';
RPAREN: ')';
PLUS: '+';
MINUS: '-';
SHOW: 'SHOW';
FOR: 'FOR';
ID: ('A'..'Z'|'a'..'z'|'0'..'9'|','|'!'|'?'|':')+ {System.out.println("Matched ID: " + $text);};

COMMENT: ';' ~('\r'|'\n')* {skip();};
WS: (' '+|'\r'|'\n'|'\t') {$channel = HIDDEN;};

[Sam Harwell]

I created a gist for the example grammar:

 

https://gist.github.com/4338081

 

--

Sam Harwell

Owner, Lead Developer

http://tunnelvisionlabs.com

 

From: antlr-di...@googlegroups.com [mailto:antlr-di...@googlegroups.com] On Behalf Of Jason Ferguson
Sent: Wednesday, December 19, 2012 10:24 AM
To: antlr-di...@googlegroups.com
Subject: [antlr-discussion] Lexer Eating Integer Constant

 

First, I'll admit: this is probably one of the ugliest grammars you will ever see in your life (ironically named SimplePolicy.g). It STARTED as a simple grammar, but I ended up needing more detailed exceptions, so I ended up making multiple exceptions extending RecognitionException. Then a RuntimeException to wrap it so I could get out of the parser methods.

This grammar is supposed to validate a group of one or more IF/THEN/ELSE?/ENDIF statements. Unfortunately, when I try to compare one of the INTEGER_FIELD tokens to a INTEGER token, the INTEGER token has totally disappeared from the TokenInputStream. I've tried sprinkling println()s through to figure out what is going on, with no luck.

I apologize for the uglieness of this grammar, but if anyone can tell me why this dies I would be greatly appreciative.