ANTLR Lexer error on single quote

234 views
Skip to first unread message

Люблю Пиво

unread,
Dec 16, 2016, 8:37:01 AM12/16/16
to antlr-discussion
I have create the grammar what give on the string

LIBRARY { com.idc.file.analystics.AnalysticsFunctions }
# there are n'ot error in this place
}

And have following error message on it. As you can see, it's lexer error.

Lexer listener token recognition error at: ''' line:2, char pos in line:17

Where I'm wrong as look like everything is fine in my point of view and how to solve it?

Regards, Vladimir p.s. Please, see grammar and JUnit file bellow

Grammar 

grammar OnlyWithespaceCommentValidator;

r
:
    (
        (
            RETURN* string
        )
        |
        (
            RETURN* not_string_term
        )
        | RETURN
        (
            RETURN+ ONE_LINE_COMMENT
        )
    )*
;

string
:
    QUOTED
;

not_string_term
:
    NOT_STRING_CHAR+
;

RETURN
:
    [\n\r]
;

NOT_STRING_CHAR
:
    (
        ~['"'|'\'']
    )
    | '|'
;

QUOTED
:
    '"' StringCharacters? '"'
    | '\'' CharacterLiteral? '\''
;

fragment
CharacterLiterals
:
    CharacterLiteral+
;

fragment
CharacterLiteral
:
    SingleCharacter
    | EscapeSequence
;

fragment
SingleCharacter
:
    ~['\\]
;

fragment
StringCharacters
:
    StringCharacter+
;

fragment
StringCharacter
:
    (
        ~["\\]
    )
    | '\''
    | EscapeSequence
;

fragment
EscapeSequence
:
    '\\' [btnfr"'\\]
;

ONE_LINE_COMMENT
:
    '#'
    (
        (
            ~[\r\n]
        )
        | '\''
    )*
;

WS
:
    [ \t\u000C]+ -> skip
;

JUnit file


Mike Lischke

unread,
Dec 16, 2016, 8:46:12 AM12/16/16
to antlr-di...@googlegroups.com
Hi Vladimir,

Am 16.12.2016 um 14:37 schrieb Люблю Пиво <vkoz...@gmail.com>:

I have create the grammar what give on the string

LIBRARY { com.idc.file.analystics.AnalysticsFunctions }
# there are n'ot error in this place
}

And have following error message on it. As you can see, it's lexer error.

Lexer listener token recognition error at: ''' line:2, char pos in line:17

Where I'm wrong as look like everything is fine in my point of view and how to solve it?

Regards, Vladimir p.s. Please, see grammar and JUnit file bellow


The grammar you attached doesn't contain important parts (like the QUOTED rule). But I guess the error comes from the fact that your input contains an unfinished string (the single quote in the comment). 


Люблю Пиво

unread,
Dec 16, 2016, 8:46:42 AM12/16/16
to antlr-discussion
package com.idc.omega.validation;

import static org.junit.Assert.assertEquals;

import java.util.ArrayList;
import java.util.BitSet;
import java.util.List;
import java.util.Set;

import org.antlr.v4.runtime.ANTLRErrorListener;
import org.antlr.v4.runtime.ANTLRInputStream;
import org.antlr.v4.runtime.CommonTokenStream;
import org.antlr.v4.runtime.Parser;
import org.antlr.v4.runtime.RecognitionException;
import org.antlr.v4.runtime.Recognizer;
import org.antlr.v4.runtime.atn.ATNConfigSet;
import org.antlr.v4.runtime.dfa.DFA;
import org.antlr.v4.runtime.tree.ParseTreeWalker;
import org.junit.Test;

import com.idc.omega.antlr.wellformedvalidator.OnlyWitespaceCommentValidatorLexer;
import com.idc.omega.antlr.wellformedvalidator.OnlyWithespaceCommentValidatorBaseListener;
import com.idc.omega.antlr.wellformedvalidator.OnlyWithespaceCommentValidatorParser;
import com.idc.omega.scanning.rule.SpinLineItem;
import com.idc.omega.validation.grammar.GrammarValidator;
import com.idc.omega.validation.spin.SpinValidationError;

public class OnlyWithespaceCommentValidatorTest {

    public class ErrorListener implements ANTLRErrorListener {
        private final String listenerName;
        public ErrorListener(String listenerName) {
            super();
            this.listenerName = listenerName;
        }

        List<String> errors=new ArrayList<String>();


        /**
         * This method is called by the parser when a full-context prediction
         * results in an ambiguity.
         */
        @Override
        public void reportAmbiguity(Parser recognizer, DFA dfa, int startIndex, int stopIndex, boolean exact,
                BitSet ambigAlts, ATNConfigSet configs) {
            // TODO Auto-generated method stub

        }

        /**
         * This method is called when an SLL conflict occurs and the parser is about
         * to use the full context information to make an LL decision.
         */
        @Override
        public void reportAttemptingFullContext(Parser recognizer, DFA dfa, int startIndex, int stopIndex,
                BitSet conflictingAlts, ATNConfigSet configs) {

        }

        /**
         * This method is called by the parser when a full-context prediction has a
         * unique result.
         */
        @Override
        public void reportContextSensitivity(Parser recognizer, DFA dfa, int startIndex, int stopIndex, int prediction,
                ATNConfigSet configs) {
            // TODO Auto-generated method stub

        }

        /**
         * Upon syntax error, notify any interested parties. This is not how to
         * recover from errors or compute error messages. ANTLRErrorStrategy
         * specifies how to recover from syntax errors and how to compute error
         * messages. This listener's job is simply to emit a computed message,
         * though it has enough information to create its own message in many cases.
         */
        @Override
        public void syntaxError(Recognizer<?, ?> recognizer, Object offendingSymbol, int line, int charPositionInLine,
                String msg, RecognitionException e) {
            errors.add(listenerName+" " +msg+" line:"+line+", char pos in line:"+charPositionInLine);

        }

        public List<String> getErrorsStr() {
            // TODO Auto-generated method stub
            return errors;
        }
    }


    protected List<String> getErrors(String pStr){
        OnlyWitespaceCommentValidatorLexer lexer = new OnlyWitespaceCommentValidatorLexer(new ANTLRInputStream(pStr));
        ErrorListener lexerErrorListener = new ErrorListener("Lexer listener");
        ErrorListener parserErrorListener = new ErrorListener("Parser listener");
        lexer.addErrorListener(lexerErrorListener);

        // Get a list of matched tokens
        CommonTokenStream tokens = new CommonTokenStream(lexer);

        OnlyWithespaceCommentValidatorParser parser = new OnlyWithespaceCommentValidatorParser(tokens);
        parser.addErrorListener(parserErrorListener);

        ParseTreeWalker walker = new ParseTreeWalker();

        walker.walk(new OnlyWithespaceCommentValidatorBaseListener(), parser.r());
        List<String> errors=parserErrorListener.getErrorsStr();
        errors.addAll(lexerErrorListener.getErrorsStr());

        return errors;
    }
    @Test
    public void testNoErrorWithWhitespaceComment(){
        List<String> errors = getErrors("    LIBRARY { com.idc.file.analystics.AnalysticsFunctions }\n"+
                "    # there are n'ot error in this place\n"+
                "}");
        errors.forEach(error->System.out.println(error));
        assertEquals(0, errors.size());
    }



}

Люблю Пиво

unread,
Dec 16, 2016, 11:02:11 AM12/16/16
to antlr-discussion
My grammar is

Люблю Пиво

unread,
Dec 16, 2016, 11:10:57 AM12/16/16
to antlr-discussion
Ops, sorry. Bad formatting
Reply all
Reply to author
Forward
0 new messages