I am trying to implement an include macro that will work in the lexer stage in ANTLR4.
I have initially implemented it in the parser and that worked just fine, however my requirements had forced me to support things like:
This requires evaluating the included file within the parse context of the calling parser (afaik this cannot be trivially resolved resolved within the parser stage).
I have been experimenting with some implementation (see below), however I have my doubts about it since I cannot be certain I didn't miss anything, which is why I seek your knowledge & experience to suggest corrections/improvements.
INCLUDE : '#include' WS+ IDENT {
final String importText = getText();
final String filename = extractName(importText);
final Reader reader = getReaderForFile(filename);
final CompileContext ss = new CompileContext(getInputStream());
final ANTLRInputStream antlrStream = new ANTLRInputStream(reader);
setInputStream(antlrStream);
} catch ( final Throwable e ) {
throw new Error("Cannot import source=[" + sourceName + "]");
private final Stack<CompileContext> includes = new Stack<CompileContext>();
private static class CompileContext {
public final IntStream input;
private CompileContext(final CharStream input) {
this.marker = input.mark();
public Token nextToken() {
Token token = super.nextToken();
if ( token.getType() == Token.EOF && !includes.empty() ) {
final CompileContext ss = includes.pop();
setInputStream(ss.input);
getInputStream().release(ss.marker);
token = this.nextToken();
if ( ((CommonToken) token).getStartIndex() < 0 ) {
token = this.nextToken();
// This was unfortunate that we had to override & duplicate it, however, we had to avoid resetting the DFA's state.
// wack Lexer state variables
_input.seek(0); // rewind the input
_type = Token.INVALID_TYPE;
_channel = Token.DEFAULT_CHANNEL;
_tokenStartCharIndex = -1;
_tokenStartCharPositionInLine = -1;
_mode = Lexer.DEFAULT_MODE;
getInterpreter().setLine(1);
getInterpreter().setCharPositionInLine(0);
// Do not reset() the interpreter, since it resets the DFA state when calling to prevAccept.reset().
// getInterpreter().reset();
I'd appreciate any insight, ideas, suggestions, improvements and bug fixes.
Thanks!
-- TR