ANTLR4 limitation with changing token type in parser init action before alternative matching

[Paula Pastragus]

Hi all,

I am currently migrating some grammar logic from ANTLR2 to ANTLR4 and I encountered a limitation related to token type mutation during parsing.

In ANTLR2, we had cases where a token type could be changed inside an init/action block, and then the updated token type could be matched immediately afterwards in one of the alternatives. For example, based on some parser context, a keyword token could be retagged as a SYMBOL, and the grammar alternative expecting SYMBOL would match correctly.

In ANTLR4, this no longer seems reliable because the alternative selection is driven by adaptivePredict(). The prediction happens before the parser action can influence the decision, so changing the token type inside the parser action does not affect the alternative chosen by the predictor.

Has anyone dealt with this kind of ANTLR2-to-ANTLR4 migration issue before? Is there a recommended way to preserve this behavior? Thanks!