code too large problem in Hive Parser

[Pengcheng Xiong]

Hello,

     We are Apache Hive developers here at Hortonworks. In order to have several new features, we extended our HiveParser, which is based on ANTLR 3.4. We always hit a code too large problem as follows

   

    "[ERROR] /Users/pxiong/Projects/backport/hive2/hive2/ql/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveParser_IdentifiersParser.java:[13583,27] code too large"

   

    However, L13583 in HiveParser_IdentifiersParser.java is a DFA

   "    static final String[] DFA5_transitionS = {

            "\1\2\5\uffff\1\2\4\uffff\1\2\7\uffff\10\2\1\uffff\25\2\1\uffff"+

            "\4\2\1\uffff\2\2\1\uffff\4\2\1\uffff\3\2\1\uffff\1\2\1\uffff"+

            "\4\2\1\uffff\20\2\1\uffff\5\2\1\uffff\1\2\1\uffff\1\2\2\uffff"+"

     And this DFA5's description is like this.

    

    public String getDescription() {

            return "68:1: groupingSetExpression : ( ( LPAREN )=> groupingSetExpressionMultiple | groupingExpressionSingle );";

        }

      And here are the details of the rules

     groupingSetExpression

@init {gParent.pushMsg("grouping set expression", state); }

@after {gParent.popMsg(state); }

   :

   (LPAREN) => groupingSetExpressionMultiple 

   |

   groupingExpressionSingle

   ;

groupingSetExpressionMultiple

@init {gParent.pushMsg("grouping set part expression", state); }

@after {gParent.popMsg(state); }

   :

   LPAREN 

   expression? (COMMA expression)*

   RPAREN

   -> ^(TOK_GROUPING_SETS_EXPRESSION expression*)

   ;

groupingExpressionSingle

@init { gParent.pushMsg("groupingExpression expression", state); }

@after { gParent.popMsg(state); }

    :

    expression -> ^(TOK_GROUPING_SETS_EXPRESSION expression)

    ;

    I have read the previous posts about dividing a single rule to multiple rules to resolve the problem. However, I think groupingSetExpression rule is already simple enough. I have also tried to divide other rules, but it seems that it does not work.

    Could anybody help us on this? Any suggestions are welcome. Thanks! 

Best

Pengcheng