SSL suport in the C++ target

[Alainmarcel]

Is there an equivalent of

parser.getInterpreter().setSLL(true);

for the C++ target?

[Mike Lischke]

Is there an equivalent of

parser.getInterpreter().setSLL(true);

for the C++ target?

:-D When I read the mail title I thought: "What the heck has a transport layer to do with an ANTLR4 runtime?".

But now it's clear what you want and yes, there is a similar call. Just look in the .h files to see what you can use (hint: parser.getInterpreter<ParserATNSimulator>()->setPredictionMode(PredictionMode::SLL)).

Mike
-- 
www.soft-gems.net

[Alainmarcel]

Thanks! that works!
I was fooled by Netbeans Project search, I could not find any reference to SSL, I'll now know better, I probably did not set the include search path correctly.
I got it to work nicely and it gave me a nice ~2x speed up!
I could not get the BailErrorStrategy to work correctly though, in some cases, it made the SSL pass actually fail instead of succeeding. I commented out below the part that causes the failure, maybe I got the shared pointer code wrong?
Also, is the RecognitionException exception checking how it is supposed to be done?

...
m_antlrParserHandler->m_parser->getInterpreter<atn::ParserATNSimulator>()->setPredictionMode(atn::PredictionMode::SLL);
 
  m_antlrParserHandler->m_parser->removeErrorListeners ();
  //Ref<BailErrorStrategy> bailref(new BailErrorStrategy());
  //m_antlrParserHandler->m_parser->setErrorHandler(bailref);
  try
    {
      m_antlrParserHandler->m_tree = m_antlrParserHandler->m_parser->top_level_rule ();
    }
  catch (RuntimeException& ex)
    {
      RuntimeException* exp = &ex;
      if (dynamic_cast<RecognitionException*>(exp))
        {
          m_antlrParserHandler->m_tokens->reset ();
          //Ref<DefaultErrorStrategy> bailref(new DefaultErrorStrategy());
          //m_antlrParserHandler->m_parser->setErrorHandler(bailref);
          m_antlrParserHandler->m_parser->addErrorListener (m_errorListener);
          m_antlrParserHandler->m_parser->getInterpreter<atn::ParserATNSimulator>()->setPredictionMode(atn::PredictionMode::LL);
          m_antlrParserHandler->m_tree = m_antlrParserHandler->m_parser->top_level_rule ();
        }
    }

[Mike Lischke]

I was fooled by Netbeans Project search, I could not find any reference to SSL, I'll now know better, I probably did not set the include search path correctly.

SLL, Alain, not SSL ;-)

I got it to work nicely and it gave me a nice ~2x speed up!
I could not get the BailErrorStrategy to work correctly though, in some cases, it made the SSL pass actually fail instead of succeeding. I commented out below the part that causes the failure, maybe I got the shared pointer code wrong?

It's working like this: The SLL algorithm is faster because it doesn't do the same depth of predication like LL prediction does. This works nicely for correct input, so most of the time you can start with SLL prediction. Only when you get a parse error with SLL it could be this is due to ambiguities that couldn't be solved that way. In this case you have to do a second run (this time with full LL prediction) to check if you really have a syntax error or just something SLL couldn't solve.

Also, is the RecognitionException exception checking how it is supposed to be done?

When you try first with SLL you don't need to collect syntax errors nor is a full error recovery necessary, since you are going to do a second parse run anyway. Hence it is useful to stop parsing as early as possible. That's what the BailErrorStrategy is for. It throws a special exception, which is not catched by the normal error processing and hence stops the ongoing parse run and returns to your code. You can then catch it to know if you should do that LL run.

Here's a full parse function with time measurement and token/parse tree dump I use for testing my MySQL grammar:

void parse(const std::string &sql, bool dumpTokenStream, bool dumpParseTree) {

  ANTLRInputStream input(sql);

  MySQLLexer lexer(&input);

  CommonTokenStream tokens(&lexer);

  MySQLParser parser(&tokens);

  lexer.serverVersion = version;

  lexer.charsets = { "_utf8", "_latin1", "_latin2", "_cp1250" };

  //lexer.sqlMode = MySQLBaseLexer::NoBackslashEscapes;

  parser.serverVersion = version;

  parser.setBuildParseTree(true);

  // First parse with the bail error strategy to get quick feedback for correct queries.

  parser.setErrorHandler(std::make_shared<BailErrorStrategy>());

  parser.getInterpreter<ParserATNSimulator>()->setPredictionMode(PredictionMode::SLL);

  parser.removeErrorListeners();

  parser.sqlMode = MySQLBaseLexer::AnsiQuotes;

  try {

    auto start = std::chrono::steady_clock::now();

    tokens.fill();

    auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - start);

    std::cout << "Token fill time: " << duration.count() / 1000.0 << " secs" << std::endl;

  } catch (IllegalStateException &) {

    std::cout << "Error: illegal state found, probably unfinished string." << std::endl;

  }

  if (dumpTokenStream) {

    for (auto token : tokens.getTokens())

      std::cout << token->toString() << std::endl;

    std::cout << std::endl;

  }

  tree::ParseTree *tree;

  auto start = std::chrono::steady_clock::now();

  try {

    tree = parser.query();

  } catch (ParseCancellationException &pce) {

    // If parsing was cancelled we either really have a syntax error or we need to do a second step,

    // now with the default strategy and LL parsing.

    tokens.reset();

    parser.reset();

    parser.setErrorHandler(std::make_shared<DefaultErrorStrategy>());

    parser.getInterpreter<ParserATNSimulator>()->setPredictionMode(PredictionMode::LL);

    parser.addErrorListener(&ConsoleErrorListener::INSTANCE);

    tree = parser.query();

  }

  auto duration = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now() - start);

  if (parser.getNumberOfSyntaxErrors() > 0 || lexer.getNumberOfSyntaxErrors() > 0) {

    std::cout << "Errors encountered: " << parser.getNumberOfSyntaxErrors() + lexer.getNumberOfSyntaxErrors()<< std::endl;

    std::cout << "Query: " << sql << std::endl;

  }

  std::cout << "Parse time: " << duration.count() / 1000.0 << " ms" << std::endl;

  if (dumpParseTree && tree != nullptr) {

    std::cout << std::endl << "Parse tree: " << tree->toStringTree(&parser) << std::endl;

  }

}

Hope that gets you started.

Mike
-- 
www.soft-gems.net

[Alainmarcel]

Thanks Mike, that works!
(And I need to proofread myself...)