C++14 Parser demo goes into infinite loop when run under Cocoa, but parses fine from shell

[Alfonso Guerra]

Greetings,

I built and ran the cpp runtime demo, which runs without error. I modularized it slightly like so:

void

processFile(const char *aFilename)

{

std::ifstream stream;

stream.open(aFilename);

outputSourceReview(stream);

CPP14Parser *parser = parseSource(stream);

tree::ParseTree *tree = parser->translationunit();

outputANTLRTreeLikeLisp(std::cout, tree, *parser);

}

CPP14Parser *

parseSource(std::istream &stream)

{

ANTLRInputStream *input = new ANTLRInputStream(stream);

CPP14Lexer *lexer = new CPP14Lexer(input);

CommonTokenStream *tokens = new CommonTokenStream(lexer);

tokens->fill();

CPP14Parser *p = new CPP14Parser(tokens);

return (p);

}

void

outputANTLRTreeLikeLisp(std::ostream &aStream, tree::ParseTree *aTree, CPP14Parser &aParser)

{

aStream << aTree->toStringTree(&aParser) << std::endl;

}

void

outputSourceReview(std::ifstream &aStream)

{

    if(aStream.is_open())

{

std::cout << "Parsing source:" << std::endl;

        std::cout << aStream.rdbuf();

std::cout << "<EOF>" << std::endl << std::endl;

aStream.clear();

aStream.seekg(0);

}

}

It still runs correctly. So far so good.

I created a Cocoa app, and in an Objective-C++ file I import that code. On the CPP14BaseVisitor.cpp generated by antlr4 it has no trouble. But it goes into an infinite loop when parsing the header file, wherein ParserATNSimulator::closureCheckingStopState() invokes ParserATNSimulator::closure_() which invokes ParserATNSimulator::closureCheckingStopState() repeating decimal.

The parser also barfs on the example source files helloworld.cpp, macro.cpp, and template_args_text.cpp when run from this Cocoa app, but parses without fail .

The tokens parsed, as indicated by the following code, appear to be identical.

  for (auto token : tokens.getTokens()) {

    std::cout << token->toString() << std::endl;

  }

Any ideas what else to look for? How one goes about debugging the runtime?

Warmest regards,

Alfonso Guerra

[Mike Lischke]

Alfonso,

I created a Cocoa app, and in an Objective-C++ file I import that code. On the CPP14BaseVisitor.cpp generated by antlr4 it has no trouble. But it goes into an infinite loop when parsing the header file, wherein ParserATNSimulator::closureCheckingStopState() invokes ParserATNSimulator::closure_() which invokes ParserATNSimulator::closureCheckingStopState() repeating decimal.

The parser also barfs on the example source files helloworld.cpp, macro.cpp, and template_args_text.cpp when run from this Cocoa app, but parses without fail .

The C++14 grammar is a pretty complex beast and I don’t know how complete and correct it is. Hard to say from your description what could be wrong. Can you create a Github project with a demo that shows the problem? Would make it much easier for someone will to take a look.

Mike
-- 
www.soft-gems.net

[Alfonso Guerra]

On Monday, April 23, 2018 at 10:39:39 AM UTC-4, Mike Lischke wrote:

The C++14 grammar is a pretty complex beast and I don’t know how complete and correct it is. Hard to say from your description what could be wrong. Can you create a Github project with a demo that shows the problem? Would make it much easier for someone will to take a look.

Mike
-- 
www.soft-gems.net

Thanks for your reply, Mike. I copied the project and stripped it down, and it ran without crashing. The short version of the cause was parsing the source files in background threads using GCD where Cocoa's default stack size of 512K caused a stack overflow. Moving the parsing to the main thread eliminated the crash.

I'll just need to directly create the processing thread and change its stack size.

Thanks again, Mike for taking the time to answer.

Warmest regards,

Alfonso

[Mike Lischke]

The C++14 grammar is a pretty complex beast and I don’t know how complete and correct it is. Hard to say from your description what could be wrong. Can you create a Github project with a demo that shows the problem? Would make it much easier for someone will to take a look.

Mike
-- 
www.soft-gems.net

Thanks for your reply, Mike. I copied the project and stripped it down, and it ran without crashing. The short version of the cause was parsing the source files in background threads using GCD where Cocoa's default stack size of 512K caused a stack overflow. Moving the parsing to the main thread eliminated the crash.

Ah, yes, I have seen this also with e.g. my MySQL grammar. That’s a pretty big problem with ANTLR4 and non-trivial grammars. To make things worse there’s no way in standard C++ to manipulate the stack size of a thread in a platform neutral way (tbh. I don’t understand why that wasn’t included in std::thread).

That means you are limited what you can parse in a background thread, even if you can change the stack size (as you cannot increase it endlessly). Still, with C++ you can parse much more complex input than with, say, Java or Javascript/Typescript (they crash much earlier). Can’t say anything about other targets, but would expect that the same applies at least to all scripting language targets.

The main reason is that with each rule that is invoked, a new function call is executed (recursive descent parsing), which will quickly get to very high stack depths, especially with complex expression (or similar) rules. If all fails you could switch to the interpreter, which uses the same prediction infrastructure, but is an iterative implementation (a simple loop), instead of a recursive one. However, you will then have to manually implement predicate + action handling. However, that doesn’t seem to be that much work (especially if you don’t use target specific code in your grammar) and I wonder how the „interpreter“ would compare to the regular parser (since it executes less code when parsing).

Mike
-- 
www.soft-gems.net