Performance of lexer generated for python2

[Reshma Godse]

Hi,

I am working on the grammar to parse input file of our application. I found that the lexer generated for python2 takes around 3+ minutes to tokenize a 600 line file. The parser's performance has improved a lot after setting PredictionMode to 'SLL'. But the lexer is becoming bottleneck. Is there any way to improve performance of lexer in python2?

Thanks for help!

Reshma

[Terence Parr]

I think a fairly literal translation was made from the Java code to Python and so it is likely to be less efficient because it might not use the fast part of Python, if that makes sense. That sounds kind of slow; hopefully we can improve speed in the future.

Ter

--
You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Dictation in use. Please excuse homophones, malapropisms, and nonsense. 

[Sam Harwell]

Hi Reshma,

 

The translation to Python did not preserve the semantics of certain performance-critical aspects of the Java code. However, since the final result was a correct parse tree, the author of the Python target ruled that any resulting impact on runtime performance did not constitute a bug, and will not be modified. The following issue and pull request contain information about one of these cases, where it was determined that the Python target is known to exhibit exponential complexity (O(k^n), k is a function of the grammar and n is the input size).

 

https://github.com/antlr/antlr4-python2/issues/23

https://github.com/antlr/antlr4-python2/pull/24

 

If you are putting your code into production with unknown inputs, you should take one of the following steps to make sure your application does not encounter these performance issues:

 

1.       Use one of the Java releases (either the reference release or my memory-optimized fork)

2.       Use the C# release (which is derived from the memory-optimized Java fork)

3.       Build your own copy of the Python runtime and manually apply the applicable patches submitted by @bdkearns (this will address some, but not all of the known performance issues in the Python target)

 

Sam

[Reshma Godse]

Hello Sam,

Thank you very much for your help. Your options will help us in deciding next set of actions.

Thanks,

Reshma

[Eric Vergnaud]

Hi Reshma,

I am the author of the Python and JavaScript targets.

I believe some of the comments in this thread are misleading.

The approach for improving performance is test-driven, and is on-going - see #36.

Anyone interested in improving performance based on facts rather than opinions is welcome.

Eric