Lexer testing question

[Matt Gilarde]

I'm fixing a bug in the Progress ABL lexer (LexProgress.cxx). The bug only occurs when lexing starts at a point after the beginning of the file. It doesn't occur if the file is fully lexed from the start to the end.

Is there a way to tell the framework to lex the file in a different order than beginning to end? For example, can I tell it to lex the first five lines and then to start lexing from line 20? I know that it will lex lines 6 through 19 if I do that but the bug depends on lexing happening out of order like this.

Thanks,

Matt

[Neil Hodgson]

Matt Gilarde:

> I'm fixing a bug in the Progress ABL lexer (LexProgress.cxx). The bug only occurs when lexing starts at a point after the beginning of the file. It doesn't occur if the file is fully lexed from the start to the end.
>
> Is there a way to tell the framework to lex the file in a different order than beginning to end? For example, can I tell it to lex the first five lines and then to start lexing from line 20? I know that it will lex lines 6 through 19 if I do that but the bug depends on lexing happening out of order like this.

TestLexers (in lexilla/test) lexes both the whole file and each line separately then compares the results. This finds most bugs with inter-line state but not all. You could write a test that lexes in some pattern that matches your issue.

From README:
> Styling and folding tests are first performed on the file as a whole, then the file is lexed
and folded line-by-line. If there are differences between the whole file and line-by-line
then a message with 'per-line is different' for styling or 'per-line has different folds' will be
printed.

If you want to lex lines 1-5, then start lexing from 20 while not having the lexer process lines 6-19 then that is not really supported by Scintilla which wants all previous lines lexed and will only request lexing whole lines. It would be more difficult to write lexers if they were required to cope with different orders or mid-line starts.

However, you may write your own test application that creates a lexer and calls Lex multiple times with your choice of arbitrary ranges. This does not necessarily mean that the lexer will process just the requested ranges as lexers are free to backtrack to the start of a feature or start of file.

Neil