ENB: Improving Leo's colorizer

[Edward K. Ream]

This Engineering Notebook post discusses work arising from PR #3282. This PR will fix problems coloring html <script> elements.  Along the way, I have discovered some improvements that I'll discuss here. Don't worry. I plan no substantial changes to Leo's colorizing code.

Note: no part of the ENB affects leoJS.

Background

jEdit rules drive Leo's default colorizer. For every language x that Leo supports, the file leo/modes/x.py describes those rules. Let's call this file the modes file for language x.

Qt's colorizer class calls Leo to colorize one line at a time. Each rule describes a pattern that may match at a given position i of a single line s.

Leo's JEditColorizer class calls a rule R at s[i] only if rule R could possibly match the character at s[i]. This optimization (and others) make Leo's colorizing exceptionally fast.

Some rules may delegate colorizing of spans of text to another mode file. Delegated coloring is complex and error-prone. Problems with delegation are at the heart of issue #3281.

Improvements

All projected improvements involve specialization of a particular rule. They:

- simplify the calls of rules in the modes file for a particular language.

- improve the speed of the specialized rule.

For example, jedit.match_plain_seq is considerably faster than jedit.match_seq because it does fewer tests. leo/modes/python.py replaces all calls to match_seq by match_plain_seq.

Special-purpose rules

The JEditColorizer class already contains special-purpose rules not found in the original jEdit rule sets. For example, jedit.match_leo_keywords succeeds if s[i] starts any Leo keyword. Leo defines many other specialized rules.

While working on PR #3282, I realized that larger-scale rules might help with problems with delegation. For example, jedit.match_html_script_element could handle the messy details of coloring the javascript text between <script> and </script> elements.

A specialized rule could also handle languages like vue.js that intermix different styles of colorizing. Currently, we Leonistas must specify the various "component" languages explicitly.

Summary

PR #3282 removes clutter from modes files and accelerates rule matching. These changes also suggest using more specialized rules to handle complex situations.

None of these changes will affect leoJS. None will change the architecture of the JEditColorizer class.

Edward