ENB: @language jupytext done right

[Edward K. Ream]

This Engineering Notebook post describes a spectacular new pattern for collaboration between Leo's mode files. Only Leo's devs will likely be interested. Please feel free to ignore it.

Note: Unless qualified, all methods are members of the JEditColorizer (jedit) class.

A new delegation pattern

I have attempted (repeatedly and in vain) to use the jedit class's existing delegation machinery to switch between python and markdown coloring modes within text controlled by @language jupytext.

That effort is doomed to fail. The "delegate" kwarg of jedit's pattern matchers is rule-based. Alas, delegation in @jupytext nodes must happen line by line.

Happily, I have stumbled upon an astounding (dead easy!) new pattern for collaboration. The leo/modes/jupytext.py mode file is the result.

Several Ahas make it hard to reconstruct my previous mental world. I've mentioned this effect often. Anyway, the general progression went something like this:

Aha 1: jupytext.py can use python code, not rules.

The initial plan (now abandoned as the result of Aha 2) was as follows:

When a mode file contains a scan_line function, _recolor should call newMainLine. This new method will call the mode's scan_line function to do all the work.

But even after this Aha, I didn't see how even a Python program could delegate coloring. Must jupytext.py somehow duplicate the work of the mode files python.py and md.py? 

Eventually, I stumbled on the way forward.

Aha 2: Add one new rule to python.py and md.py!!

This Aha happened mysteriously. Call it what you will; it felt like luck at the time. It's one of my all-time best.

- In md.py, I added a new rule, md_jupytext_comment.

- In python.py, I changed one existing rule, python_comment.

Aha 3: There is no need for newMainLine or scan_line!

I reverted jupytext.py so that it is a standard mode file. This file contains just two rules:

- jupytext_comment does all the delegation!

- jupytext_directive handles Leo's directive.

These new rules are straightforward, as PR #4147 shows. Note that the diffs for md.py are large because I converted it from @clean to @file.

Summary

The mode file jupytext.py handles delegation elegantly without using the "delegation" kwarg of Leo's pattern matchers. The new design pattern is breathtaking in its simplicity.

Edward