Syntax coloring mystery solved!!

[Edward K. Ream]

tl;dr: Calling doc.markContentsDirty(0, len(p.b)) suffices to recolor a node! This should be done instead of calling highlighter.rehighlight explicitly.

I could never have found this bug if I hadn't seen how fast pyzo colorizing code was within pyzo itself compared to the same code within Leo.

Having refresh-from-disk do the coloring correctly and quickly was the crucial lucky break.  Tracing showed that a big @edit node was being colorized correctly despite highlighter.rehighlight not being called.

After a bit of noodling (without beer) I saw that the body pane's QTextDocument must be repeatedly calling highlighter.highlightBlock. I verified this was so (without huge traces) by examining the before and after counts of highlighter.n_calls.  As hoped, these jumped, indicating that something (QTextDocument is the only candidate) was in control.

This is a lucky, lucky break on Friday the thirteenth ;-) It is likely that Leo's existing colorizers can be speeded up by 10x. If so, there will be no need for the big text hack. Crucially, Qt does not show the text until it has been fully colorized.

Furthermore, close study of the pyzo code shows that actual syntax coloring could now be deferred until idle time, thereby further improving performance.

Pyzo is full of superb code, and pyzo's colorer is no exception.  I'm not sure whether to adapt pyzo's code for all syntax coloring, but I'm tempted.  It would be a big project.

In the meantime, I can adapt two good ideas from the pyzo colorer:

1. Precomputing QTextFormat objects saves a lot of time. Leo's present colorizers do far too much work withing the main colorizing loop.

2. Using a special tag for unterminated strings is genius.  The three little dots are a great visual cue, and terminating the token at the end of the line means that adding a quote does not cascade to all following lines.  A great hack.

I'm glad this is behind us.  I was starting to doubt my sanity, and starting to imagine bugs that would never be found...

The way is now open for more pyzo integration.  I am seriously considering replacing Leo's custom text widget with pyzo's.  It a big project, but it will pay huge dividends.

Edward

[Terry Brown]

Congrats on finding the culprit.

Couple of things I'd like to see in Leo syntax highlighting, highlighting of trailing whitespace, and highlighting of https://en.wikipedia.org/wiki/Non-breaking_space and also of tab (\t).  To be quite honest I've only made one foray into the syntax highlighting code, some time ago, and really can't remember the details.  So don't know if this is hard or trivial.

Cheers -Terry

---

From: Edward K. Ream <edre...@gmail.com>
To: leo-editor <leo-e...@googlegroups.com>
Sent: Friday, January 13, 2017 12:23 PM
Subject: Syntax coloring mystery solved!!

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

[Edward K. Ream]

​​

​​

On Fri, Jan 13, 2017 at 5:25 PM, 'Terry Brown' via leo-editor <leo-e...@googlegroups.com> wrote:

​> ​

Congrats on finding the culprit.

​Thank you.

​

​Strangely, the performance bug remains (sometimes) in the legacy coloring code. I allude to this in this morning's announcement.
​

​The bug now seems even more mysterious. Calling QTextDocument.markContentsDirty does not cause a recoloring.  As before, calling rehighlight directly is super slow.

Note: if a huge node is selected when Leo starts, that node is colored quickly(!).  But revisiting the node later is slow.

I have spent several hours on this new mystery.  The best clue is that the body pane's QTextDocument keeps changing with the legacy code, but remains invariant (as expected) when using pyzo.

Presumably, finding out why the document changes will allow the bug to be fixed.

​> ​

Couple of things I'd like to see in Leo syntax highlighting, highlighting of trailing whitespace, and highlighting of https://en.wikipedia.org/wiki/Non-breaking_space and also of tab (\t). 

​The show/toggle-invisibles commands will do some of this.  ​

​Please file a bug report if non-breaking spaces aren't shown to your satisfaction.

EKR

[Edward K. Ream]

On Sat, Jan 14, 2017 at 7:23 AM, Edward K. Ream <edre...@gmail.com> wrote:

​> ​

Calling QTextDocument.markContentsDirty

​ ​

​[in the legacy code]​

does not cause a recoloring.

​Hurray! Counts show that leo_h.highlightBlock is being called the expected number of times. But nothing happens.

A call to viewPort.update() should make the new styles visible.  I remember a call something like that in the pyzo code.

It looks like Leo's legacy colorers look can compete on speed with pyzo's. This is great news. It means that we can use all the legacy mode-related machinery unchanged.

EKR

[Edward K. Ream]

On Saturday, January 14, 2017 at 9:52:59 AM UTC-5, Edward K. Ream wrote:

On Sat, Jan 14, 2017 at 7:23 AM, Edward K. Ream <edre...@gmail.com> wrote:

​> ​

Calling QTextDocument.markContentsDirty

​ ​

​[in the legacy code]​

does not cause a recoloring.

​Hurray! Counts show that leo_h.highlightBlock is being called the expected number of times. But nothing happens.

A call to viewPort.update() should make the new styles visible.

Actually, the jEdit colorizer was not being inited properly.  The colorizer is now working well and very quickly.

One bug remains. Switching nodes does not always properly set the colorizing language. This is surprisingly difficult, made all the more so by subtle differences in which methods get called when.

On my Python 3 machine, it appears that LeoHighlighter.rehighlight is never called.  That shouldn't be a big problem.

One would think that LeoTree.set_body_text_after_select could properly init everything. But bugs still remain.

Rebecca and I are leaving tomorrow for Madison.  We will be on the road about a week. I would like to up what I have for reference, but I probably can't in the trunk.  Time to create a branch :-)

Edward

[Edward K. Ream]

On Saturday, January 14, 2017 at 9:01:37 PM UTC-5, Edward K. Ream wrote:

Rebecca and I are leaving tomorrow for Madison.  We will be on the road about a week. I would like to up what I have for reference, but I probably can't in the trunk.  Time to create a branch :-)

Done, and pushed, I think.  It's called fast-colorizing, rev 2beba78b5. Let me know if the branch doesn't show up.  This is my first branch...

It may take quite a bit of tweaking to get all nodes colorized with the proper language.  Furthermore, the code needs to be housecleaned. I'll attend to all this when I can, but it will likely be next week.

Edward

[Terry Brown]

On Sat, 14 Jan 2017 18:22:26 -0800 (PST)

"Edward K. Ream" <edre...@gmail.com> wrote:

> Done, and pushed, I think. It's called fast-colorizing, rev
> 2beba78b5. Let me know if the branch doesn't show up. This is my
> first branch...

Branch is there in the correct place.

Safe travels.

Cheers -Terry

[Edward K. Ream]

On Sat, Jan 14, 2017 at 11:52 PM, 'Terry Brown' via leo-editor <leo-e...@googlegroups.com> wrote:

Branch is there in the correct place.

​Thanks for the confirmation.​

 

​Rev 0a2a983 fixes all known colorizer bugs, and rev fa3d41e is a bit of make-work while eating my own dog food.

Terry, please test this branch for a day or so and report any problems. I'll follow this with interest on my travels. If all seem well, feel free to merge the branch.

Edward

[Terry Brown]

On Sun, 15 Jan 2017 06:18:53 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

> Terry, please test this branch for a day or so and report any
> problems. I'll follow this with interest on my travels. If all seem
> well, feel free to merge the branch.

Initial look I'm not seeing anything but way faster colorizing :-)
Will report if anything else shows up.

Cheers -Terry