Generic syntax highlighting for spyder

[David Joy]

Hi All,

I do a lot of work with spyder, especially cross-language work.
Spyder has great support for python, not so great support for Ruby, or
shell stuff, or MATLAB (yeah, I know)... Is there any interest in a
more general syntax parser for spyder?

I did a quick fork and prototyped a highlighter based on pygments:
http://pygments.org/

Prototype here: http://code.google.com/r/videan42-spyderlib-pygments

I wrote it for tcsh, but it should generalize to any of the 100-some
odd languages Pygments supports.

Cheers,
-David

[stefan]

On Tuesday, April 24, 2012 7:34:53 PM UTC-7, David Joy wrote:

Hi All,

Is there any interest in a more general syntax parser for spyder?

Oh - that sounds nice. I have wished for this feature on a few occasions. Especially for some other 'scripting' languages I use, and config files, bash, R etc. Maybe not for Matlab, though ;-)
You have a supporter in me.
Stefan

[Pierre Raybaut]

Hi David,

This sounds very interesting indeed.

I did some tests a while ago with a syntax highlighter based on
pygments (I started with code from IPython Qt console which is doing
something similar). But the problem was the performance: I compared
the two solutions with a very long Python script and the
pygments-based code was 2-3 times slower if I remember correctly. And
I think that Spyder syntax highlighting is already very slow as it is
(it is obvious when restarting Spyder with a lot of opened scripts:
the splash screen stays a while on "Loading editor plugin...").

So, have you an idea of the performance of your code?

Note that even if it's slower than Spyder's builtin highlighter, we
may use it anyway to add support for more languages (but not for
already supported languages for which we would keep the current
implementation).

Anyway I like the idea of a generic highlighter and I thank you for
looking into it.

Cheers
Pierre

> --
> You received this message because you are subscribed to the Google Groups "spyder" group.
> To post to this group, send email to spyd...@googlegroups.com.
> To unsubscribe from this group, send email to spyderlib+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/spyderlib?hl=en.
>

[luc.k]

I do a lot of work on windows and vbscripting (mainly because I work in an application that exposes business logic trough a com interface and on windows the default scripting languages are vbscript/javascript). 

For this I use pspad (http://www.pspad.com/) which is a good and complete tool for different scripting languages but not open source.

The highlighting there is done using "highlighters" per language, which are in simply ini files with keywords and so on.

[David Joy]

I wrote a quick speed test tonight.

On my core i5 windows laptop, the Spyder highlighter can color a 35000 line Python file in 1.9 seconds.  Pygments runs in 4.3 seconds... so yeah 2-3 times slower.

I was reading through the Qt docs and examples, and it seems some people have implemented log viewers in Qt by only buffering a couple hundred lines of text at a time.  I tried it, and if you prevent Qt from loading more than a few hundred lines, it doesn't take nearly as long to color (or load for that matter).  It seems to involve manipulating the blocks in the editor window, but I'm not sure about an unobtrusive way to implement this without tearing up the editor.

For now, I'm going to finish my hack as a fallback parser for languages Spyder doesn't already have, but I'd love to get this working more generally.

> To unsubscribe from this group, send email to spyderlib+unsubscribe@googlegroups.com.

[Pierre Raybaut]

Thanks for the feedback. Improving the syntax highlighter speed would be a good thing: I find it quite slow when compared to text editors wrote in a compiled language. But that is not an easy task.

There are two options that I didn't have the opportunity to explore:

1. Simplify the tons of regular expressions of the Python highlighter in order: I saw some implementations of a Python highlighter based on QSyntaxHighlighter here and there which were running faster than Spyder's implementation (but they were also either incomplete or buggy). 

2. Do not highlight the whole source code file at once but only the displayed part. The QSyntaxHighlighter class has been designed this way. But I chose to highlight the whole file anyway because I needed to parse the whole file to be able to create the "Structure" tree (class/function browser). So, highlighting only the displayed text should improved performance drastically but we would have to find another way to analyze code structure. Actually until now I assumed that doing this apart from syntax highlighting would take as much time as the current implementation, but maybe it's not true. 

-Pierre

> To unsubscribe from this group, send email to spyderlib+...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/spyderlib?hl=en.
>

--

You received this message because you are subscribed to the Google Groups "spyder" group.

To view this discussion on the web visit https://groups.google.com/d/msg/spyderlib/-/wLvGQDea_V8J.

To post to this group, send email to spyd...@googlegroups.com.

To unsubscribe from this group, send email to spyderlib+...@googlegroups.com.

[anatoly techtonik]

That's a very nice initiative. I wish I could have more time to help with that as I also maintain Colorer library to some degree and it would be interesting to be able to plug in and experiment with several syntax highlighting engines. Do you think it is now possible to classify syntax engines by features and make them pluggable?