LaTeX lexer patch coming

[Josepmaria Roca]

Hi all,

There are some open bugs about LaTeX lexer:

3081692

1187857

1493111

1778404

1856356

2082547

Some of them are duplicated or related.

I'm working on the LaTeX lexer due an editor Scintilla based has this bugs and does not properly handle the LaTeX higlighting. The editor is Geany.

I've two questions before sending the patch:

1) Currently, the lexer is in LexOthers.cxx. I can keep the lexer inside this file but, do you think that is a better approach to open a new file LexLatex.cxx?
The changes I've done are not so big and I think is not needed. But now can be a good time to do it. May be we can work on a better folding behavior (2082547).

2) What are the implications on introducing new states on lexers?

To solve the "verbatim" bugs (3081692, 1187857, 1856356) I need a new state SCE_L_VERBATIM. I modified the Scintilla.iface and executed the HFacer.py. I added also some new states. I've success with that and now Geany works as expected (hope SciTE too, but testing is still needed for that). If the client does understand a state, it fallback to default? if so, then the bugs will be fixed anyway.

Nice to see you again!

[Neil Hodgson]

Josepmaria Roca:

> There are some open bugs about LaTeX lexer:

Therre is also a TeX lexer in LexTeX.cxx. Some of the bugs may be
referring to it.

> 1) Currently, the lexer is in LexOthers.cxx. I can keep the lexer inside
> this file but, do you think that is a better approach to open a new file
> LexLatex.cxx?

If it grows, then it should be split out.

> 2) What are the implications on introducing new states on lexers?
> To solve the "verbatim" bugs (3081692, 1187857, 1856356) I need a new state
> SCE_L_VERBATIM. I modified the Scintilla.iface and executed the HFacer.py. I
> added also some new states. I've success with that and now Geany works as
> expected (hope SciTE too, but testing is still needed for that). If the
> client does understand a state, it fallback to default?

In general display will be OK (its likely to just use a default
font and colour) but user scripts that depend on lexical state, such
as an indexer, could break. There's no really good solution to this:
versioning lexers so clients can ask for any previous behaviour would
add considerable complexity so the issue has just been ignored. It
helps to minimize changes: don't rename or renumber states unless
really necessary.

Neil

[Josepmaria Roca]

Hi, I attached two patches:

1) latex_lexer.diff

2) scite_latex.diff

The first, improves the LaTeX lexer inside LexOthers.cxx.

The second, modifies latex.properties to take advantage of the new states.

If you want to test it, ensure SciTE is not importing tex.properties by commenting the line in SciTEGlobal.properties, and importing the latex one.

#import tex

import latex

CHANGES ON LATEX LEXER:

Now, the latex lexer provides more states as can be seen in Scintilla.iface.

Previous clients that does not know the new states will work except for the state SCE_L_TAG in case of the closing tag. You can check what happens on SciTE if does not patch the latex.properties.

With an evil latex document like the attached one (test.tex), previous lexer works horribly. 

All the open bugs related to LaTeX lexer can be closed because are fixed (LaTeX, not TeX!)

Hope it will be commited smoothly.

Next step will be to work with folding, but I've not so many time now.
My idea is to increase the folding level at SCE_L_TAG and decrease at SCE_L_TAG2. 

--
You received this message because you are subscribed to the Google Groups "scintilla-interest" group.
To post to this group, send email to scintilla...@googlegroups.com.
To unsubscribe from this group, send email to scintilla-inter...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scintilla-interest?hl=en.

[Josepmaria Roca]

latex_lexed.diff seems to revert some changes on other lexers. Do not commit. I'll fix it.

Sorry

[Josepmaria Roca]

Ok, this is the right patch.

Sorry for the mistake.

[Neil Hodgson]

The example file demonstrates the improvements well.

Committed with a minor change to cast the result of strlen to an
unsigned int as otherwise it casuses a wanrning for 64-bit compiles.

Neil

[Josepmaria Roca]

Thanks Neil.

For curiosity, how do you get the warnings? with what builder / platform have you tested?

I'm building in Linux 64bit (gcc 4.4.6) and I didn't received the warnings of the cast.

$ file LexOthers.o 

LexOthers.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

Anyway, I understand what was the problem:

http://en.wikipedia.org/wiki/Size_t

  Neil

[Neil Hodgson]

Josepmaria Roca:

> For curiosity, how do you get the warnings? with what builder / platform
> have you tested?

The current Windows SDK (7.1) compiler:
Microsoft (R) C/C++ Optimizing Compiler Version 16.00.40219.01 for x64

> I'm building in Linux 64bit (gcc 4.4.6) and I didn't received the warnings
> of the cast.

There have been similar errors with gcc on OS X inside Xcode but
that was probably due to turning an option on.

Neil

[Josepmaria Roca]

I played with gcc options and I cannot reproduce the warnings on Linux x64.

I think it's about how is defined size_t. It's platform dependent, so, same gcc options could produce different warnings.

Thanks

  Neil