nesting lexer

[Herbert Breunung]

hello coders,
has anybody here thought about nesting lexers, say that you have
independend html js and php lexer but if is all accur in one file i
can say from here to there this lexer is in charde and from now on the
js. was there any attempt to achieve such with scintilla?

thanks a lot
herbert aka lichtkind

[Philippe Lhoste]

On 11/11/2009 18:26, Herbert Breunung wrote:
> has anybody here thought about nesting lexers,

Yes. Well, at least I did, although at a rather high level theoretical level... :-)

> say that you have
> independend html js and php lexer but if is all accur in one file i
> can say from here to there this lexer is in charde and from now on the
> js. was there any attempt to achieve such with scintilla?

I was a bit confused by the sentence...

My thoughts was it would be nice for a lexer to find out a portion of the source belongs
to another lexer (eg. <% or <! or <script> in HTML, asm keyword in some language, etc.),
go ahead to find the end of portion, and then save its context and call the other lexer,
indicating what area to process. The other lexer does its job and yields control on the
calling lexer. Which continue (if needed).

Probably very hard to implement in current state of lexers, as one needs to avoid state
conflict, to handle these context changes, etc. But it would avoid some redundancy (HTML
lexer duplicating JS lexer job, etc.).

AFAIK, Neil studied the topic in his experimental SinkWorld, which is, IIRC, a rewrite of
Scintilla on new bases.

--
Philippe Lhoste
-- (near) Paris -- France
-- http://Phi.Lho.free.fr
-- -- -- -- -- -- -- -- -- -- -- -- -- --

[Neil Hodgson]

Philippe Lhoste:

> AFAIK, Neil studied the topic in his experimental SinkWorld, which is, IIRC, a rewrite of
> Scintilla on new bases.

Yes, SinkWorld implemented composite lexers and multiple lexer
layers. You could have a mode for your web page templates that
combined a server side scripting and control layer (say a Java lexer
for JSPs), a HTML lexer for the content and a Javascript lexer for
client side scripts. Then there could be spell-checking and URL
recognition layers above that. Unfortunately it got a bit too complex
to work on so has been abandoned but you can use the code if you want.
http://www.scintilla.org/Tentacle.html

Even before SinkWorld, there were experimental lexers for PHP and
ASP inside LexHTML.cxx that tried to break up lexing into layers.
These were not maintained and were removed in 1.79.

This is a worthwhile area to work on but I don't really have the
time and energy. Possibly something with fewer features than attempted
in SinkWorld would be tractable.

Neil

[Herbert Breunung]

thanks a lot for answers, i wished i could add the but have currently
no nuits for that.

cheers
herbert