[Lexer] Arbitrary line information

[RaptorX]

Hi Neil...

I was wondering if it is possible for a lexer to set permanent arbitrary information to a line in the same sense as markers or folding points.
That is, for the lexer to save some information that will follow the line around even if the user moved the line and that will be merged to adjacent lines if the current line was deleted.

This is due to the fact that the language I am lexing at the moment has the following options:

var =
(

this is a string block and because of that ; this is NOT a comment

this is a %variable% reference

and `t that is the escape secuence for the tab

)

var =
(% ` Com

another block of text but this time with some options

in this example ; this IS a comment (because we specified Com)

`t `n `t these escape sequences are literal now! (because we put a back tick in the options)

and this percent sign %variable% is not a variable reference anymore!! (because we put a percent sign in the options)

 

)

The point is that after the put those options my lexer should start marking those lines and set different lexing states for the text and it should be like that all the time while the user is inside those parenthesis...

moving those lines around shouldnt break the lexing.

So far I have tried with SetLineState without success, because as I mentioned in another thread, somehow the information saved by that function gets lost after a few calls (thats what i think).

Thanks in advance for your help!.

[Neil Hodgson]

RaptorX:

I was wondering if it is possible for a lexer to set permanent arbitrary information to a line in the same sense as markers or folding points.
That is, for the lexer to save some information that will follow the line around even if the user moved the line and that will be merged to adjacent lines if the current line was deleted.

   There's a good chance an arbitrary merging rule will not perform the operation you need.

So far I have tried with SetLineState without success, because as I mentioned in another thread, somehow the information saved by that function gets lost after a few calls (thats what i think).

   The linestate should be handled the same way as the lexer state of each byte - it moves forward from valid data and sets up later state. In particular, state should be attached to the line that produced that state and the initialisation should be from the line before the start of the lexing. If state is attached to the next line then it may be separated by a newly initialised line state when a line is inserted.

   Neil

[RaptorX]

First of all you are aware that the text inside the parenthesis will have a lexical state (string block) but that lexical state should be able to exit to another state (comments, escape secuences) based on options set at the beginning of the block right?

that means that after the options each new line should be able to "know" which options are set and "inherit" those options until the closing parenthesis arrives, after that those options have no meaning... I have been able to do that without issues. My problem comes when I am out of that block and then try to add a new line to an existing block, all option information for that block is lost which is not intended, hence my question about marking arbitrary information to a line.

So let me see if I understood correctly...

I should attach the line state to the options line, would i have to do that manually for each new line until the parenthesis close? or will it end only when i set a new linestate?

what would the difference be when the Lexer reads a whole file and sets everything up in one go and when the user is typing and going back and forth between lines (I think the Lex function is restarted and all LineState information is lost, am I right?)

Thanks for the reply.

[Colomban Wendling]

Hi,

Le 28/04/2013 13:50, RaptorX a �crit :

> First of all you are aware that the text inside the parenthesis will
> have a lexical state (string block) but that lexical state should be
> able to exit to another state (comments, escape secuences) based on
> options set at the beginning of the block right?
>
> that means that after the options each new line should be able to "know"
> which options are set and "inherit" those options until the closing
> parenthesis arrives, after that those options have no meaning... I have
> been able to do that without issues. My problem comes when I am out of
> that block and then try to add a new line to an existing block, all
> option information for that block is lost which is not intended, hence
> my question about marking arbitrary information to a line.
>
> So let me see if I understood correctly...
>
> I should attach the line state to the options line, would i have to do
> that manually for each new line until the parenthesis close? or will it
> end only when i set a new linestate?

You will have to set the line state on every line you want it to be,
it's not automatically copied to next lines.

> what would the difference be when the Lexer reads a whole file and sets
> everything up in one go and when the user is typing and going back and
> forth between lines

If your lexing function gets called with a startPos not on the first
line, you should fetch the previous line state, otherwise you know
you're (re-)lexing everything anyway:

state = sc.currentLine > 0 ? styler.GetLineState(sc.currentLine - 1) : 0

So you should set the state on each line affected by the options (the
options line and each one in the block).

> (I think the Lex function is restarted and all
> LineState information is lost, am I right?)

No, only the part that was invalidated (e.g. has to be re-styled) lost
everything; the part before still has full style and line state.

I suggest you to take a look at the PO lexer (LexPO.cxx). It is fairly
simple and uses line states.


Regards,
Colomban

[RaptorX]

Awesome!

I am not sure why I was assuming that the line state would be valid for each subsequent line until i set a different state!!

Probably a brainfart of mine or possibly because I was treating line states in the same way as lexical states which wont exit unless told to...

Anyways I will certainly look at that lexer and will give this a try. 

I think this not only fixes this case but also helps clean out the ton of variables that i have to set/clear for special states like "inExpression" or "inCommand"...

Thanks for your replies!

[Neil Hodgson]

RaptorX:

> I am not sure why I was assuming that the line state would be valid for each subsequent line until i set a different state!!

Its a technique that works well for some features and is implemented in the SparseState class used by the C++ lexer for remembering raw string terminators. The data is stored as a sequence of (line number, state) pairs which will often use less memory. If you structure your lexer as an object lexer you can store lexing state in any way you like.

Neil

[Philippe Lhoste]

On 28/04/2013 15:04, Colomban Wendling wrote:
> I suggest you to take a look at the PO lexer (LexPO.cxx). It is fairly
> simple and uses line states.

LexLua also manage LineState to handle multiline comments / strings (syntax [===[), which
is close of what you want.
Looking at some non-trivial lexers is often a source of good information (I remember
looking at the Perl lexer to see how line state was handled, when I needed to save some
information for the PoV lexer).

--
Philippe Lhoste
-- (near) Paris -- France
-- http://Phi.Lho.free.fr
-- -- -- -- -- -- -- -- -- -- -- -- -- --