So suppose that token can be multiline, the token continuation character is '\'. Suppose additionaly that it could have an comments inside continued token .
In colorizer I want to highlight keywords and class names.
I have a source code input file like this:
i\
n\
t //that stands for 'int' - it is a keyword and must be highlighted properly
OR:
i\
//....
//around about 100 lines of comments or blank lines like:
//...
nt //that stands for 'int' too and must be highlighted.
The question is simple, how I could do that highlighting trick without retokenizing full file with full Lexer for every line?
i\
f( a == b) ... // here 'if' - is a keyword an need to be highlighted as a keyword
BUT:
if = a + b // here 'if' - is just an identifier and must be highlighted as an identifier
So, colorizing is heavily affected by semantic context of a code statement. If I understand you right, I need firstly gather all information from the parse and then, try to use it in my colorizer?
Can you explain what is Semantic highlighting asynchronous approach and how It could be implemented in VS (or just give a reference where I can read about it), please. I have a strong need to implement this feature.
I also understand that token continuation info is really parser thing and could not be accessible in a colorizer lexer until full file parsing. So I think, that my only hope to properly colorize tokens is to store parsing information and use it somehow in colorizer. But I faced with this first time and really very confused (how to deal with this situation). Actually I have never heard how about 'semantic highlighting approach', especially how to do it as a lower-priority asynchronous operation. Is this must be done in colorizer or in something other plase (and if so, what is this 'magic' place)?
Many thanks.