Adding preprocessor comment styles for C++ lexer

[Neil Hodgson]

A bug report (#3487406) is concerned with problems when comments occur within macro definitions when the "styling.within.preprocessor" option is off. One potential solution is to add new styles for comments inside preprocessor text. This would also allow some more choice when choosing how these comments would appear: they could be made indistinguishable from the rest of the preprocessor lines or could be blended in more than the standard comments.

The most common use of comments inside preprocessor directives is to mark the end of constructs like this:

#ifdef UNICODE
#define strcpy wcscpy
#endif /* UNICODE */

There are complex examples in multi-line macros like this:

#define BFD_FAKE_SECTION(SEC, FLAGS, SYM, NAME, IDX) \
/* name, id, index, next, prev, flags, user_set_vma, */ \
{ NAME, IDX, 0, NULL, NULL, FLAGS, 0, \
\
/* linker_mark, linker_has_input, gc_mark, gc_mark_from_eh, */ \
0, 0, 1, 0, \
...

One question here is which of the 6 comment styles would need to be duplicated as preprocessor comment styles. Are documentation comments ever located inside preprocessor text? Looking at the Doxygen documentation didn't show any examples -- it mostly shows separate lines of document comments and few examples of inline doc comments.

There is a cost to defining new styles as it requires applications to add definitions for the visual appearance of these styles.

Bug:
http://sourceforge.net/tracker/?func=detail&aid=3487406&group_id=2439&atid=102439

Neil

[mr.maX]

On Sunday, June 17, 2012 3:03:38 AM UTC+2, Neil Hodgson wrote:

  A bug report (#3487406) is concerned with problems when comments occur within macro definitions when the "styling.within.preprocessor" option is off. One potential solution is to add new styles for comments inside preprocessor text. This would also allow some more choice when choosing how these comments would appear: they could be made indistinguishable from the rest of the preprocessor lines or could be blended in more than the standard comments.

For the sake of completeness in this discussion, I would also like to point out that mentioned bug report already comes with a patch that fixes handling of comments inside macros. The patch fixes the issue by properly returning to the preprocessor style when terminating comment style that's inside preprocessor.

The approach of using additional styles for comments inside macros would also fix the problem, but in my opinion the amount of necessary code changes does not justify it. Depending on all comment types that will be supported inside macros, the resulting patch may end up being bigger than the original one.

I would also like to mention that the main reason why handling of comments inside macros should be fixed is because in some occasions they can disrupt further code folding in the document.

-- 

Regards,

Marko Njezic - mr.maX @ MAX Interactive corp.

Personal: http://www.markonjezic.com/

MAX's Empire: http://www.maxempire.com/

MAX's HTML Beauty++ 2004: http://www.htmlbeauty.com/

[Neil Hodgson]

Since there have been no statements that documentation comments need to be supported inside preprocessor directives, it seems reasonable to apply a simple modification that adds a single new style SCE_C_PREPROCCOMMENT.

Example before and after: http://www.scintilla.org/bfd.png

Line comments can also appear at the end of preprocessor directives but this is far less common than stream comments, so has not been added in the proposed change.

Diff attached.

[mr.maX]

On Tuesday, June 19, 2012 2:41:31 AM UTC+2, Neil Hodgson wrote:

  Since there have been no statements that documentation comments need to be supported inside preprocessor directives, it seems reasonable to apply a simple modification that adds a single new style SCE_C_PREPROCCOMMENT.

Although this probably isn't widely used - it is possible to embed documentation comments inside macros. One example: http://stackoverflow.com/questions/7880811/how-to-doxygen-comment-generated-code

 

   Example before and after: http://www.scintilla.org/bfd.png

   Line comments can also appear at the end of preprocessor directives but this is far less common than stream comments, so has not been added in the proposed change.

   Diff attached. 

Your patch does not handle transition back to preprocessor state always correctly. The following code will not be highlighted properly:

#define MACRO()\

/* comment */\

{ ... <- wrong highlighting from here on

The patch also doesn't handle multiline comments at all and stops highlighting at first encountered line end.

[Neil Hodgson]

mr.maX:

> Your patch does not handle transition back to preprocessor state always correctly. The following code will not be highlighted properly:
>
> #define MACRO()\
> /* comment */\
> { ... <- wrong highlighting from here on

OK

> The patch also doesn't handle multiline comments at all and stops highlighting at first encountered line end.

I was thinking of the infamous old
#define COMMENT /*
#define ENDCOMMENT */
COMMENT Make the comments pretty! ENDCOMMENT
but it appears that went away with C99.

New patch attached. Not real happy with moving the sc.Forward() out of the for statement as it complicates termination checking.

[mr.maX]

On Tuesday, June 19, 2012 5:01:13 AM UTC+2, Neil Hodgson wrote:

   New patch attached. Not real happy with moving the sc.Forward() out of the for statement as it complicates termination checking. 

I would only change for loop to the while loop, now that the loop expression consists only of condition check.

On a side note, I have been testing various different scenarios and found one case where highlighting will be wrong:

#define MYMACRO() \[CRLF]

myfunc() { .. } \[CRLF]

/* comment */[CRLF]

\[CRLF] <- this line should not be highlighted as preprocessor, because previous line does not end with backslash

{ ...

The reason why this happens is due to the way how line end is treated in order to allow eol filled style as explained in the previously mentioned bug report.

[Neil Hodgson]

mr.maX:

> #define MYMACRO() \[CRLF]
> myfunc() { .. } \[CRLF]
> /* comment */[CRLF]
> \[CRLF] <- this line should not be highlighted as preprocessor, because previous line does not end with backslash

> { …

This problem can be reproduced without comments:

#d M\

\
x
y

Neil

[Neil Hodgson]

I think the answer is to move the revert-to-default before the line continuation handling.

New patch attached.

Neil

[mr.maX]

On Wednesday, June 20, 2012 4:54:46 AM UTC+2, Neil Hodgson wrote:

  I think the answer is to move the revert-to-default before the line continuation handling.

That appears to have fixed the issue.

[Neil Hodgson]

Change committed but with name of new style not abbreviated: SCE_C_PREPROCESSORCOMMENT.

Neil