Long comments/strings should not allow nesting with [[...]] in Lua 5.1+

[Paul K]

Neil:

It's been reported to me that the Lua lexer in Scintilla lexes embedded long Lua comments incorrectly, which can be seen with the following fragment:

--[[

abc

[[

def ]]

print("abc")

--]] 

print("def")

The lexer shows `print("abc")` as being a part of the comment and also has two fold points (starting on lines 1 and 3), whereas it should only be one (starting on line 1).
I'm using Scintilla v3.6.3, so it may be fixed in the latter version, but I didn't see it mentioned in the History file.

I see a comment `// [[-only allowed to nest` in LexLua.cxx, but this is not correct, as [[ comments can not be nested. It's also incorrect that it accepts long strings in long comments and long comments in long strings, as there is no such nesting as well.

It looks like there was nesting in Lua 5.0, but Lua 5.1+ doesn't support that, so it needs to be updated (or made configurable). Here is the description from the manual in the "Changes" section: "The long string/long comment syntax ([[string]]) does not allow nesting. You can use the new syntax ([=[string]=]) in these cases." (from https://www.lua.org/manual/5.1/manual.html#7.1). Just to clarify, there is no nesting with [[, but long comments/strings can still be nested, just need to use different opening strings.

Paul.

[KHMan]

On 11/20/2017 11:20 PM, Paul K wrote:
> Neil:
>
> It's been reported to me that the Lua lexer in Scintilla lexes
> embedded long Lua comments incorrectly, which can be seen with the
> following fragment:
>
> --[[
> abc
> [[
> def ]]
> print("abc")
> --]]
> print("def")
>
> The lexer shows `print("abc")` as being a part of the comment and
> also has two fold points (starting on lines 1 and 3), whereas it
> should only be one (starting on line 1).
> I'm using Scintilla v3.6.3, so it may be fixed in the latter
> version, but I didn't see it mentioned in the History file.
>
> I see a comment `// [[-only allowed to nest` in LexLua.cxx, but
> this is not correct, as [[ comments can not be nested. It's also
> incorrect that it accepts long strings in long comments and long
> comments in long strings, as there is no such nesting as well.
>
> It looks like there was nesting in Lua 5.0, but Lua 5.1+ doesn't
> support that, so it needs to be updated (or made configurable).
> Here is the description from the manual in the "Changes" section:

> "The long string/long comment syntax (|[[/string/]]|) does not
> allow nesting. You can use the new syntax (|[=[/string/]=]|) in

> these cases."
> (from https://www.lua.org/manual/5.1/manual.html#7.1). Just to
> clarify, there is no nesting with [[, but long comments/strings
> can still be nested, just need to use different opening strings.

The behaviour is/was intentional. I guess it might have been me
who did the relevant modifications for that change in long
comment/string behaviour.

The intention was to make it permissive so that it sort of works
for all Lua 4 to Lua 5.x (the "trying to please everyone"
strategy). Obviously the behaviour breaks down somewhere, and I
guess it's not good when one wants precise highlighting.

I don't have the time to deal with this at the moment. What do you
think is the best solution? Personally I don't have very high
demands for Lua highlighting so I don't have an opinion on this.
So feel free to hack away... :-)

--
Cheers,
Kein-Hong Man (esq.)
Selangor, Malaysia

[Paul K]

> The intention was to make it permissive so that it sort of works 
> for all Lua 4 to Lua 5.x (the "trying to please everyone" 
> strategy). Obviously the behaviour breaks down somewhere, and I 
> guess it's not good when one wants precise highlighting. 

Right; I realize that it was the behavior for Lua version up to 5.0, so probably had some reason to stay.

> What do you think is the best solution?

I think embedding with `[[` should be disallowed as it's been this way since Lua 5.1. If you do want to keep support for Lua 5.0 and older, then a configuration setting can be added, but since it's not going to work correctly for `[=+[` strings anyway, I don't see a reason to support those older versions. Also, ideally long strings and comments should be checked independently (right now a long string inside a long comment opens a new level and generates a new fold, and vice versa), but it's probably not a big deal as long as the nesting is done correctly.

Paul.

[Neil Hodgson]

Paul K:

> Neil:
>
> It's been reported to me that the Lua lexer in Scintilla lexes embedded long Lua comments incorrectly, which can be seen with the following fragment:

Please stop addressing comments to me. Around 375 people have worked on Scintilla. Asking one individual makes it less likely that others will reply.

Neil

[KHMan]

Please, feel free to modify LexLua the way you want. Don't hold
your breath waiting for me... :-)

Personally, I have no objection to default behaviour changing to
Lua 5.1+. I think most users won't notice, was nesting popular? I
didn't need to use it at all.

As an editor user, I'm not a terribly demanding type. So I have no
opinion on the best or correct folding behaviour for these things.
Your quality requirements should be much higher, so you should
dictate the behaviour of LexLua.