Lua syntax ambiguity

[Martin Eden]

Ahoy list,

Let's say we're writing Lua code generator. He just applies grammar rules.

So let's take parts of Lua grammar and apply them.

  chunk ::= block

  block ::= {stat} [retstat]

  stat ::= varlist ‘=’ explist

  varlist ::= var {‘,’ var}

  var ::=  Name | prefixexp ‘[’ exp ‘]’

  explist ::= exp {‘,’ exp}

  exp ::=  false | true | LiteralString | prefixexp | tableconstructor

  prefixexp ::= var

  tableconstructor ::= ‘{’ [fieldlist] ‘}’

  fieldlist ::= field {fieldsep field} [fieldsep]

  field ::= ‘[’ exp ‘]’ ‘=’ exp

  fieldsep ::= ‘,’ | ‘;’

Problem is that "LiteralString" can start with "[" (long quotes).
And so when writing index in "field" we emit "[", then we go
to "exp" which may be "LiteralString" and we will emit another "[".

So we'll have text like "t={[[[a]]]=true}".

It's generated by perfectly abiding EBNF rules and greedy parser will
miserably fail on it.

I doubt there is a way to fix it by mangling EBNF grammar.
That's just consequence that character to open index is "[",
same character was chosen to start "long quotes", and table
index can be literal string.

-- Martin

[Halalaluyafail3]

The syntax is token based as far as I understand, so in t={[[[a]]]=true} it sees: punctuator {, string literal [[[a]], punctuator ], punctuator =, keyword true, and punctuator }. That does not meet the syntax so it is invalid.

[Andrey Dobrovolsky]

Hi Martin,

I think it wouldn't be complicated for code generator to insert extra
space before and after an index value:

t={[ [[a]] ]=true}

and it would be enough, for correct parsing.

Simple modification of the rule

field ::= "[ " exp " ]" ‘=’ exp

probably is the minimal change, however I don't know whether string
literals of the length greater than 1 are allowed in EBNF. If not then

field ::= '[' ' ' exp ' ' ']' ‘=’ exp

Thanks for an interesting corner case,
-- Andrew

нд, 16 серп. 2026 р. о 18:00 Halalaluyafail3 <luigi...@gmail.com> пише:

> --
> You received this message because you are subscribed to the Google Groups "lua-l" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/lua-l/1a35c810-a4be-4697-86bf-454af3892e23n%40googlegroups.com.

[Martin Eden]

On 2026-08-16 19:58, Andrey Dobrovolsky wrote:
> Hi Martin,

Hi Andrew,

> Simple modification of the rule
>
>  field ::= "[ " exp " ]" ‘=’ exp
>
> probably is the minimal change, however I don't know whether string
> literals of the length greater than 1 are allowed in EBNF.

Yeah string literals are allowed in BNF but this modification will
make code strings like "t={[a]=true}" invalid. And using space
in literal considered bad manners there.

> I think it wouldn't be complicated for code generator to insert extra
> space before and after an index value:
>
> t={[ [[a]] ]=true}
>
> and it would be enough, for correct parsing.

Actually lexer needs any separator (one or more of any of (space,
newline, comment))
between "[" from index opening and "[" from string opening.

But it can't be inserted nicely in code generation. Because proper code just
writes to output stream and can't (and shouldn't) read existing output
stream.


So we have something like

  // serialize key and value
  // ...
  emit_opening_index()
  serialize(KeyNode)
  // ...

And that flag ("bro if you gonna open with "[" then plz insert any
whitespace you like before it") is ugly thing to embed in code.

I've handled it by temporarily monkey-patching output stream method.
Just another similar-ugliness solution

https://github.com/martin-eden/lua_table_serializer/blob/06e5bedb6a248a950158b7fb4062a8d6f61957d6/src/workshop/concepts/codec_lua_graph/compile/Initializer.lua#L138-L143


In theory if we aim to comply with "our language is described by formal
grammar"
we can state that any long quote MUST have at least one "=".

So "[[abc]]" becomes invalid, minimal valid will be "[=[abc]=]".
This will resolve this syntax clash.

> Thanks for an interesting corner case,
> -- Andrew

I'm glad you liked it.

-- Martin

[Lars Müller]

As others have said, this isn't really a bug. The grammar is token-based; tokens are properly defined in prose. There is no ambiguity here, the language is described by the formal grammar plus a precise description of the tokenization. The only minor issue is that the "lexical conventions" section in the reference manual neglects to mention that the tokenizer is greedy. I would consider this no big problem; virtually every tokenizer is, not every last detail needs to be written down.

This does not call for an ugly solution however. Quite the opposite, just mirror the parsing. Parsing is text -> tokens -> AST. "Unparsing" should be AST -> tokens -> text. First convert an AST to a stream of tokens, then convert that stream of tokens to text, inserting separating suppressed tokens as necessary. If you're lazy, an emit(ttype, content) function which decides based on the last written token.

By the way, a hypothetical naive approach of directly feeding the grammar into some tool is also doomed to fail on many more token concatenations than "[[[" (this just happens to be an ambiguous-looking one). "if1thenend" is a single token, made up of the concatenation of four tokens. A fuzzer which simply applies the grammar rules may produce this along with many more abominations like it.
Conversely, a parser directly based on the grammar will be way too permissive (this kind of thing happened to me some years ago when I tried to translate the grammar to a PEG for LPEG); a naive grammar will be ambiguous. Should "localx = 42" be "local x = 42" or "_G.localx = 42"?

Aside: I am not even sure the situation is salvageable in a CFG at all. In greedy tokenization, long strings are closed by the *first* matching delimiter. Is there a way to enforce this with a CFG? I don't think so. You can of course express valid long strings as

inner ::= '[' {.} ']'
padded ::= inner | '=' padded '='
long ::= '[' padded ']'

where . stands for any character. But I see no way to ensure that {.} must not contain a string of equal signs of length matching the delimiters, for all infinitely many possible numbers of delimiters. The following "program" would be valid per the naive CFG, but is obviously invalid when tokenized and subsequently parsed:

--[[ blah blah blah ]]
syntax error!
blah blah blah ]]

- Lars

On Sun, Aug 16 2026 at 21:09:21 +02:00:00, 'Martin Eden' via lua-l <lu...@googlegroups.com> wrote:

On 2026-08-16 19:58, Andrey Dobrovolsky wrote: > Hi Martin, Hi Andrew, > Simple modification of the rule > >  field ::= "[ " exp " ]" ‘=’ exp > > probably is the minimal change, however I don't know whether string > literals of the length greater than 1 are allowed in EBNF. Yeah string literals are allowed in BNF but this modification will make code strings like "t={[a]=true}" invalid. And using space in literal considered bad manners there. > I think it wouldn't be complicated for code generator to insert extra > space before and after an index value: > > t={[ [[a]] ]=true} > > and it would be enough, for correct parsing. Actually lexer needs any separator (one or more of any of (space, newline, comment)) between "[" from index opening and "[" from string opening. But it can't be inserted nicely in code generation. Because proper code just writes to output stream and can't (and shouldn't) read existing output stream. So we have something like   // serialize key and value   // ...   emit_opening_index()   serialize(KeyNode)   // ... And that flag ("bro if you gonna open with "[" then plz insert any whitespace you like before it") is ugly thing to embed in code. I've handled it by temporarily monkey-patching output stream method. Just another similar-ugliness solution https://github.com/martin-eden/lua_table_serializer/blob/06e5bedb6a248a950158b7fb4062a8d6f61957d6/src/workshop/concepts/codec_lua_graph/compile/Initializer.lua#L138-L143 In theory if we aim to comply with "our language is described by formal grammar" we can state that any long quote MUST have at least one "=". So "[[abc]]" becomes invalid, minimal valid will be "[=[abc]=]". This will resolve this syntax clash. > Thanks for an interesting corner case, > -- Andrew I'm glad you liked it. -- Martin

--

You received this message because you are subscribed to the Google Groups "lua-l" group. To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/lua-l/eea51f32-acbe-49e2-b072-496c3cfb42e4%40disroot.org.

[Martin Eden]

On 2026-08-17 02:48, 'Lars Müller' via lua-l wrote:

> As others have said, this isn't really a bug. The grammar is
> token-based; tokens are properly defined in prose. There is no
> ambiguity here, the language is described by the formal grammar plus
> a precise description of the tokenization. The only minor issue is
> that the "lexical conventions" section in the reference manual
> neglects to mention that the tokenizer is greedy. I would consider
> this no big problem; virtually every tokenizer is, not every last
> detail needs to be written down.

I am not flagging it as fatal flaw. Just quirk that imo is not properly
documented. (Unlike "v=f\n()" vs "v= f()" parsing quirk,
which is properly described.)

> This does not call for an ugly solution however. Quite the opposite,
> just mirror the parsing. Parsing is text -> tokens ->
> AST. "Unparsing" should be AST -> tokens -> text. First convert an
> AST to a stream of tokens, then convert that stream of tokens to
> text, inserting separating suppressed tokens as necessary. If you're
> lazy, an emit(ttype, content) function which decides based on the
> last written token.

I agree with this approach.

But having "emit(ttype, content)" which has internal state is bad.
It makes her non-deterministic. And she has two responsibilities:
emit data and be aware of surroundings.

What we need is decision-making function that emits separator
depending of types of adjacent elements: emit_sep(prev_type, next_type).

So pseudocode becomes

  // serialize key and value
  // ...

  next_event = event_start_index
  emit_sep(prev_event, next_event)
  serialize(KeyNode)
  prev_event = next_event
  // ...

We can move that "prev_event" tracking by adding internal state
to "emit_sep()". Again, tradeoff here is non-determinism.
But this function has only one responsibility:

  // serialize key and value
  // ...

  emit_sep(event_start_index)
  serialize(KeyNode)
  // ...

Yeah, I think I feel the right implementation now. Thanks for hint!

> By the way, a hypothetical naive approach of directly feeding the
> grammar into some tool is also doomed to fail on many more token
> concatenations than "[[[" (this just happens to be an
> ambiguous-looking one). "if1thenend" is a single token, made up of
> the concatenation of four tokens. A fuzzer which simply applies the
> grammar rules may produce this along with many more abominations like
> it.

Yeah it's easy to write wrong code.

> Conversely, a parser directly based on the grammar will be way too
> permissive (this kind of thing happened to me some years ago when I
> tried to translate the grammar to a PEG for LPEG); a naive grammar
> will be ambiguous. Should "localx = 42" be "local x = 42"
> or "_G.localx = 42"?

Well, two alphanumerics MUST be separated. That follows from their grammar.
What to do with alphanumeric with something else depends of language.
F.e. in bash proper assignment is "a=b", not "a = b".
"a=1b=2" is invalid in Lua but may be valid in other language.

> Aside: I am not even sure the situation is salvageable in a CFG at
> all. In greedy tokenization, long strings are closed by the *first*
> matching delimiter. Is there a way to enforce this with a CFG? I
> don't think so. You can of course express valid long strings as

What is CFG?

> inner ::= '[' {.} ']'
> padded ::= inner | '=' padded '='
> long ::= '[' padded ']'

Mathematically nice grammar.

Reminds me "binary_number = ( '0' | '1' ) [binary_number]".

Not practical tho. You will have to do N calls just to read 2*N "=" chars.

> where . stands for any character. But I see no way to ensure that
> {.} must not contain a string of equal signs of length matching the
> delimiters, for all infinitely many possible numbers of delimiters.
> The following "program" would be valid per the naive CFG, but is
> obviously invalid when tokenized and subsequently parsed:
>
> --[[ blah blah blah ]]
> syntax error!
> blah blah blah ]]

Yeah long quotes are fun. In my parser grammar element can (also) be
function. And in my first implementation function for matching opening
quote was storing information of chunk length in some vault. This
information was used by function for matching closing quote.

(Later I rewrote that part to incomprehensible regexp "%[(%=*)%[.-%]%1%]"
because I wanted more speed.)

> - Lars

-- Martin

[Roberto Ierusalimschy]

> Let's say we're writing Lua code generator. He just applies grammar rules.
>
> So let's take parts of Lua grammar and apply them.
>

>   [...]

If you blindly apply the grammar rules, you end up with these too:

a = bc = d

ifa = 3 thenb = 1endx = 4

There is nothing special about '['.

On the other hand, you can blindly add a space after each token and
everything works.

-- Roberto

[Martin Eden]

On 2026-08-17 19:02, Roberto Ierusalimschy wrote:
> On the other hand, you can blindly add a space after each token and
> everything works.
>
> -- Roberto

Of course we can just emit separators everywhere, just in case.
Who cares?

“The History of every major Galactic Civilization tends to pass through
 three distinct and recognizable phases, those of Survival, Inquiry and
 Sophistication, otherwise known as the How, Why, and Where phases. For
 instance, the first phase is characterized by the question 'How can we
 eat?' the second by the question 'Why do we eat?' and the third by the
 question 'Where shall we have lunch?”

                                                       -- Douglas Adams

But implementing it properly (emit only required separators) looks
hard for me. Node serializers have no information about surroundings,
and their caller don't know what serializer will emit.


F.e. "t={['a']=1}": we emitted "[", we know we are just opened index.
But to what "a" will be serialized is not our business. Maybe it will
be ""a"", maybe "'a'", maybe "[[a]]", maybe '\097', maybe
"(function()return 'a'end)()"...

So we must decide whether or not emit " " before calling node serializer.
That's impossible with such design.

( I still consider Lua is a nice language for manually writing code
(and trying to understand it later). (Comparing with what C had become,
C# and Perl.)

That syntax quirks ("[[[a]]]", "a=f ()" is not "a=f\n()", no empty
charsets for regexps) arise mostly from code generation.

Maybe I should just use LISP?
)

-- Martin

[Sean Conner]

It was thus said that the Great 'Martin Eden' via lua-l once stated:

> > Aside: I am not even sure the situation is salvageable in a CFG at
> > all. In greedy tokenization, long strings are closed by the *first*
> > matching delimiter. Is there a way to enforce this with a CFG? I
> > don't think so. You can of course express valid long strings as
>
> What is CFG?

It stands for "Context Free Grammar". Parsing theory is a bit of a rabbit
hole.

-spc

[Sean Conner]

It was thus said that the Great 'Martin Eden' via lua-l once stated:
>

> But implementing it properly (emit only required separators) looks
> hard for me. Node serializers have no information about surroundings,
> and their caller don't know what serializer will emit.
>
> F.e. "t={['a']=1}": we emitted "[", we know we are just opened index.
> But to what "a" will be serialized is not our business. Maybe it will
> be ""a"", maybe "'a'", maybe "[[a]]", maybe '\097', maybe
> "(function()return 'a'end)()"...

What are you doing that requires this level of detail?

That aside, it sounds like you have some form of AST that you want to turn
into textual Lua (possibly with a minimum of extra space for some reason).
Per your example:

t={['a']=1}

At some point you have a string. By what criteria are you using to write
a string as [[...]]? Why not "..."? Or '...'? Avoid the issue entirely.

-spc

[Martin Eden]

Hi Sean,

On 2026-08-18 20:36, Sean Conner wrote:
> What are you doing that requires this level of detail?

Just printing tables.

> That aside, it sounds like you have some form of AST that you want to turn
> into textual Lua (possibly with a minimum of extra space for some reason).
> Per your example:
>
> t={['a']=1}
>
> At some point you have a string. By what criteria are you using to write
> a string as [[...]]? Why not "..."? Or '...'? Avoid the issue entirely.

Objective is not "avoid issue and make job done". Objective is
"deal with it. Preferably nicely".

I have string quoting function at Sophistication stage.
She can quote string in many ways. We like her.

https://github.com/martin-eden/lua_table_serializer/blob/06e5bedb6a248a950158b7fb4062a8d6f61957d6/src/workshop/concepts/lua/quote_string.lua

Another function is embedding her output in index brackets [].
Code there is at Inquiry stage. It knows about possible syntax clash
and has means to avoid it (without adding unnecessary spaces).
Implementation for that is not conceptually nice, so I created this
topic to get hints, clarify things for myself or at least warn
fellow masters about this case.
> -spc
-- Martin

[Sean Conner]

It was thus said that the Great 'Martin Eden' via lua-l once stated:

> Hi Sean,
>
> On 2026-08-18 20:36, Sean Conner wrote:
> > What are you doing that requires this level of detail?
> Just printing tables.

Okay. I have a similar function. Two actually, one to dump to a string
(which has been occasionally useful) and a second one that dumps to stdout.

> > That aside, it sounds like you have some form of AST that you want to
> > turn
> >into textual Lua (possibly with a minimum of extra space for some reason).
> >Per your example:
> >
> > t={['a']=1}
> >
> > At some point you have a string. By what criteria are you using to
> > write
> >a string as [[...]]? Why not "..."? Or '...'? Avoid the issue entirely.

> Objective is not "avoid issue and make job done". Objective is
> "deal with it. Preferably nicely".

I mean, how does your code determine which quoting method to use? In mine
[1], I just use double quotes, and will escape control codes or byte values
larger than 126. For example:

> dump("x",x)
x =
{
bar = C_FUNCTION,
file = file (0xcbc720),
[true] = false,
["one\ntwo"] = 12.000000,
foo = "1\n2\t3\v4\a5\000\001\002b\226\141\133",
["[[a]]"] = 2.000000,
x = x,
}

I sidestepped the issue of using [[...]] and '...' by not using them [2].

-spc

[1] https://github.com/spc476/lua-conmanorg/blob/fa5bcdbdcc0ceeeadd238a470c063f8b843c11cd/lua/table.lua#L78

[2] Yes, some values are not valid Lua (like "bar = C_FUNCTION") but
some Lua values are not easy to deal with (userdata, functions and
threads come to mind). I was able to solve this somewhat with my
CBOR [3] implemetation, but never published it. Also, this means
that some tables can't be properly round-tripped with my code; that
doesn't bother me too much and hasn't been an issue.

[3] Concise Binary Object Representation:

defined: <https://cbor.io/>
my module: https://github.com/spc476/CBOR

[Martin Eden]

Mine implementation values diversity. So all three quotes can be used.


Long quotes are nice for multi-line strings without ASCII control chars.
(Newline is not considered control char for this case.)

Short quotes can start with ' or ", whichever occurs less.

There are many ways to decide which (and how) characters should be
escaped with \. If string length is 1, 2, 4 or 8 and contains high ASCII
or control chars this is likely binary value. We quote all it's chars.

Else we quote control chars using \123. (Before there was fancy quoting
for \b \f \v etc but I disliked it.)

And mine serialization is to stream. Which can be whatever you can write
function Write(str): file, pipe, string. That means that it's not
limited by RAM amount (well, _less_ limited).

functions, metatables, uservalues and threads are not serialized.
I don't know how and never had practical need to do this.

Selling point is that my implementation serializes graphs:
table can contain self-references. And produced string contains
loadable Lua code.

-- Martin

[Sewbacca]

Forgive me if I haven't understood your problem properly, but the problem is about the very last step in the process.
I suggest you have two phases one for generating the token stream, so separation is clear and another one to generate the resulting string.

There concatenate two tokens and see if by reparsing it you get those two tokens back, if not add a space.

So [[[ would be detected as not being the same as [ [[.

~ Sewbacca

[Martin Eden]

On 2026-08-16 21:09, 'Martin Eden' via lua-l wrote:
> But it can't be inserted nicely in code generation. Because proper
> code just
> writes to output stream and can't (and shouldn't) read existing output
> stream.

I think I found acceptable solution for handling this.


We're doing AST tree serialization. AST node has field "type" which
is string with node type name (like "string" or "table").

Serialization function is recursive and writes node to output stream.

Core output stream has only one method: Write(str).

Problem is that when serializing index in table ("[" serialize(key) "]"),
serialize() can write something starting with "[" which will spoil
index bracket.

So we must know from what that next token will begin.
So we need stream that has method to peek at output.

And generally, when key node type is that problematic "string" --
we call serialize() for that peekable sub-stream, peeking
and it's first char, writing separator to main stream if needed,
writing sub-stream data to main stream:


local serialize
serialize =
  function(Node, OutputStream)
    -- ...
    if (KeyNode.type == 'string') then
      local StringStream = new(StringStream)
      serialize(KeyNode, StringStream)
      local node_str = StringStream:ToString()
      if starts_with(node_str, '[') then
        OutputStream:Write(' ')
      end
      OutputStream:Write(node_str)
    else
      serialize(KeyNode, OutputStream)
    end
    -- ...
  end

-- Martin

[Francisco Olarte]

Hi Martin.

I have been tracking this thread, and IMO, you are complicating things a lot.

I would personally just insert a space in places which could need it,
there are not that many, but if youMUST absolutely avoid extra
spaces...

On Wed, 19 Aug 2026 at 16:47, 'Martin Eden' via lua-l
<lu...@googlegroups.com> wrote:
> I think I found acceptable solution for handling this.

...

> Problem is that when serializing index in table ("[" serialize(key) "]"),
> serialize() can write something starting with "[" which will spoil
> index bracket.
> So we must know from what that next token will begin.
> So we need stream that has method to peek at output.
> And generally, when key node type is that problematic "string" --
> we call serialize() for that peekable sub-stream, peeking
> and it's first char, writing separator to main stream if needed,
> writing sub-stream data to main stream:

Have you considered changing serialize(key,stream) to
serialize(key,stream,context)? Extra parameter is IMO justified, and
can default to nil for the current behaviour, but in this case you can
pass, i.e., '[' as context when you are serializing a key and the
serialize function can then insert a space if it is going to emit a
'[[' string.

In fact, all my serialization functions are of the form
local function _serialize(data, context) ... lots of code probably
with some helper calls... end
local function serialize(data)
local r = _serialize(data, INITIAL_CONTEXT)
return some_function_probably_identity(r)
end

contexts, in my case, are rather complex, they include streams, a
state object to detect loops and duplicates, key path tracking info (
so I can patch recursive tables when I write things like lua source ),
indentation settings and states, what type of thing I am writing,
whatever is needed. And normally I have several published entry points
which use different contexts. But the pattern has served me well for a
long time in several languages.

Francisco Olarte.