Lua 5.5: LUA_COMPAT_LOOPVAR does not fully restore the previous behavior

[Sens Fu]

Hi all,

I would like to report what I believe is an oversight (or at least an
under-documented limitation) in the new LUA_COMPAT_LOOPVAR option in
Lua 5.5.

The comment in luaconf.h states:

    @@ LUA_COMPAT_LOOPVAR makes for-loop control variables not read-only,
    ** as they were in previous versions.

The phrase "as they were in previous versions" suggests that enabling
this option is supposed to restore the Lua 5.4 behavior. In practice,
the option only flips the variable kind from RDKCONST to VDKREG (see
lparser.c, around the LOOPVARKIND macro). The compiler stops rejecting
the assignment, but the underlying single-slot design of 5.5 is
unchanged: the user-visible loop variable and the internal control
variable share the same stack register (e.g. ra+3 for a generic for).
As a consequence, any assignment to the loop variable inside the body
corrupts the value that OP_TFORCALL passes back to the iterator on the
next iteration.

Minimal reproduction (built with LUA_COMPAT_LOOPVAR=1):

    local t = {}
    for i = 1, 50 do t["k" .. i] = i end

    for k, v in pairs(t) do
      k = nil      -- silently corrupts the iterator's control value
    end

Expected (matches Lua 5.4): loop completes, all 50 entries visited.

Actual (Lua 5.5 with LUA_COMPAT_LOOPVAR=1):

    lua: invalid key to 'next'
    stack traceback:
        [C]: in for iterator 'for iterator'
        ...

The same problem affects numeric for (assignments to the loop variable
shift the index, although the count is preserved so no error is
raised -- just incorrect indices).

The Lua 5.4 generic for kept the iterator's control value in a
separate hidden slot, so writing to the user variable was harmless.
With the 5.5 single-slot layout the assignment goes directly to the
slot that the next OP_TFORCALL reads. Therefore turning the constness
off is necessary but not sufficient to recreate the 5.4 semantics.

I see three reasonable directions:

1. Clarify the manual / luaconf.h comment to state that
   LUA_COMPAT_LOOPVAR only removes the compile-time check, and that
   assignments to the loop variable may corrupt the iteration. This
   matches the current implementation but contradicts "as they were in
   previous versions".

2. Restore the 5.4 semantics under the compatibility option by keeping
   a hidden control slot separate from the user-visible loop variable.
   A parser-only patch is sufficient: reserve one extra hidden local
   ("(for control)") after the user variables, emit OP_MOVE to save
   the iterator's returned control at the start of each iteration body,
   and emit OP_MOVE to restore it before OP_TFORCALL / OP_FORLOOP. No
   VM change is required; the cost is one stack slot and two MOVE
   instructions per loop, and only when LUA_COMPAT_LOOPVAR is defined.

3. Remove LUA_COMPAT_LOOPVAR entirely, since its current semantics are
   hard to specify precisely.

I tried (2) locally and it correctly recovers the 5.4 behavior for
both numeric and generic for, including closures captured inside the
body (the restore is emitted after leaveblock so any OP_CLOSE captures
the user's modification before the control is restored). The standard
test suite passes unchanged when LUA_COMPAT_LOOPVAR is left at its
default (0).

I am happy to share the patch if the team finds it useful, but I would
mainly like to know which of the three directions reflects the
intended design.

Thanks for Lua,

<frodo>

[Roberto Ierusalimschy]

> I would like to report what I believe is an oversight (or at least an
> under-documented limitation) in the new LUA_COMPAT_LOOPVAR option in
> Lua 5.5.
>
> The comment in luaconf.h states:
>
> @@ LUA_COMPAT_LOOPVAR makes for-loop control variables not read-only,
> ** as they were in previous versions.
>
> The phrase "as they were in previous versions" suggests that enabling
> this option is supposed to restore the Lua 5.4 behavior. In practice,
> the option only flips the variable kind from RDKCONST to VDKREG (see
> lparser.c, around the LOOPVARKIND macro). The compiler stops rejecting
> the assignment, but the underlying single-slot design of 5.5 is
> unchanged: the user-visible loop variable and the internal control
> variable share the same stack register (e.g. ra+3 for a generic for).
> As a consequence, any assignment to the loop variable inside the body
> corrupts the value that OP_TFORCALL passes back to the iterator on the
> next iteration.

You are right. Thanks for the feedback.

-- Roberto