Lua 5.4 (+ 5.5 too probably) OP_SETLIST 32 bit index overflow reachable from Lua source code

[nobody]

Hi,

with actual values the opcode / function length limit makes it
impossible to reach the relevant region, but OP_LOADNIL being able to
set multiple values at once means it's possible to build and compile a
table constructor long enough to overflow and wrap around. The required
source file would be over 8GB in size, but it can be generated and
streamed into the loader on the fly:

```
l = string.rep( "1", 1024, "," )
sf = coroutine.wrap( function( )
    coroutine.yield( "f = function() return table.unpack( { "..l.." } )
end\n" )
    coroutine.yield( "t = { \n\t" )
    local ll = l:gsub( "1", "nil" ):rep( 8192, ",\n\t" )
    for i = 1, 257 do  coroutine.yield( ll )  end
    coroutine.yield( ", f()\n}" )
    coroutine.yield( nil )
end )
local f = assert( load(sf) )
f()
for k, v in pairs( t ) do  print( k )  end
```

Running the code in 5.4 takes around 4 minutes and needs a little over
32GB of RAM.

With 257 repetitions of the inner ll block, the final block of 1s (from
f()) will end up at low indices again (prints 1 thru 1024).
With 256 repetitions, it'll end up at high indices but `next` will
complain about an invalid key when iterating with `pairs` (prints
2147483394 thru 2147483649 (256 values) before "invalid key to 'next'"
error), suggesting more serious corruption.

It's pretty clear that the 32 bit signed math in OP_SETLIST is to blame.
Just shifting to unsigned math is probably not enough, as 5.5 does just
that and still shows weirdness:

In 5.5, with/out assertions (-DLUAI_ASSERT -DLUA_USE_APICHECKS) and with
256/257/512/513 repetitions:
- 256 with/out assert: "table overflow" error (ok)
- 257 / 512 / 513 + assert: triggers lvm.c:1885 (B == 0)
- 257 / 512 / 513 no assert: runs ≈forever, no clue what it does (first
run forgotten about and aborted after 2 hours(!) at still below 1GB)

5.1 (with unpack in place of table.unpack) prints just 2 and exits, so
there's probably also some weirdness going on there.

Don't have 5.2 / 5.3 installed right now and didn't bother testing those.

-- nobody / Marco Schöpl

[Roberto Ierusalimschy]

> with actual values the opcode / function length limit makes it impossible
> to reach the relevant region, but OP_LOADNIL being able to set multiple
> values at once means it's possible to build and compile a table constructor
> long enough to overflow and wrap around. The required source file would be
> over 8GB in size, but it can be generated and streamed into the loader on
> the fly:
>

> [...]

Many thanks for the feedback. I don't have right now a machine with more
than 32G to test that code, but it seems the issue is really with the
array size in constructors. Can you check whether this change would fix
it?

--- a/lparser.c
+++ b/lparser.c
@@ -872,6 +872,7 @@ static void closelistfield (FuncState *fs, ConsControl *cc) {
luaK_setlist(fs, cc->t->u.info, cc->na, cc->tostore); /* flush */
cc->na += cc->tostore;
cc->tostore = 0; /* no more items pending */
+ checklimit(fs, cc->na, INT_MAX - LUAI_MAXSTACK, "items in a constructor");
}
}

-- Roberto

[nobody]

On 2025-08-26 20:37, Roberto Ierusalimschy wrote:
> Can you check whether this change would fix it?
>
> --- a/lparser.c
> +++ b/lparser.c
> @@ -872,6 +872,7 @@ static void closelistfield (FuncState *fs, ConsControl *cc) {
> luaK_setlist(fs, cc->t->u.info, cc->na, cc->tostore); /* flush */
> cc->na += cc->tostore;
> cc->tostore = 0; /* no more items pending */
> + checklimit(fs, cc->na, INT_MAX - LUAI_MAXSTACK, "items in a constructor");
> }
> }

That results in a clear error message for the examples:

./lua5.4-patched: test256.lua:11: (load):2095923: too many items in a
constructor (limit is 2146483647) in main function near 'nil'
./lua5.4-patched: test257.lua:11: (load):2095923: too many items in a
constructor (limit is 2146483647) in main function near 'nil'

and since vararg returns seem to be limited to 512K-1 items (?) and that
leaves 1M spare slots, that should indeed be safe?

-- nobody

[Roberto Ierusalimschy]

No matter what, all "vararg" returns (or multiple returns in general)
must be on the stack, and therefore are limited by LUAI_MAXSTACK.

-- Roberto