A signed integer overflow in lua_gc

[Sergey Bronnikov]

Hello,

An undefined behavior (signed integer overflow) is happening on execution of a Lua chunk below. It is reproduced on the latest version of PUC Rio Lua (a5522f06d2679b8f18534fd6a9968f7eb539dc31) that built with enabled UndefinedBehaviourSanitizer instrumentation.
UB is introduced by the commit 85a3c1699c9587a9e952b4ab75b1c6c310ebebea.

How to reproduce:

1. git clone https://github.com/lua/lua
2. cd lua
3. make -j CC=clang MYCFLAGS="-O0 -fsanitize=signed-integer-overflow -fno-sanitize-recover=signed-integer-overflow" MYLDFLAGS="-O0 -fsanitize=signed-integer-overflow -fno-sanitize-recover=signed-integer-overflow"
4. create a C file:
cat << EOF > overflow.c
#include "lua.h"
#include "lualib.h"
#include "lauxlib.h"

int main() {
        lua_State *L = luaL_newstate();
        luaL_openlibs(L);
        /* Do not change the file name. */
        luaL_loadfilex(L, "happy_new_year_2026.lua", NULL);
        lua_call(L, 0, 0);
        lua_settop(L, 0);
        lua_close(L);

        return 0;
}
EOF

5. Build: clang overflow.c -O0 -fsanitize=signed-integer-overflow -o overflow -fno-sanitize-recover=signed-integer-overflow -I. -L. -llua -lm
6. Download a happy_new_year_2026.lua (also attached): curl -O https://gist.githubusercontent.com/ligurio/33ac73e68d47b5244f08bbbf0e900d34/raw/a0e7ca81258cd047e673ae3a187ddd3a7e9a9ddc/happy_new_year_2026.lua
7. Execute: ./overflow
lapi.c:1208:33: runtime error: signed integer overflow: -4 - 9223372036854775807 cannot be represented in type 'l_mem' (aka 'long')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lapi.c:1208:33

Sergey

[Roberto Ierusalimschy]

> An undefined behavior (signed integer overflow) is happening on execution

> of a Lua chunk below. [...]

Many thanks for the feedback. A simpler example is this:

----------------------------------------------------------------------
-- Lua must be compiled with '-fsanitize=undefined'

-- allocate some space to make gc debt negative (stack space does not
-- invoke GC), and then call 'collectgarbage"step"' with a huge step.
local function deep (n)
if n == 0 then
collectgarbage("step", math.maxinteger)
else
deep(n - 1)
end
end

deep(26)
----------------------------------------------------------------------

BTW, the issue is older than commit 85a3c1699.

-- Roberto