Bug in luaL_newmetatable: undersized table hint causes permanent corruption under OOM

[Payo Nel]

Hi Roberto,

I found a bug in luaL_newmetatable (lauxlib.c, line 318 on current master)

where the table creation hint of 2 is too small, causing an incomplete

metatable to persist in the registry after a memory allocation failure.

The problem:

luaL_newmetatable creates the metatable with lua_createtable(L, 0, 2)

(2 hash slots), sets the __name field, and registers it in the registry --

all before returning to the caller. The caller then typically adds

metamethods via luaL_setfuncs.

A concrete example is newbox() in lauxlib.c, which adds __gc and __close

to the _UBOX* metatable. With only 2 hash slots, the table fills up after

__name + __gc, and inserting __close triggers a rehash. If memory

allocation fails at that point:

1. The rehash throws LUA_ERRMEM

2. The metatable remains in the registry with only __name and __gc

   (no __close)

3. On the next attempt, luaL_newmetatable finds the metatable already

   in the registry and returns 0

4. The caller's if (luaL_newmetatable(...)) block is skipped --

   luaL_setfuncs is never called again

5. The __close metamethod is permanently missing

When lua_toclose is later called on the UBox, checkclosemth cannot find

__close and raises "variable '(C temporary)' got a non-closable value",

followed by "error in error handling".

Reproduction:

Build with the debug/test allocator (ltests.h) and run a memory stress

loop that calls load + string.dump + load while limiting allocations via

T.alloccount. The bug triggers consistently around 46 allocations allowed.

Fix:

--- a/lauxlib.c

+++ b/lauxlib.c

@@ -318,7 +318,7 @@

   if (luaL_getmetatable(L, tname) != LUA_TNIL)

     return 0;

   lua_pop(L, 1);

-  lua_createtable(L, 0, 2);  /* create metatable */

+  lua_createtable(L, 0, 4); /* create metatable (room for __name + methods) */

   lua_pushstring(L, tname);

   lua_setfield(L, -2, "__name");

   lua_pushvalue(L, -1);

Changing the hint from 2 to 4 gives the metatable enough hash slots to

accommodate __name plus the typical 2-3 metamethods without triggering a

rehash, eliminating the failure window.

Here is my repro:

```

if T==nil then return end

local MEMERRMSG = "not enough memory"

local function deep(n) if n > 0 then deep(n-1) end end

local testprog = [[

local t = {"x"}

AA = "aaa"

for i = 1, #t do AA = AA .. t[i] end

return true

]]

local f = function ()

  local a = load(testprog)

  local b = a and string.dump(a)

  a = b and load(b)

  return a and a()

end

-- testalloc phase

io.write("=== testalloc phase ===\n"); io.flush()

local M = 0

while true do

  collectgarbage(); collectgarbage()

  deep(4)

  io.write("alloc M=" .. M .. "\n"); io.flush()

  T.alloccount(M)

  local a, b = T.testC("pcall 0 1 0; pushstatus; return 2", f)

  T.alloccount()

  if a and b == "OK" then break end

  if b ~= "OK" and b ~= MEMERRMSG then

    io.write("  ERROR: " .. tostring(a) .. "\n"); io.flush()

    error(a, 0)

  end

  M = M + 1

end

io.write("alloc done at M=" .. M .. "\n"); io.flush()

-- testbytes phase

io.write("=== testbytes phase ===\n"); io.flush()

collectgarbage()

M = T.totalmem()

local oldM = M

while true do

  collectgarbage(); collectgarbage()

  deep(4)

  io.write("bytes M=" .. M .. "\n"); io.flush()

  T.totalmem(M)

  local a, b = T.testC("pcall 0 1 0; pushstatus; return 2", f)

  T.totalmem(0)

  if a and b == "OK" then break end

  if b ~= "OK" and b ~= MEMERRMSG then

    io.write("  ERROR: " .. tostring(a) .. "\n"); io.flush()

    error(a, 0)

  end

  M = M + 7

end

io.write("bytes done\n"); io.flush()

```

Running this will end with:

...

alloc M=46

  ERROR: variable '(C temporary)' got a non-closable value

./lua: error in error handling

Best regards,

Payo

[Roberto Ierusalimschy]

> I found a bug in luaL_newmetatable (lauxlib.c, line 318 on current master)
> where the table creation hint of 2 is too small, causing an incomplete
> metatable to persist in the registry after a memory allocation failure.
>
> The problem:
>

> [...]

Your diagnostic seems correct. Many thanks for the feedback.

-- Roberto

[Roberto Ierusalimschy]

> I found a bug in luaL_newmetatable (lauxlib.c, line 318 on current master)
> where the table creation hint of 2 is too small, causing an incomplete
> metatable to persist in the registry after a memory allocation failure.
>

> [...]

> Fix:
>
> --- a/lauxlib.c
> +++ b/lauxlib.c
> @@ -318,7 +318,7 @@
> if (luaL_getmetatable(L, tname) != LUA_TNIL)
> return 0;
> lua_pop(L, 1);
> - lua_createtable(L, 0, 2); /* create metatable */
> + lua_createtable(L, 0, 4); /* create metatable (room for __name + methods) */
> lua_pushstring(L, tname);
> lua_setfield(L, -2, "__name");
> lua_pushvalue(L, -1);
>
> Changing the hint from 2 to 4 gives the metatable enough hash slots to
> accommodate __name plus the typical 2-3 metamethods without triggering a
> rehash, eliminating the failure window.

The problem diagnosis is correct, but this fix is not. First, the
metatable may need more than 2 extra slots. Second, the creation of
the keys (strings) also can trigger allocation errors.

-- Roberto