Lua 5.4.4+ for loop stack overflow inside deep recursive function call

[John Bit]

Tested on Arch Linux (glibc 2.42, gcc 15.2.1 20251112, make 4.4.1).

Builds obtained from official download on website (/ftp/lua-X.Y.Z.tar.gz)

Following codeblock causes stack overflow on version 5.4.4 and further (up to current 5.5.0).

--------

local function f(n)
  if n == 0 then return end
  for _ in pairs({1}) do return f(n - 1) end
end

f(1e6)

--------

Produced error:

--------
lua: stackov.lua:3: stack overflow
stack traceback:
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        ...     (skipping 142833 levels)
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in upvalue 'f'
        stackov.lua:3: in local 'f'
        stackov.lua:6: in main chunk
        [C]: in ?

--------

Notably: since 5.5.0 instead of 142833 skipped levels, 166642 is skipped

On versions prior to 5.4.4 (5.4.3 and earlier) it does not produce any errors.

Probably it has something to do which changes introduced in 5.4.4 about to-be-closed variables and their logic/handling inside of nested functions or for loops.

Maybe it is similar case to "C-stack overflow with deep nesting of coroutine.close." issue reported by Xmilia Hermit on 13 Oct 2022.

[Philippe Verdy]

These are NOT trailing function calls (to finction f), because the function call is not returning immediately but continuing the loop. (https://www.lua.org/pil/6.3.html).

With 1 million recursive non-trailing function calls, it's quite normal you get a stack overflow, stacks are not infinite.

Stack frames for Lua-to-Lua function calls in 5.4.4 may just be a bit larger than they were in 5.4.3 (possibly yes to handle to-be-closed variables, even if there's no to-be-closed variable in this example, ther's at least some empty fields added in the stack frame. Note also that the for loop in that function uses an iterator, it is still not freed when making the non-trailing recursive function call, even if that iterator is terminated (the iterator here just runs once, may be there's a case to detect that the loop is terminated, so that the iterator variable is no longer needed even before initiating the function call (and confirmed by the fact the call is returning immediately so that any loop will be terminated at this return), and this would allow the trailing call logic (but the logic is broken now, and such possible prior optimization is no longer working, so stack frames are then created and cumulated).

But I doubt this is caused by the to-be-closed logic.

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/lua-l/15c6cf42-d992-4d14-8c05-8e43ff624688%40yandex.ru.

[Shmuel Zeigerman]

On 06/01/2026 15:18, Philippe Verdy wrote:

With 1 million recursive non-trailing function calls, it's quite normal you get a stack overflow, stacks are not infinite.

Tested in on Windows with Lua 5.1 through 5.4 and LuaJIT.
Lua 5.4 causes stack overflow when parameter 'n' is less than  200,000.
Other Lua / LuaJIT versions don't cause stack overflow even at n == 100,000,000.

-- 
Shmuel

[Ką Mykolas]

On Tue, Jan 6, 2026 at 3:42 PM Shmuel Zeigerman <sh...@013net.net> wrote:

Tested in on Windows with Lua 5.1 through 5.4 and LuaJIT.
Lua 5.4 causes stack overflow when parameter 'n' is less than  200,000.
Other Lua / LuaJIT versions don't cause stack overflow even at n == 100,000,000.

Interesting. Running luac5.3 -l -l -l  produces a TAILCALL instruction (and a bit less instructions in general):

1 param, 7 slots, 2 upvalues, 5 locals, 3 constants, 0 functions
1 [2] EQ       0 0 -1 ; - 0
2 [2] JMP       0 1 ; to 4
3 [2] RETURN   0 1
4 [3] GETTABUP 1 0 -2 ; _ENV "pairs"
5 [3] NEWTABLE 2 1 0
6 [3] LOADK     3 -3 ; 1
7 [3] SETLIST   2 1 1 ; 1
8 [3] CALL     1 2 4
9 [3] JMP       0 4 ; to 14
10 [3] GETUPVAL 5 1 ; f
11 [3] SUB       6 0 -3 ; - 1
12 [3] TAILCALL 5 2 0
13 [3] RETURN   5 0
14 [3] TFORCALL 1 1
15 [3] TFORLOOP 3 -6 ; to 10
16 [4] RETURN   0 1 

While luac5.4 -l -l -l produces no TAILCALLs:

1 param, 8 slots, 2 upvalues, 6 locals, 1 constant, 0 functions
1 [2] EQI       0 0 0
2 [2] JMP       1 ; to 4
3 [2] RETURN   1 1 0k ; 0 out
4 [3] GETTABUP 1 0 0 ; _ENV "pairs"
5 [3] NEWTABLE 2 0 1 ; 1
6 [3] EXTRAARG 0
7 [3] LOADI     3 1
8 [3] SETLIST   2 1 0
9 [3] CALL     1 2 5 ; 1 in 4 out
10 [3] TFORPREP 1 5 ; to 16
11 [3] GETUPVAL 6 1 ; f
12 [3] ADDI     7 0 -1
13 [3] MMBINI   0 1 7 0 ; __sub
14 [3] CALL     6 2 0 ; 1 in all out
15 [3] RETURN   6 0 0k ; all out
16 [3] TFORCALL 1 1
17 [3] TFORLOOP 1 7 ; to 11
18 [3] CLOSE     1
19 [4] RETURN   1 1 0k ; 0 out  

[Xmilia Hermit]

John Bit:

> local function f(n)
>   if n == 0 then return end
>   for _ in pairs({1}) do return f(n - 1) end
> end
>
> f(1e6)

This is expected. From the manual:

A call of the form return functioncall not in the scope of a
to-be-closed variable is called a tail call.

and

The loop starts by evaluating explist to produce four values: an
iterator function, a state, an initial value for the control variable,
and a closing value.
[...]
The closing value behaves like a to-be-closed variable (see §3.3.8),
which can be used to release resources when the loop ends. Otherwise, it
does not interfere with the loop.

So, the loop has a to-be-closed value and so the `return f(n - 1)` is
not a tail call.
This changed with the introduction of to-be-closed values.

Regards,
Xmilia