luaL_traceback raising an api check assertion

[Edoardo Lolletti]

Hi everyone,
I use luaL_traceback to get the stacktrace of an error whenever it
occurs, today, while debugging some things, I noticed that with api
check assertions enabled, calling luaL_traceback on a coroutine's stack
triggered this assertion:
> lapi.c:531: (L->top.p <= L->ci->top.p) && "stack overflow"

On my end I tried also calling luaL_checkstack with 1 element before
this call, since the function is expected to push 1 result:
> luaL_checkstack(L, 1, nullptr);
> luaL_traceback(L, L, nullptr, 1);
but the assertion is still trigered.

By looking at lua code, i noticed that in lua 5.3 luaL_traceback is
calling checkstack on its own with 10 elements, before doing anything,
and in fact changing my checkstack call to check for 10 elements makes
the assertion not trigger.


Edoardo

[Xmilia Hermit]

> On my end I tried also calling luaL_checkstack with 1 element before
> this call, since the function is expected to push 1 result:
> > luaL_checkstack(L, 1, nullptr);
> > luaL_traceback(L, L, nullptr, 1);
> but the assertion is still trigered.

Calling functions from the auxiliary library requires 5 extra stack
slots, not one. Even if the function only returns one argument it might
require more stack slots while running.
From https://www.lua.org/manual/5.4/manual.html#5:
Several functions in the auxiliary library use internally some extra
stack slots. When a function in the auxiliary library uses less than
five slots, it does not check the stack size; it simply assumes that
there are enough slots.

Regards,
Xmilia

[Edoardo Lolletti]

I changed my code to account for the 5 extra stack slots, and things
seemed to work fine, but now i'm still getting those assertions even
with those explicit checks, in a different place this time

> ocgcore_lua_api_check(void * state, const char * error_message)
> lua_next(lua_State * L, int idx) Riga 1261 C++
> findfield(lua_State * L, int objidx, int level) Riga 56 C++
> findfield(lua_State * L, int objidx, int level) Riga 62 C++
> pushglobalfuncname(lua_State * L, lua_Debug * ar) Riga 83 C++
> pushfuncname(lua_State * L, lua_Debug * ar) Riga 101 C++
> luaL_traceback(lua_State * L, lua_State * L1, const char * msg, int
level) Riga 158 C++
> interpreter::print_stacktrace(lua_State * L) Riga 602 C++

This is how luaL_traceback is being called
> luaL_checkstack(L, 6, nullptr);
> luaL_traceback(L, L, nullptr, 1);

The raised assertion is
c:\edopro\ygopro-master\lua5.4\lapi.c:1261: (L->top.p <= L->ci->top.p)
&& "stack overflow


This is happening on lua 5.4.6, i haven't tested with the current 5.4.7
rc but i doubt it'll have any impact

[Roberto Ierusalimschy]

> I changed my code to account for the 5 extra stack slots, and things seemed
> to work fine, but now i'm still getting those assertions even with those
> explicit checks, in a different place this time

Indeed, luaL_traceback may need more than 5 extra stack slots, due
to the recursion in 'findfield'. We will fix that. Thanks for the
feedback.

-- Roberto