An assertion is triggered in the luaG_runerror()

[Sergey Bronnikov]

Hello,

an assertion is triggered on execution of a Lua chunk below:

./lua -e "
> local obj = setmetatable({}, {
  __index = function(l, ...)
        local _ = l > 1
  end
})
obj:method();
> "
lua: ldebug.c:865: void luaG_runerror(lua_State *, const char *, ...): Assertion `(((((&((ci)->func.p)->val)))->tt_) == (((((6) | ((0) << 4))) | (1 << 6))))' failed.
Aborted (core dumped)

Reproduced on the latest version of Lua (53b41d0cddd80bf33fdc631bdd32e3ba53842b89)
that built with enabled macro -DLUAI_ASSERT. The assertion started appearing after the following commit:

commit 36c1f6d949a4d3dfcbe898d80b1be1efe8e5325c
Author: Roberto I <rob...@inf.puc-rio.br>
Date:   Wed Apr 29 15:17:55 2026 -0300

    Small correction in luaP_opmodes
   
    OP_VARARGPREP neither sets nor uses L->top.

How to reproduce:

make -j MYCFLAGS="-O0 -DLUAI_ASSERT -ggdb"

Full backtrace:

(gdb) bt                                                                                                        
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78            
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89          
#3  0x00007ffff7c4527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26            
#4  0x00007ffff7c288ff in __GI_abort () at ./stdlib/abort.c:79                                                  
#5  0x00007ffff7c2881b in __assert_fail_base (fmt=0x7ffff7dd01e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x5555555ba666 "(((((&((ci)->func.p)->val)))->tt_) == (((((6) | ((0) << 4))) | (1 <
< 6))))", file=file@entry=0x5555555ba634 "ldebug.c", line=line@entry=865,                
    function=function@entry=0x5555555baa2f "void luaG_runerror(lua_State *, const char *, ...)")              
    at ./assert/assert.c:96                                                                                      
#6  0x00007ffff7c3b517 in __assert_fail (                                                                        
    assertion=0x5555555ba666 "(((((&((ci)->func.p)->val)))->tt_) == (((((6) | ((0) << 4))) | (1 << 6))))",    
    file=0x5555555ba634 "ldebug.c", line=865,                                                                    
    function=0x5555555baa2f "void luaG_runerror(lua_State *, const char *, ...)") at ./assert/assert.c:105
#7  0x0000555555567724 in luaG_runerror (L=0x5555555d6828, fmt=0x5555555ba980 "attempt to compare %s with %s")  
    at ldebug.c:865  
#8  0x0000555555567a88 in luaG_ordererror (L=0x5555555d6828, p1=0x7fffffffa620, p2=0x5555555d69d0)
    at ldebug.c:813
#9  0x0000555555586c78 in luaT_callorderTM (L=0x5555555d6828, p1=0x7fffffffa620, p2=0x5555555d69d0, event=TM_LT)
    at ltm.c:205
#10 0x0000555555586d12 in luaT_callorderiTM (L=0x5555555d6828, p1=0x7fffffffa620, v2=1, flip=1, isfloat=0,
    event=TM_LT) at ltm.c:223
#11 0x00005555555993e8 in luaV_execute (L=0x5555555d6828, ci=0x5555555dd010) at lvm.c:1696
#12 0x000055555556c015 in ccall (L=0x5555555d6828, func=0x5555555d69b0, nResults=1, inc=1) at ldo.c:774
#13 0x000055555556bf08 in luaD_call (L=0x5555555d6828, func=0x5555555d69b0, nResults=1) at ldo.c:784
#14 0x000055555558679e in luaT_callTMres (L=0x5555555d6828, f=0x5555555dc9a0, p1=0x5555555d6970,
    p2=0x5555555dc970, res=0x5555555d6980) at ltm.c:129
#15 0x000055555558a17f in luaV_finishget (L=0x5555555d6828, t=0x5555555d6970, key=0x5555555dc970,
    val=0x5555555d6980, tag=32 ' ') at lvm.c:312
#16 0x0000555555591714 in luaV_execute (L=0x5555555d6828, ci=0x5555555d7bd0) at lvm.c:1437
#17 0x000055555556c015 in ccall (L=0x5555555d6828, func=0x5555555d6950, nResults=0, inc=65537) at ldo.c:774
#18 0x000055555556c058 in luaD_callnoyield (L=0x5555555d6828, func=0x5555555d6950, nResults=0) at ldo.c:792
#19 0x0000555555563d4f in f_call (L=0x5555555d6828, ud=0x7fffffffcc28) at lapi.c:1071
#20 0x0000555555569d59 in luaD_rawrunprotected (L=0x5555555d6828, f=0x555555563d20 <f_call>, ud=0x7fffffffcc28)
    at ldo.c:166
#21 0x000055555556cbc1 in luaD_pcall (L=0x5555555d6828, func=0x555555563d20 <f_call>, u=0x7fffffffcc28,
    old_top=80, ef=64) at ldo.c:1096
#22 0x0000555555563bfb in lua_pcallk (L=0x5555555d6828, nargs=0, nresults=0, errfunc=3, ctx=0, k=0x0)
    at lapi.c:1097
#23 0x000055555555c35a in docall (L=0x5555555d6828, narg=0, nres=0) at lua.c:168
#24 0x000055555555c2c2 in dochunk (L=0x5555555d6828, status=0) at lua.c:204
#25 0x000055555555c28b in dostring (L=0x5555555d6828,
    s=0x7fffffffd7e1 "\nlocal obj = setmetatable({}, {\n  __index = function(l, ...)\n        local _ = l > 1\n  
end\n})\nobj:method();\n", name=0x5555555b738e "=(command line)") at lua.c:215
#26 0x000055555555c023 in runargs (L=0x5555555d6828, argv=0x7fffffffd248, n=3) at lua.c:369
#27 0x000055555555b8cf in pmain (L=0x5555555d6828) at lua.c:757
#28 0x000055555556b65a in precallC (L=0x5555555d6828, func=0x5555555d6910, status=2, f=0x55555555b770 <pmain>)
    at ldo.c:663
#29 0x000055555556bbb8 in luaD_precall (L=0x5555555d6828, func=0x5555555d6910, nresults=1) at ldo.c:732
#30 0x000055555556bfee in ccall (L=0x5555555d6828, func=0x5555555d6910, nResults=1, inc=65537) at ldo.c:772
#31 0x000055555556c058 in luaD_callnoyield (L=0x5555555d6828, func=0x5555555d6910, nResults=1) at ldo.c:792
#32 0x0000555555563d4f in f_call (L=0x5555555d6828, ud=0x7fffffffd0a8) at lapi.c:1071
#33 0x0000555555569d59 in luaD_rawrunprotected (L=0x5555555d6828, f=0x555555563d20 <f_call>, ud=0x7fffffffd0a8)
    at ldo.c:166
#34 0x000055555556cbc1 in luaD_pcall (L=0x5555555d6828, func=0x555555563d20 <f_call>, u=0x7fffffffd0a8,
    old_top=16, ef=0) at ldo.c:1096
#35 0x0000555555563bfb in lua_pcallk (L=0x5555555d6828, nargs=2, nresults=1, errfunc=0, ctx=0, k=0x0)
    at lapi.c:1097
#36 0x000055555555b68b in main (argc=3, argv=0x7fffffffd248) at lua.c:788

Sergey

[Roberto Ierusalimschy]

> an assertion is triggered on execution of a Lua chunk below:
>
> ./lua -e "
> > local obj = setmetatable({}, {
> __index = function(l, ...)
> local _ = l > 1
> end
> })
> obj:method();
> > "
> lua: ldebug.c:865: void luaG_runerror(lua_State *, const char *, ...):
> Assertion `(((((&((ci)->func.p)->val)))->tt_) == (((((6) | ((0) << 4))) |
> (1 << 6))))' failed.
> Aborted (core dumped)

Thanks for the feedback. Indeed, the commit 36c1f6d was wrong:
Although OP_VARARGPREP does not need to be preceded by an "OT"
instruction (it is never preceded by any instruction), it does need
a correct L->top to work properly. (The comment in lopcodes.h:420
is misleading.)

-- Roberto