An assertion is triggered in luaG_errormsg

[Sergey Bronnikov]

Hello,

an assertion is triggered on execution of a Lua chunk below:

local chunk = [[
_G:xpcall(function()
  t(debug.setlocal(4, 2, d))
end)
]]
(load(chunk))()

$ ./lua dostring.lua
lua: ldebug.c:835: luaG_errormsg: Assertion `(((((((&(errfunc)->val))->tt_)) & 0x0F)) == (6))' failed.
Aborted (core dumped)

Reproduced on the latest version of Lua (fca974486d12aa29bb6d731fdb5b25055157ece8)
that built with enabled macro -DLUAI_ASSERT.

How to reproduce:

--- a/makefile
+++ b/makefile
@@ -71,7 +71,7 @@ LOCAL = $(TESTS) $(CWARNS)
 # To enable Linux goodies, -DLUA_USE_LINUX
 # For C89, "-std=c89 -DLUA_USE_C89"
 # Note that Linux/Posix options are not compatible with C89
-MYCFLAGS= $(LOCAL) -std=c99 -DLUA_USE_LINUX
+MYCFLAGS= $(LOCAL) -std=c99 -DLUA_USE_LINUX -DLUAI_ASSERT -Og -g
 MYLDFLAGS= $(LOCAL) -Wl,-E
 MYLIBS= -ldl
 
CFLAGS=-ggdb make -j
 
Full backtrace:

#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c4527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c2881b in __assert_fail_base (fmt=0x7ffff7dd01e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x555555595928 "(((((((&(errfunc)->val))->tt_)) & 0x0F)) == (6))",
    file=file@entry=0x5555555913f5 "ldebug.c", line=line@entry=835,
    function=function@entry=0x55555559b3c0 <__PRETTY_FUNCTION__.3> "luaG_errormsg") at ./assert/assert.c:96
#6  0x00007ffff7c3b517 in __assert_fail (
    assertion=assertion@entry=0x555555595928 "(((((((&(errfunc)->val))->tt_)) & 0x0F)) == (6))",
    file=file@entry=0x5555555913f5 "ldebug.c", line=line@entry=835,
    function=function@entry=0x55555559b3c0 <__PRETTY_FUNCTION__.3> "luaG_errormsg") at ./assert/assert.c:105
#7  0x00005555555632f6 in luaG_errormsg (L=L@entry=0x5555555af828) at ldebug.c:835
#8  0x0000555555563626 in luaG_runerror (L=L@entry=0x5555555af828,
    fmt=fmt@entry=0x5555555914af "attempt to %s a %s value%s") at ldebug.c:861
#9  0x0000555555563674 in typeerror (L=L@entry=0x5555555af828, o=o@entry=0x5555555b60e0,
    op=op@entry=0x555555592c09 "call", extra=0x5555555b63e8 " (global 't')") at ldebug.c:749
#10 0x000055555556373f in luaG_callerror (L=L@entry=0x5555555af828, o=o@entry=0x5555555b60e0) at ldebug.c:772
#11 0x0000555555563dc4 in tryfuncTM (L=L@entry=0x5555555af828, func=func@entry=0x5555555b60e0, status=status@entry=1)
    at ldo.c:517
#12 0x0000555555565397 in luaD_precall (L=L@entry=0x5555555af828, func=0x5555555b60e0, nresults=<optimized out>)
    at ldo.c:731
#13 0x000055555557d514 in luaV_execute (L=L@entry=0x5555555af828, ci=0x5555555b64d0) at lvm.c:1716
#14 0x00005555555655f2 in ccall (L=L@entry=0x5555555af828, func=0x5555555b60d0, nResults=nResults@entry=1,
    inc=inc@entry=65537) at ldo.c:755
#15 0x0000555555565782 in luaD_callnoyield (L=L@entry=0x5555555af828, func=<optimized out>,
    nResults=nResults@entry=1) at ldo.c:773
#16 0x000055555556339e in luaG_errormsg (L=L@entry=0x5555555af828) at ldebug.c:839
#17 0x0000555555563626 in luaG_runerror (L=L@entry=0x5555555af828,
    fmt=fmt@entry=0x5555555914af "attempt to %s a %s value%s") at ldebug.c:861
#18 0x0000555555563674 in typeerror (L=L@entry=0x5555555af828, o=o@entry=0x5555555b60a0,
    op=op@entry=0x555555592c09 "call", extra=0x5555555b63e8 " (global 't')") at ldebug.c:749
#19 0x000055555556373f in luaG_callerror (L=L@entry=0x5555555af828, o=o@entry=0x5555555b60a0) at ldebug.c:772
#20 0x0000555555563dc4 in tryfuncTM (L=L@entry=0x5555555af828, func=func@entry=0x5555555b60a0, status=status@entry=1)
    at ldo.c:517
#21 0x0000555555565397 in luaD_precall (L=L@entry=0x5555555af828, func=0x5555555b60a0, nresults=<optimized out>)
    at ldo.c:731
#22 0x000055555557d514 in luaV_execute (L=L@entry=0x5555555af828, ci=0x5555555b6380) at lvm.c:1716
#23 0x00005555555655f2 in ccall (L=L@entry=0x5555555af828, func=0x5555555b6090, nResults=nResults@entry=1,
    inc=inc@entry=65537) at ldo.c:755
#24 0x0000555555565782 in luaD_callnoyield (L=L@entry=0x5555555af828, func=<optimized out>,
    nResults=nResults@entry=1) at ldo.c:773
#25 0x000055555556339e in luaG_errormsg (L=L@entry=0x5555555af828) at ldebug.c:839
#26 0x0000555555563626 in luaG_runerror (L=L@entry=0x5555555af828,
    fmt=fmt@entry=0x5555555914af "attempt to %s a %s value%s") at ldebug.c:861
#27 0x0000555555563674 in typeerror (L=L@entry=0x5555555af828, o=o@entry=0x5555555b6060,
    op=op@entry=0x555555592c09 "call", extra=0x5555555b63e8 " (global 't')") at ldebug.c:749
#28 0x000055555556373f in luaG_callerror (L=L@entry=0x5555555af828, o=o@entry=0x5555555b6060) at ldebug.c:772
#29 0x0000555555563dc4 in tryfuncTM (L=L@entry=0x5555555af828, func=func@entry=0x5555555b6060, status=status@entry=1)
    at ldo.c:517
#30 0x0000555555565397 in luaD_precall (L=L@entry=0x5555555af828, func=0x5555555b6060, nresults=<optimized out>)
    at ldo.c:731
#31 0x000055555557d514 in luaV_execute (L=L@entry=0x5555555af828, ci=0x5555555b5f10) at lvm.c:1716
#32 0x00005555555655f2 in ccall (L=L@entry=0x5555555af828, func=0x5555555af9f0, nResults=nResults@entry=1,
    inc=inc@entry=65537) at ldo.c:755
#33 0x0000555555565782 in luaD_callnoyield (L=L@entry=0x5555555af828, func=<optimized out>,
    nResults=nResults@entry=1) at ldo.c:773
#34 0x000055555556339e in luaG_errormsg (L=L@entry=0x5555555af828) at ldebug.c:839
#35 0x0000555555563626 in luaG_runerror (L=L@entry=0x5555555af828,
    fmt=fmt@entry=0x5555555914af "attempt to %s a %s value%s") at ldebug.c:861
#36 0x0000555555563674 in typeerror (L=L@entry=0x5555555af828, o=o@entry=0x5555555af9e0,
    op=op@entry=0x555555592c09 "call", extra=0x5555555932af "") at ldebug.c:749
#37 0x000055555556373f in luaG_callerror (L=L@entry=0x5555555af828, o=o@entry=0x5555555af9e0) at ldebug.c:772
#38 0x0000555555563dc4 in tryfuncTM (L=L@entry=0x5555555af828, func=func@entry=0x5555555af9e0, status=status@entry=0)
    at ldo.c:517
#39 0x0000555555565397 in luaD_precall (L=L@entry=0x5555555af828, func=func@entry=0x5555555af9e0,
    nresults=nresults@entry=-1) at ldo.c:731
#40 0x00005555555655d7 in ccall (L=L@entry=0x5555555af828, func=0x5555555af9e0, nResults=-1, inc=inc@entry=65537)
    at ldo.c:753
#41 0x0000555555565782 in luaD_callnoyield (L=L@entry=0x5555555af828, func=<optimized out>, nResults=<optimized out>)
    at ldo.c:773
#42 0x000055555555dadf in f_call (L=L@entry=0x5555555af828, ud=ud@entry=0x7fffffffcaf0) at lapi.c:1071
#43 0x0000555555564473 in luaD_rawrunprotected (L=L@entry=0x5555555af828, f=f@entry=0x55555555dacc <f_call>,
    ud=ud@entry=0x7fffffffcaf0) at ldo.c:166
#44 0x0000555555565bb2 in luaD_pcall (L=L@entry=0x5555555af828, func=func@entry=0x55555555dacc <f_call>,
    u=u@entry=0x7fffffffcaf0, old_top=224, ef=<optimized out>) at ldo.c:1077
#45 0x00005555555608f0 in lua_pcallk (L=L@entry=0x5555555af828, nargs=nargs@entry=0, nresults=nresults@entry=-1,
    errfunc=errfunc@entry=2, ctx=ctx@entry=2, k=k@entry=0x555555585aac <finishpcall>) at lapi.c:1097
#46 0x0000555555585b61 in luaB_xpcall (L=0x5555555af828) at lbaselib.c:503
#47 0x0000555555564e3f in precallC (L=L@entry=0x5555555af828, func=func@entry=0x5555555af9a0,
    status=<optimized out>, status@entry=1, f=0x555555585aed <luaB_xpcall>) at ldo.c:644
#48 0x000055555556543b in luaD_precall (L=L@entry=0x5555555af828, func=0x5555555af9a0, nresults=<optimized out>)
    at ldo.c:713
#49 0x000055555557d514 in luaV_execute (L=L@entry=0x5555555af828, ci=0x5555555b6a80) at lvm.c:1716
#50 0x00005555555655f2 in ccall (L=L@entry=0x5555555af828, func=0x5555555af950, nResults=-1, inc=inc@entry=65537)
    at ldo.c:755
#51 0x0000555555565782 in luaD_callnoyield (L=L@entry=0x5555555af828, func=<optimized out>, nResults=<optimized out>)
    at ldo.c:773
#52 0x000055555555dadf in f_call (L=L@entry=0x5555555af828, ud=ud@entry=0x7fffffffce30) at lapi.c:1071
#53 0x0000555555564473 in luaD_rawrunprotected (L=L@entry=0x5555555af828, f=f@entry=0x55555555dacc <f_call>,
    ud=ud@entry=0x7fffffffce30) at ldo.c:166
#54 0x0000555555565bb2 in luaD_pcall (L=L@entry=0x5555555af828, func=func@entry=0x55555555dacc <f_call>,
    u=u@entry=0x7fffffffce30, old_top=80, ef=<optimized out>) at ldo.c:1077
#55 0x00005555555608f0 in lua_pcallk (L=L@entry=0x5555555af828, nargs=nargs@entry=0, nresults=nresults@entry=-1,
    errfunc=errfunc@entry=3, ctx=ctx@entry=0, k=k@entry=0x0) at lapi.c:1097
#56 0x000055555555c1ad in docall (L=L@entry=0x5555555af828, narg=0, nres=nres@entry=-1) at lua.c:162
#57 0x000055555555c68b in handle_script (L=L@entry=0x5555555af828, argv=0x7fffffffd2e0) at lua.c:266
#58 0x000055555555cd24 in pmain (L=0x5555555af828) at lua.c:731
#59 0x0000555555564e3f in precallC (L=L@entry=0x5555555af828, func=func@entry=0x5555555af910,
    status=<optimized out>, status@entry=2, f=0x55555555cb86 <pmain>) at ldo.c:644
#60 0x000055555556543b in luaD_precall (L=L@entry=0x5555555af828, func=func@entry=0x5555555af910,
    nresults=nresults@entry=1) at ldo.c:713
#61 0x00005555555655d7 in ccall (L=L@entry=0x5555555af828, func=0x5555555af910, nResults=1, inc=inc@entry=65537)
    at ldo.c:753
#62 0x0000555555565782 in luaD_callnoyield (L=L@entry=0x5555555af828, func=<optimized out>, nResults=<optimized out>)
    at ldo.c:773
#63 0x000055555555dadf in f_call (L=L@entry=0x5555555af828, ud=ud@entry=0x7fffffffd150) at lapi.c:1071
#64 0x0000555555564473 in luaD_rawrunprotected (L=L@entry=0x5555555af828, f=f@entry=0x55555555dacc <f_call>,
    ud=ud@entry=0x7fffffffd150) at ldo.c:166
#65 0x0000555555565bb2 in luaD_pcall (L=L@entry=0x5555555af828, func=func@entry=0x55555555dacc <f_call>,
    u=u@entry=0x7fffffffd150, old_top=16, ef=<optimized out>) at ldo.c:1077
#66 0x00005555555608f0 in lua_pcallk (L=L@entry=0x5555555af828, nargs=nargs@entry=2, nresults=nresults@entry=1,
    errfunc=errfunc@entry=0, ctx=ctx@entry=0, k=k@entry=0x0) at lapi.c:1097
#67 0x000055555555cdfa in main (argc=2, argv=0x7fffffffd2d8) at lua.c:759

Sergey

[Sainan]

Usage of debug library voids warranty.

-- Sainan

[Sergey Bronnikov]

Yeah, I remember.

I've just reported an issue found by fuzzing. I don't claim it is bug.