Undefined behavior in luaO_applyparam()

[Sergey Bronnikov]

Hello,

there is an issue with undefined behavior in lobject.c:

lobject.c:99:22: runtime error: left shift of negative value -2070
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lobject.c:99:22

How to reproduce:

1. git clone https://github.com/lua/lua
2. cd lua

3. make all CC=clang MYCFLAGS="-std=c99 -DLUA_USE_LINUX -m32 -fsanitize=undefined -fno-sanitize-recover=all -O1 -g" MYLDFLAGS="-m32 -fsanitize=undefined -g" MYLIBS="-lm"

4. create a file with Lua chunk:
cat << EOF > gc.lua
local unpack = unpack or table.unpack
local function band(x, ...)
    local chunk = "return " .. x
    for _, v in ipairs({...}) do chunk = chunk .. " & " .. v end
    return assert(load(chunk))()
end

local function build_table(n)
    local t = {}
    for i = 1, n do table.insert(t, i) end
    return t
end

local function fn(param, val)
    local data = {}
    local _ = { data }
    collectgarbage("param", param, val)
    local band_args = build_table(450)
    local _ = band(unpack(band_args))
end

fn("majorminor", 3054)
fn("pause", 65)
fn("pause", 65)

EOF

5. ./lua gc.lua
lobject.c:99:22: runtime error: left shift of negative value -2070
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lobject.c:99:22

PUC Rio Lua 5.5.1, 7579fc9d7ed90240487251dfb69168f8e64e9294

Sergey

[Roberto Ierusalimschy]

> there is an issue with undefined behavior in lobject.c:
>
> lobject.c:99:22: runtime error: left shift of negative value -2070
> SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lobject.c:99:22

Many thanks for the feedback. I was unable to reproduce the bug, but
I suspect the problem is that luaO_applyparam is being called with
a negative 'x'. Can you check that? You can make the following change
in lobject.c:

-----------------------------------------------
l_mem luaO_applyparam (lu_byte p, l_mem x) {
int m = p & 0xF; /* mantissa */
int e = (p >> 4); /* exponent */
+ if (x < 0) abort();
if (e > 0) { /* normalized? */
e--; /* correct exponent */
m += 0x10; /* correct mantissa; maximum value is 0x1F */
-----------------------------------------------

If that is the case, could you check who is doing that call?
(I suspect it is 'checkmajorminor'.)

Thanks again,

-- Roberto

[Sergey Bronnikov]

Hello, Roberto,

right, before the runtime error the function luaO_applyparam() is called with x == -69.

The full backtrace when running under UBSan is below:

lobject.c:99:22: runtime error: left shift of negative value -2070

    #0 0x60dc400f in luaO_applyparam /home/sergeyb/sources/cache/lua/lobject.c:99
    #1 0x60dc1e18 in checkmajorminor /home/sergeyb/sources/cache/lua/lgc.c:1475
    #2 0x60dc1e18 in singlestep /home/sergeyb/sources/cache/lua/lgc.c:1640
    #3 0x60dc2bcd in incstep /home/sergeyb/sources/cache/lua/lgc.c:1716
    #4 0x60dc2bcd in luaC_step /home/sergeyb/sources/cache/lua/lgc.c:1751
    #5 0x60de3cae in luaV_execute /home/sergeyb/sources/cache/lua/lvm.c:1431
    #6 0x60db9c8f in ccall /home/sergeyb/sources/cache/lua/ldo.c:768
    #7 0x60db9f26 in luaD_callnoyield /home/sergeyb/sources/cache/lua/ldo.c:786
    #8 0x60da9a79 in f_call /home/sergeyb/sources/cache/lua/lapi.c:1071
    #9 0x60db69e0 in luaD_rawrunprotected /home/sergeyb/sources/cache/lua/ldo.c:166
    #10 0x60dba64b in luaD_pcall /home/sergeyb/sources/cache/lua/ldo.c:1090
    #11 0x60daf8d9 in lua_pcallk /home/sergeyb/sources/cache/lua/lapi.c:1097
    #12 0x60da6915 in docall /home/sergeyb/sources/cache/lua/lua.c:168
    #13 0x60da80fc in handle_script /home/sergeyb/sources/cache/lua/lua.c:272
    #14 0x60da80fc in pmain /home/sergeyb/sources/cache/lua/lua.c:760
    #15 0x60db8b56 in precallC /home/sergeyb/sources/cache/lua/ldo.c:657
    #16 0x60db97f7 in luaD_precall /home/sergeyb/sources/cache/lua/ldo.c:726
    #17 0x60db9c5d in ccall /home/sergeyb/sources/cache/lua/ldo.c:766
    #18 0x60db9f26 in luaD_callnoyield /home/sergeyb/sources/cache/lua/ldo.c:786
    #19 0x60da9a79 in f_call /home/sergeyb/sources/cache/lua/lapi.c:1071
    #20 0x60db69e0 in luaD_rawrunprotected /home/sergeyb/sources/cache/lua/ldo.c:166
    #21 0x60dba64b in luaD_pcall /home/sergeyb/sources/cache/lua/ldo.c:1090
    #22 0x60daf8d9 in lua_pcallk /home/sergeyb/sources/cache/lua/lapi.c:1097
    #23 0x60da850b in main /home/sergeyb/sources/cache/lua/lua.c:788
    #24 0xee21ccb8  (/lib/i386-linux-gnu/libc.so.6+0x24cb8) (BuildId: a7e93008cd7dad7a713e3eaf1f9dec78ce7c68a6)
    #25 0xee21cd7b in __libc_start_main (/lib/i386-linux-gnu/libc.so.6+0x24d7b) (BuildId: a7e93008cd7dad7a713e3eaf1f9dec78ce7c68a6)
    #26 0x60da6686 in _start (/home/sergeyb/sources/cache/lua/lua+0x1a686) (BuildId: e4609705508d5ce7a2ee82619587d922a9213bc7)

The backtrace with applied patch:

#1  0xf77c1df7 in ?? () from /lib/i386-linux-gnu/libc.so.6
#2  0xf776f0b5 in raise () from /lib/i386-linux-gnu/libc.so.6
#3  0xf77562bf in abort () from /lib/i386-linux-gnu/libc.so.6
#4  0x5658cfb9 in luaO_applyparam (p=142 '\216', x=-69) at lobject.c:93
#5  0x5658ae19 in checkmajorminor (g=0x566393b0, L=0x56639688) at lgc.c:1475
#6  singlestep (L=L@entry=0x56639688, fast=fast@entry=0) at lgc.c:1640
#7  0x5658bbce in incstep (g=0x566393b0, L=0x56639688) at lgc.c:1716
#8  luaC_step (L=0x56639688) at lgc.c:1751
#9  0x565accba in luaV_execute (L=0x56639688, ci=0x5663ff70) at lvm.c:1431
#10 0x56582c90 in ccall (L=L@entry=0x56639688, func=func@entry=0x5663974c, nResults=nResults@entry=-1, inc=65537) at ldo.c:768
#11 0x56582f27 in luaD_callnoyield (L=0x56639688, func=0x5663974c, nResults=-1) at ldo.c:786
#12 0x56572a7a in f_call (L=0x56639688, ud=0xffffc1b8) at lapi.c:1071
#13 0x5657f9e1 in luaD_rawrunprotected (L=0x56639688, f=0x56572a46 <f_call>, ud=0xffffc1b8) at ldo.c:166
#14 0x5658364c in luaD_pcall (L=0x56639688, func=0x56572a46 <f_call>, u=0xffffc1b8, old_top=60, ef=48) at ldo.c:1090
#15 0x565788da in lua_pcallk (L=0x56639688, nargs=0, nresults=-1, errfunc=3, ctx=0, k=0x0) at lapi.c:1097
#16 0x5656f916 in docall (L=L@entry=0x56639688, narg=narg@entry=0, nres=nres@entry=-1) at lua.c:168
#17 0x565710fd in handle_script (argv=<optimized out>, L=0x56639688) at lua.c:272
#18 pmain (L=<optimized out>) at lua.c:760
#19 0x56581b57 in precallC (L=L@entry=0x56639688, func=func@entry=0x5663971c, status=status@entry=2, f=0x5657033d <pmain>)
    at ldo.c:657
#20 0x565827f8 in luaD_precall (L=0x56639688, func=0x5663971c, nresults=1) at ldo.c:726
#21 0x56582c5e in ccall (L=L@entry=0x56639688, func=func@entry=0x5663971c, nResults=nResults@entry=1, inc=65537) at ldo.c:766
#22 0x56582f27 in luaD_callnoyield (L=0x56639688, func=0x5663971c, nResults=1) at ldo.c:786
#23 0x56572a7a in f_call (L=0x56639688, ud=0xffffc4e8) at lapi.c:1071
#24 0x5657f9e1 in luaD_rawrunprotected (L=0x56639688, f=0x56572a46 <f_call>, ud=0xffffc4e8) at ldo.c:166
#25 0x5658364c in luaD_pcall (L=0x56639688, func=0x56572a46 <f_call>, u=0xffffc4e8, old_top=12, ef=0) at ldo.c:1090
#26 0x565788da in lua_pcallk (L=0x56639688, nargs=2, nresults=1, errfunc=0, ctx=0, k=0x0) at lapi.c:1097
#27 0x5657150c in main (argc=2, argv=0xffffc624) at lua.c:788
(gdb)

Sergey