Assertion failure of `A <= ((1<<8)-1) && B <= ((1<<8)-1) && C <= ((1<<8)-1) && (k & ~1) == 0'

[jy l]

Hi, we found a crashing test case using the fuzzing driver in https://github.com/ligurio/lua-c-api-tests/

The compilation flag:

```

make CFLAGS="-DLUA_USE_APICHECK -DLUAI_ASSERT " -j

```

The reproduction:
```

./lua -e 'load("local " .. string.rep("_,", 256) .. "_")'
lua: lcode.c:400: luaK_codeABCk: Assertion `A <= ((1<<8)-1) && B <= ((1<<8)-1) && C <= ((1<<8)-1) && (k & ~1) == 0' failed.
Aborted (core dumped)

```

[Philippe Verdy]

The assertion made in locode.c may be the fault on the value of k (which is just expected to be a boolean condition, and where the assertion just checks the least significant bit of k.

For me it looks like this is an "UB" in C, and the assertion is probably broken):

  lua_assert(a <= MAXARG_A && b <= MAXARG_B &&
             c <= MAXARG_C && (k & ~1) == 0);

which should just be (without the one bit restriction of k):

  lua_assert(a <= MAXARG_A && b <= MAXARG_B &&
             c <= MAXARG_C && !k);
Is there a possibility that the C compiler may compile/optimize computed conditional values using sometimes the value -1 (or any other non-zero value) for true, assuming that boolean checks should just check zero/non-zero integer value and not specific bits?

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/lua-l/46e16b30-8658-42d9-b9b3-79a9d3bc11den%40googlegroups.com.

[Philippe Verdy]

Also I note this function:

void luaK_storevar (FuncState *fs, expdesc *var, expdesc *ex) {...}

which uses this switch values:

   case VINDEXUP: {
      codeABRK(fs, OP_SETTABUP, var->u.ind.t, var->u.ind.idx, ex);
      break;
    }
    case VINDEXI: {
      codeABRK(fs, OP_SETI, var->u.ind.t, var->u.ind.idx, ex);
      break;
    }
    case VINDEXSTR: {
      codeABRK(fs, OP_SETFIELD, var->u.ind.t, var->u.ind.idx, ex);
      break;
    }
    case VINDEXED: {
      codeABRK(fs, OP_SETTABLE, var->u.ind.t, var->u.ind.idx, ex);
      break;
where the value of ex can be any arbitrary computed integer from exp2RK() and does not seem to be restricted to boolean values 0 and 1, it could be any other even integer generating the assertion failure in luaK_codeABCk():

static void codeABRK (FuncState *fs, OpCode o, int a, int b,
                      expdesc *ec) {
  int k = exp2RK(fs, ec);
  luaK_codeABCk(fs, o, a, b, ec->u.info, k);
}

[Roberto Ierusalimschy]

> ./lua -e 'load("local " .. string.rep("_,", 256) .. "_")'
> lua: lcode.c:400: luaK_codeABCk: Assertion `A <= ((1<<8)-1) && B <=
> ((1<<8)-1) && C <= ((1<<8)-1) && (k & ~1) == 0' failed.
> Aborted (core dumped)
> ```

Thanks for the report.

-- Roberto