VM/GC question for gurus.

[Francisco Olarte]

Using embeded lua 5.4.8 ( cannot test 5.5.0 now )...

I have a C function published into lua, stripping irrelevant details looks like:

int mp_loadfilex(lua_State * L) {
  auto name = luaL_checkstring(L,1);
  int fd = ::open(name, O_RDONLY | O_NOATIME, 0);
 
  struct stat st;
  ::fstat(fd, &st);
 
  auto mm = ::mmap(nullptr, st.st_size, PROT_READ,
                   MAP_SHARED | MAP_POPULATE, fd, 0);
  auto mode = luaL_optstring(L,2, nullptr);
 
  int status = luaL_loadbufferx(L,(const char *)mm, st.st_size, name, mode);
 
  ::munmap(mm, st.st_size);
  ::close(fd);
 
  return 1;
}

Simple and easy, implement loadfile using mmap ( there is some extra code and reasons to go this way ).

Function works well but when benchmarking it by running it in a loop and comparing with loadfile my mem usage skyrocketed until I hit the Linux OOM killer.

It did not fail when doing a 100 repetitions loop ten times, but failed on a 1000 reps loop. 

Test file is just returning a big table with several subtables, 

wc-l kk*

   35092   217644  3285233 kk.lua
    1168     3853   799364 kk.luacs

I use the cs ( compiled stripped ) version, which according to luac has

main <kk.lua:0,0> (103696 instructions at 0x5c27e4036cc0)
0+ params, 7 slots, 1 upvalue, 0 locals, 28958 constants, 0 functions

Calling loadfile does not fail. Calling collectgarbage in the loop fix it, as does executing the result of loadfilex or just creating an empty table in the loop. Or doing a print.

Taking a look at 5.4.8 lvm.c and a couple other places lead me to think lua does not take into account the memory usage of the lua_load created function. Can this be true ?

If so, which would be the recommended way to avoid it? It will probably not happen in real code, where the call will always be evaluated and surrounded by lots of lua code, but I would prefer to put a properly commented lua_newtable, or lua_gc(LUAGC_STEP) instruction inside the C function for future proofing.

Francisco Olarte.

[Francisco Olarte]

Trying to clarify, when I say

On Wed, 18 Mar 2026 at 17:49, Francisco Olarte <fol...@peoplecall.com> wrote:

Taking a look at 5.4.8 lvm.c and a couple other places lea me to think lua does not take into account the memory usage of the lua_load created function. Can this be true ?

I do not mean it does not account it, I know it does. I mean it does not notice the memory usage has grown a lot due to the function being loaded, and as the test loop does not execute and makes it garabage, and has not any other memory consuming op it does not notice memory is growing and gc needs to be invoked ( I notice lvm.c has a checkGc macro which does this in some interesting point, but maybe the thight benchmark loop fails to hit any of them ).

Francisco Olarte.

[Roberto Ierusalimschy]

> > Taking a look at 5.4.8 lvm.c and a couple other places lea me to think lua
> > does not take into account the memory usage of the lua_load created
> > function. Can this be true ?
> >
>
> I do not mean it does not account it, I know it does. I mean it does not
> notice the memory usage has grown a lot due to the function being loaded,
> and as the test loop does not execute and makes it garabage, and has not
> any other memory consuming op it does not notice memory is growing and gc
> needs to be invoked ( I notice lvm.c has a checkGc macro which does this in
> some interesting point, but maybe the thight benchmark loop fails to hit
> any of them ).

You may be right; we will check that. luaL_loadfilex creates a new
string with the file name (calling 'lua_pushfstring'), and that call
checks the collector. But lua_load does not do that check.

-- Roberto

[Francisco Olarte]

On Wed, 18 Mar 2026 at 18:28, Roberto Ierusalimschy <rob...@inf.puc-rio.br> wrote:

> I do not mean it does not account it, I know it does. I mean it does not
> notice the memory usage has grown a lot due to the function being loaded,

... 

You may be right; we will check that. luaL_loadfilex creates a new
string with the file name (calling 'lua_pushfstring'), and that call
checks the collector. But lua_load does not do that check.

Will calling lua_gc(L, LUAGC_STEP, st.st_size /* This is the buffer size */ ) 

after successful loadbufferX be adequate as a contention measure? Or maybe just doing lua_createtable(L,0,0) ? ( I am not sure of a constant string, I think interning may skip string creation and checks ) Although it does only show up in tests/benchmarks in an artificially memory low VM  I would prefer to plug the hole.

Francisco Olarte.

 

-- Roberto

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/lua-l/20260318172850.GA71584%40arraial.inf.puc-rio.br.

[Roberto Ierusalimschy]

> > You may be right; we will check that. luaL_loadfilex creates a new
> > string with the file name (calling 'lua_pushfstring'), and that call
> > checks the collector. But lua_load does not do that check.

If the chunk is text, the parser checks the collector, but with a binary
chunk really there is no check. The following program shows the problem:

local s = "return 34"
s = string.dump(load(s))
for i = 1, 1e8 do load(s) end
io.write(collectgarbage"count" * 1024, "\n")

> Will calling lua_gc(L, LUAGC_STEP, st.st_size /* This is the buffer size */
> )
> after successful loadbufferX be adequate as a contention measure? Or maybe
> just doing lua_createtable(L,0,0) ? ( I am not sure of a constant string, I
> think interning may skip string creation and checks ) Although it does only
> show up in tests/benchmarks in an artificially memory low VM I would
> prefer to plug the hole.

'lua_pushstring' and its variations all check the GC, independently
of interning. (That is why luaL_loadfile has no issues.)

-- Roberto

[Francisco Olarte]

On Wed, 18 Mar 2026 at 18:56, Roberto Ierusalimschy <rob...@inf.puc-rio.br> wrote:

> > You may be right; we will check that. luaL_loadfilex creates a new
> > string with the file name (calling 'lua_pushfstring'), and that call
> > checks the collector. But lua_load does not do that check.

If the chunk is text, the parser checks the collector, but with a binary
chunk really there is no check. The following program shows the problem:

    local s = "return 34"
    s = string.dump(load(s))
    for i = 1, 1e8 do load(s) end
    io.write(collectgarbage"count" * 1024, "\n")

You beat me to it. I read lua_load, notice the undump() call and redid my tests using the uncompiled chunk, it was 10 times slower but did not exhibit the OOM problem.

 

'lua_pushstring' and its variations all check the GC, independently
of interning. (That is why luaL_loadfile has no issues.)

OK, will add a commented pushstring then. 

Thanks.

   Francisco Olarte.

[Francisco Olarte]

Just for the archives:

On Wed, 18 Mar 2026 at 18:59, Francisco Olarte <fol...@peoplecall.com> wrote:

'lua_pushstring' and its variations all check the GC, independently
of interning. (That is why luaL_loadfile has no issues.)

OK, will add a commented pushstring then. 

Adding a lua_pushstring/lua_pop fixed it as expected.

Thanks.

   Francisco Olarte.