mmap support for compiled Lua

[Stefan Heinzmann]

Hi all,

I was wondering how difficult it would be to change the compiled Lua in
such a way that it won't need to be loaded into memory, but can be
mapped into memory using mmap (or similar), and used in situ. I figure
the compiled bytecode representation would have to be position
independent for that to work.

I am motivated to ask because I can see potential memory savings in
embedded systems, for example through bytecode in read-only memory.

Kind regards
Stefan

[Sainan]

Afaik, bytecode already is position-independent in the sense that all jumps are relative, but the more important matter is that Lua needs to allocate structs (Closure, Proto, State, CallInfo, stack, etc.) which need to link to each other and this obviously is more complex than just mapping a bytecode file into memory and taking a pointer to it.

However, what you can do is give Lua a file or a pointer to a memory-mapped file and ask it to load your bytecode from that. That way you only have 1 copy of the bytecode in RAM, assuming your filesystem is not in RAM. :P

-- Sainan

[Marc Balmer]

I was wondering how difficult it would be to change the compiled Lua in such a way that it won't need to be loaded into memory, but can be mapped into memory using mmap (or similar), and used in situ. I figure the compiled bytecode representation would have to be position independent for that to work.

I am motivated to ask because I can see potential memory savings in embedded systems, for example through bytecode in read-only memory.

It is actually quite easy to do, we use that all the time.  In our case, we play evil games: scripts on disk are encrypted, we mmap into memory, decrypt them and then pass the byteocde to lua_loadbuffer. Omitting the evil games, this looks approximately like this:

if (stat(path, &sb))

errx(1, "can't stat %s", path);

if ((fd = open(path, O_RDONLY)) == -1)

errx(1, "can't open %s", path);

if ((bytecode = mmap(0, sb.st_size, PROT_READ, MAP_PRIVATE, fd,

    (off_t)0L)) == MAP_FAILED)

errx(1, "can't map application file into memory");

if (luaL_loadbuffer(L, (char *)bytecode, outl, NULL))

errx(1, "failed to load %s: %s", path, lua_tostring(L, -1));

munmap(bytecode, sb.st_size);

close(fd);

if (lua_pcall(L, 0, LUA_MULTRET, 0))

errx(1, "failed to execute %s: %s", path, lua_tostring(L, -1));

- mb

[bil til]

Am Sa., 10. Mai 2025 um 14:15 Uhr schrieb 'Stefan Heinzmann' via lua-l
<lu...@googlegroups.com>:

>
> I am motivated to ask because I can see potential memory savings in
> embedded systems, for example through bytecode in read-only memory.

I assume you want to send the Lua BIN file to your system ROM Flash,
and then start the functions from ROM.

I tried it already. Modifying the code for loading from ROM is not
difficult for STM32 / ARM Cortex M, as this supports access from ROM
as from RAM without problems, just different memory address of course.

... just the problem is, that the BIN code generated by Lua does NOT
align the code to BIN file start in 32bit / 4byte alignment... . So
then only the first function is 32bit aligned and not the later
ones... .

ARM Cortex generally allows access to non-32bit-aligned numbers, but
only with very restricted assembler commands - you can not really load
coding from non-aligned memory directly - this does not make sense.

As I understood Roberto, they have this issue "on screen" already and
want to change this, but only in Lua V6 (within lua subversions the
binary file must noch change...).

[Roberto Ierusalimschy]

> As I understood Roberto, they have this issue "on screen" already and
> want to change this, but only in Lua V6 (within lua subversions the
> binary file must noch change...).

That is not for "V6", it is for 5.5, and it is already implemented
(see it in github). Note that only the bytecode, a few structures, and
strings are used directly from the "static" memory. (As Sainan pointed
out, several structures contain pointers, which make them difficult to
use unmodified.)

-- Roberto

[bil til]

Wow, what a SUPER GREAT news, thanks a lot.

Strings are no problem to start on byte code. ARM Cortex-M CAN read
byte wise without problems for strings. (I am not sure what you mean
by "structures"? ... if you mean e. g. int arrays, this of course
WOULD be a problem if these are NOT aligned...).

Just such typical numbers as integers MUST be aligned on 32 bit. As
also the Lua code commands are integers, every function code of course
also MUST be aligned 32 bit against BIN file start, then all should
work.

Am Fr., 16. Mai 2025 um 18:57 Uhr schrieb Roberto Ierusalimschy
<rob...@inf.puc-rio.br>:

> --
> You received this message because you are subscribed to the Google Groups "lua-l" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/lua-l/20250516165722.GC386049%40arraial.inf.puc-rio.br.

[Roberto Ierusalimschy]

> Strings are no problem to start on byte code. ARM Cortex-M CAN read
> byte wise without problems for strings. (I am not sure what you mean
> by "structures"? ... if you mean e. g. int arrays, this of course
> WOULD be a problem if these are NOT aligned...).

The bytecode format now ensures these data is aligned relative to the
start. So, if the initial address where the bytecode is loaded is
aligned, these structures will be aligned too.

-- Roberto

[Bas Groothedde]

That is not for "V6", it is for 5.5, and it is already implemented
(see it in github). Note that only the bytecode, a few structures, and
strings are used directly from the "static" memory. (As Sainan pointed
out, several structures contain pointers, which make them difficult to
use unmodified.)

-- Roberto

Maybe a little off-topic, but regarding the 5.5 bytecode

- What will be the major differences between 5.4 and 5.5 bytecode (aside from alignment)
- What is the "variant" in the instruction format?

Cheers,

~b

[Roberto Ierusalimschy]

> Maybe a little off-topic, but regarding the 5.5 bytecode
>
> - What will be the major differences between 5.4 and 5.5 bytecode (aside
> from alignment)

I don't think there are major differences.

> - What is the "variant" in the instruction format?

It is just another format. There is iABC and there is ivABC. Only
OP_NEWTABLE and OP_SETLIST use ivABC. (The comments for OP_NEWTABLE
in 'lopcodes.h' must be corrected.)

-- Roberto

[Bas Groothedde]

> On 20 May 2025, at 19:59, Roberto Ierusalimschy <rob...@inf.puc-rio.br> wrote:
>
> I don't think there are major differences.
>> - What is the "variant" in the instruction format?
> It is just another format. There is iABC and there is ivABC. Only
> OP_NEWTABLE and OP_SETLIST use ivABC. (The comments for OP_NEWTABLE
> in 'lopcodes.h' must be corrected.)
>
> -- Roberto

That’s splendid, thank you.

~ b