Persistence of threads

[Sewbacca]

I'm pondering how to serialize running coroutines to disk. There exists pluto but it is unfortunately specifically written for Lua 5.1.

Here's where I'm coming from: I have a microcontroller that has 512KB of SRAM when the controller is running and 16KB of sleep resistant ram when the microcontroller is doing nothing, which it does quite often.

This is a problem for any user space application that runs accross sleep schedules.

My first thought was to serialize table to sleep ram, such that any Lua code knows it's state and can continue from where it left of.

It would be much nicer from a user's POV however if I could simply serialize the whole running program as a coroutine, such that sleeping would not affect the code structure.

My third thought was that I could simply run the whole VM within the 16KB sleep resistent part of the microcontroller, but I feel like that might not be enough RAM even for Lua. It would be nice however if I'd be proven wrong.

It might work if I could store values that are recreated during init time on main ram, and any user values that need to survive sleeps on the sleep resistent ram, but I dont know of any mechanism from Lua that allows for that.

Any thoughts, ideas or recommendations are welcome!

[Andrey Dobrovolsky]

Hi Sewbacca!

I suppose that You have some flash memory in Your system. Usually
flash ROM size exceeds an amount of RAM in microcontrollers. Haven't
You considered backing up to flash upon putting Your system asleep?

Regards,
Andrew

пт, 31 жовт. 2025 р. о 20:30 'Sewbacca' via lua-l <lu...@googlegroups.com> пише:

> --
> You received this message because you are subscribed to the Google Groups "lua-l" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/lua-l/5386e1b3-cb31-41e1-9870-e1f64c0a6b1c%40kolabnow.com.

[Sewbacca]

You mean backing up the VM state to flash? That might work, if I load them back at the exact memory addresses, so that no pointer goes dangling.

I'm a little bit worried for the flash health, because that would mean a read/write every minute. It could also drain the battery.

[Andrey Dobrovolsky]

I thought about Your data being incorporated into a global table.
After finishing the script before closing state some C code should
store necessary data to the flash memory. During the next awakening a
new state should be created, then another C code should restore global
table fields and then Your script may continue.
But Your device is battery powered, and must start every minute, not
good for common flash endurance. Additional efforts must be applied to
minimize the amount of the data to be stored and rotating the flash
pages which are used to store Your data.
Some thoughts, not knowing Your details. Not a general solution, just
custom for Your particular needs.

What do You think about such an approach?

-- Andrew

пт, 31 жовт. 2025 р. о 22:47 'Sewbacca' via lua-l <lu...@googlegroups.com> пише:

>
> You mean backing up the VM state to flash? That might work, if I load them back at the exact memory addresses, so that no pointer goes dangling.
>
> I'm a little bit worried for the flash health, because that would mean a read/write every minute. It could also drain the battery.
>

> --
> You received this message because you are subscribed to the Google Groups "lua-l" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/lua-l/09f6e1b2-f41a-42ed-a2a1-2fc9b529732f%40kolabnow.com.

[Sewbacca]

If the data stored between wake-ups is small enough (<16KB) then I can
ditch the flash altogether.

What I am looking for is a way to serialize coroutines. The only library
i know which is capable of doing so is pluto, but that one only works
for Lua 5.1. That the microcontroller flushes main RAM every time it
goes to sleep was really just a prompt for me to revisit this concept.
Serializing threads would be in general a useful feature, since we
already can serialize any other Lua value. The idea was to have
programs, that require complex continuous logic, run as a coroutine,
which some C function could write to or load back from the sleep
resistant memory, when it yields. That is really is not a required
feature for this project and if it turns out that I can't do it with the
16 KB available SRAM, such that flash is required, then I will skip it.

I would still be interested in a solution in general, even if it can't
be done for this specific project.

[Andrey Dobrovolsky]

I know only about pma - persistent memory allocator - by Terence
Kelly, https://web.eecs.umich.edu/~tpkelly/pma/. This tool was
integrated into gawk as an experimental feature,
https://www.gnu.org/software/gawk/manual/html_node/Persistent-Memory.html.
But it is applied to the data only. If I am not mistaken even not for
user-defined functions.
Making the whole Lua VM state persistent seems to be a very complicated task.

Andrew

сб, 1 лист. 2025 р. о 03:34 'Sewbacca' via lua-l <lu...@googlegroups.com> пише:

> --
> You received this message because you are subscribed to the Google Groups "lua-l" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/lua-l/8300d581-6de0-40e8-b8f1-86901fab0c0f%40kolabnow.com.

[Sewbacca]

That's an interesting feature and worth checking out at some time.

I've measured Lua's memory footprint and even small hello world programs exceed 20KB RAM which is too much for my sleep resistant RAM. I didn't try it on the microcontroller though, but I suspect if anything it will not save me more than 4KB.

For the time being I think I will skip the feature of sleepy threads, because accessing flash for that is not a good use of resources.

Does anyone know if someone has ported Pluto for 5.4 or if someone has written another heavy duty serialization library that can serialize threads?

~ Sewbacca

[Andrey Dobrovolsky]

> Here's where I'm coming from: I have a microcontroller that has 512KB of SRAM when the controller is running and 16KB of sleep resistant ram when the microcontroller is doing nothing, which it does quite often.

Sorry for being off-topic, but some MCUs have STOP mode with all clock
sources disabled while all the internal RAM is kept powered.

-- Andrew

нд, 2 лист. 2025 р. о 22:21 'Sewbacca' via lua-l <lu...@googlegroups.com> пише:

> --
> You received this message because you are subscribed to the Google Groups "lua-l" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/lua-l/efb5592d-7c61-4daa-9008-94342f1ebb1d%40kolabnow.com.

[Foster]

> some MCUs have STOP mode with all clock sources disabled while all the internal RAM is kept powered

The new Raspberry Pi chips do this (RP2040 RP2350 with the RP2350 having an actual stop mode).  These draw less than 1 microamp in this state.  With the RTC doing a wake up every minute to poll sensors, battery life is in months.   So lots of way to save power across the long term

[Roberto Ierusalimschy]

> I've measured Lua's memory footprint and even small hello world programs exceed 20KB RAM which is too much for my sleep resistant RAM. I didn't try it on the microcontroller though, but I suspect if anything it will not save me more than 4KB.

You can reduce this initial footprint by not loading all standard libraries,
only _G and package. The others you can only preload.

-- Roberto

[Sewbacca]

On 11/3/2025 2:26 PM, Foster wrote:

> some MCUs have STOP mode with all clock sources disabled while all the internal RAM is kept powered

The new Raspberry Pi chips do this (RP2040 RP2350 with the RP2350 having an actual stop mode). 

I'm using the esp32-s3 which has a deep sleep mode when only the co processor is running and a light sleep mode when the main RAM is being kept alive. The thing I wanna program is a "smartwatch" Watchy from SQFMI which is just a small board with an E Ink display slapped on it.
Being low power and all, it lasts a week before the battery runs out, using the original firmware. I don't wanna sacrifice that for programmable convenience.

These draw less than 1 microamp in this state.  With the RTC doing a wake up every minute to poll sensors, battery life is in months.   So lots of way to save power across the long term

--

That is quite low, the datasheet for the esp32-s3 says it draws 7 microamps in deepsleep and 240 microamps in light sleep.

~ Sewbacca

[Sewbacca]

I didn't think of that, I did some more testing and wanna share what I've found:

- Running Lua without anything draws 7-8KB - Running just the base and nothing else about 9-10KB - Running a simple fibonacci program that prints the first 20 numbers 11-12KB (using only _G) - Running a simple terminal ascii clock updating constantly using a busy loop with os, string and io loaded uses just below 16KB (calling collectgarbage() every cycle)

The sleep resistant memory of the esp32-s3 has 16KB in total, so I could just barely run a simple terminal ascii clock within it. The main benefit of pausing Lua instead of restarting it every time would be that complicated programs would not have to worry about which data has to be stored and when and how it should be reconstructed.

Serializing threads could circumvent this issue by only serializing non static data (i.e. no standard packages _G etc.) or giving it unique ids derived of their global variable name, if they are referenced from a local.
We can serialize almost every data type in Lua, even functions thanks to string.dump and the debug API. coroutines however seem to be the only exception, despite being transparent to the debug API.

[Spar]

What implementation do you use? The original Lua or something like eLua, Lua RTOS, NodeMCU Lua, embLua?

--

You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/lua-l/55004919-cee1-4686-8dc1-17522c92a14f%40kolabnow.com.

[Sewbacca]

On 11/4/2025 10:42 AM, Spar wrote:
> What implementation do you use?

I've used the stock lua.c file from the GitHub repo, linked against the
packaged version of my distro running Lua 5.4.8

[bil til]

Using Lua 5.5 should in any way bring quite a bit of RAM savings, if
you know how to compile your text files to bin files externally, and
then load down BIN code... ..

(Versions before load the Lua bin code generally to RAM, even if you
load down BIN. In V5.5 only the Lua BIN code of Lua text files will
store into RAM, but if you load down BIN files, the BIN code will keep
in RAM with some minor changes in source code... ).

But 16kiloByte RAM is really VERY few in any case. Typically it would
be better, if during sleep phases you run a special C code which is
very restricted. And then during wakeup only you run Lua, and I hope
during this wake up phases hopefully a bit more than 16kB RAM :).

[Sewbacca]

On 11/4/2025 11:08 PM, bil til wrote:

Using Lua 5.5 should in any way bring quite a bit of RAM savings, if
you know how to compile your text files to bin files externally, and
then load down BIN code... ..

I reran a simple test with Lua 5.5, comparing a huge program with all packages disabled, which does nothing, but uses a whole bunch of randomly generated constant strings, using either the precompiled .luac binary or the source file. I ran the program both with 5.5 and 5.4:

Lua 5.5.0  Copyright (C) 1994-2025 Lua.org, PUC-Rio
pre-compiled:
max mem used 9051760
source chunk:
max mem used 15796118
Lua 5.4.8  Copyright (C) 1994-2025 Lua.org, PUC-Rio
pre-compiled:
max mem used 7872161
source chunk:
max mem used 12650342

I used lua 5.4.8 and the beta from https://lua.org/work, so not the github version. Apparently Lua 5.4 is better in saving some memory, even though Lua 5.5 should save more space if it uses lua_pushexternalstring.

I'm probably doing something wrong, because .cursize appears to be negative (on both versions) after closing the state, instead of 0 what I would've expected, so here is the relevant code snippet I used to measure used memory (by modifying lua.c).

typedef struct info {
	ptrdiff_t maxsize;
	ptrdiff_t cursize;
} info;

void *luaL_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
	info* i = ud;
	i->cursize += nsize;
	i->cursize -= osize;
	i->maxsize = i->cursize > i->maxsize ? i->cursize : i->maxsize;

	if (nsize == 0) {
		free(ptr);
		return NULL;
	}
	else
	return realloc(ptr, nsize);
}

int main (int argc, char **argv) {
	int status, result;
	info used = { .cursize = 0, .maxsize = 0 };
	lua_State *L = lua_newstate(luaL_alloc, &used, luaL_makeseed(NULL));  /* create state */

But 16kiloByte RAM is really VERY few in any case. Typically it would
be better, if during sleep phases you run a special C code which is
very restricted. And then during wakeup only you run Lua, and I hope
during this wake up phases hopefully a bit more than 16kB RAM :).

I also tried running a small program with Lua 5.5 and got no memory benefit beyond what I've already found by not loading any unnecessary libraries, so I guess for very small applications, the new lua_pushexternalstringstring is negligible.

[Sainan]

> I'm probably doing something wrong, because .cursize appears to be negative (on both versions) after closing the state, instead of 0 what I would've expected

iirc, in the case `ptr == NULL`, `osize` is actually a 'tag' so cannot be assumed to be 0 (even tho 0 would be the correct value).

-- Sainan