[ANN] Lua table serializer, 2026-08-30

[Martin Eden]

Hello list,

I've updated my Lua table serializer. It serializes any Lua table
to load()-able Lua code.

There are not much of visible changes. I reviewed and updated most of
it's implementation. I wanted it be logical and simple to follow and
maintain.

As a result of "Lua syntax ambiguity" discussion, resolving syntax
clashes and adding styling whitespaces are now job of tokens output
stream. (Thanks for participants on that topic!)

Side effect is that ";" (statement terminator) is added after each
statement. Writing code to eliminate it will make implementation worse.

Also I like deployment format: program is single file with readable and
hackable Lua code. No other Lua code dependencies:

https://github.com/martin-eden/lua_table_serializer/blob/master/deploy/serialize_lua_graph.lua

Feedback is welcome.

-- Martin

[Luiz Henrique de Figueiredo]

Nice too, thanks.

I was curious about the 5.3+ restriction..
It seems that it is easy to make it work in Lua 5.2. The changes below
to deploy/serialize_lua_graph.lua seem to work:
local math_type = math.type or type
--assert_integer(counter)
(Fixing assert_integer would be better but you'll know how to do this
correctly.)
--lhf

[Martin Eden]

Thanks for feedback, lhf!


I think that's the old-school spirit of open-source:
you can do any changes to any program.

As for me, I'm just sharing tools I use in my environment.
I don't know how and where others will want to use them.

So I'm mostly on Lua 5.3/5.5 and 5.2 is out of my scope.
But for given code 5.2 support can be done with simple local change.

-- Martin

[David Sicilia]

Thanks for posting -- question, does it happen to support incremental updates to the stored data?  Like if you have a large data structure and you want to change only a small part of it, does it have to reload/rewrite the entire thing?

David

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/lua-l/7b1b6235-025d-4d05-99f7-7876c3074ef8%40disroot.org.

[Martin Eden]

On 2026-08-30 19:12, David Sicilia wrote:
> Thanks for posting -- question, does it happen to support incremental
> updates to the stored data?  Like if you have a large data structure
> and you want to change only a small part of it, does it have to
> reload/rewrite the entire thing?
> David

It serializes table with data that was present at call time.


-- Martin

[TopchetoEU]

> --
> You received this message because you are subscribed to the Google Groups
> "lua-l" group. To unsubscribe from this group and stop receiving emails
> from it, send an email to lua-l+un...@googlegroups.com. To view this
> discussion visit
> https://groups.google.com/d/msgid/lua-l/7b1b6235-025d-4d05-99f7-7876c3074ef
> 8%40disroot.org.

By the way, you could just directly generate bytecode, instead of lua. It
would probably make generation a lot simpler than dealing with a syntax, and a
bit faster as an added bonus (as you don't need to compile the just generated
code).

[Thijs Schreijer]

If the serialized format looks like this:

local T_1 = { };
local T_2 = { };

Don't you run out of locals if the table is sufficiently complex?

Dunno what the current limit is, but in the past I've been bitten by that when building a jsonschema validator (Lua code was generated for a given schema).

Thijs

[Martin Eden]

On 2026-08-31 14:21, 'Thijs Schreijer' via lua-l wrote:
> If the serialized format looks like this:
>
> local T_1 = { };
> local T_2 = { };
>
> Don't you run out of locals if the table is sufficiently complex?

Yeah it may run out of locals.


From code's point of view "local T_1 = { };" is value-capture statement.
Fact that interpreter can't allocate more than 250 names is not our
problem.

( I still consider that Lua is for meat-written code. This serializer is
just a demo. It could had been adopted to other programming language
but most of them can't even imagine indexing value by set type.
Moon shines here. )

"Ser" serializer hit same problem. It solved it by writing intermediate
tables to result table and returning zero'th element as result:

  local T = { }
  T[1] = { }
  T[0][1] = T[1]
  ...
  return T[0]

It avoids local's limit but code does not look natural.

-- Martin

[Martin Eden]

On 2026-08-31 13:35, 'TopchetoEU' via lua-l wrote:
> By the way, you could just directly generate bytecode, instead of lua. It
> would probably make generation a lot simpler than dealing with a syntax, and a
> bit faster as an added bonus (as you don't need to compile the just generated
> code).

Well, I wanted output format that (when saved to file) is hackable
by someone with text editor and sufficient intelligence.
That's source code.

That's why I went though design troubles in generating relatively
readable output.

For example you have some command-line tool written in Lua.
It accepts optional set of command-line arguments. If argument is
not specified it uses default value.

You can store defaults as table in separate file. Tool will load it
and user can change defaults to what fits for him.

You can always compile generated file to bytecode via "luac".
A bit longer save time, faster load time and data is less hackable.

-- Martin

[sur-behoffski]

On 2026-08-30 21:29, 'Martin Eden' via lua-l wrote:
> Hello list,
>
> I've updated my Lua table serializer. It serializes any Lua table
> to load()-able Lua code.
>
> There are not much of visible changes. I reviewed and updated most of
> it's implementation. I wanted it be logical and simple to follow and

> maintain. [...]
>
> https://github.com/martin-eden/lua_table_serializer/blob/master/deploy/serialize_lua_graph.lua
>
G'day Martin,

Lua 5.3 onwards supports the UTF-8 escape sequence '\u{XXX}' when
parsing source. Sometime ago, I suggested adding a '%Q' format
specifier to string.format, which is identical to "%q", except that
it looks for valid UTF-8 sequences, and, if found, emits the '\u{XX}'
equivalent. This '%Q' proposal hasn't been adopted by the Lua
language/libraries/ecosystem.

However, if interested, you might wish to add a flag to your
serializer, requesting this behaviour.

- The upside is that the output is more likely to be ASCII-based,
and therefore easily editable;

- One downside is that there would need to be extra work, with
an impact on efficiency, when serialising strings; and

- Another downside is that the serialised table is not compatible
with Lua 5.1 or 5.2; I have not investigated compatibility
relating to other Lua implementations (e.g. LuaJIT).

cheers, s-b etc.

[Martin Eden]

Hello sur-behoffsky,

On 2026-09-01 04:04, sur-behoffski wrote:
> it looks for valid UTF-8 sequences, and, if found, emits the '\u{XX}'
> equivalent

Intention of string quoter here is produce string representation that is
editable in text editor. And I never had problems with UTF-8 in text
strings.

They are sequences of high ASCII bytes. Our string quoter have
no business with high ASCII. So that sequences are remained as is,
and represented as glyphs in editor both in quoted and raw strings.

So I see no need to quote UTF-8 to low ASCII.

-- Martin

[Martin Eden]

.. and well, data may have high ASCII that is NOT UTF-8 code sequences.

So string quoter MAY find those "bad" sequences and demote them to low
ASCII.

But it's up to him. Editable text is just intention, not obligation.

His module is [concepts.lua.quote_string]. Feel free to adjust it to
your needs.

-- Martin