Singleton in C API

[Stefano Cossu]

Hello there,
I want to implement a singleton in the Lua C API. A `new()` function
should check if an object with a unique ID (simply obtained by
concatenating a static prefix and the arguments passed to the function)
exists in the registry: if it exists, a reference to the existing one
should be returned, otherwise a new object is created and returned.

The object should be unique per process and should span across threads.

I have looked around possible options for this, including using light
userdata references, but I can't figure out how to correctly garbage
collect multiple instances of the singleton. If the first instance
(which contains the full userdata object) is garbage-collected first,
the other instances will be pointing to garbage.

Alternatively, since this singleton restriction is only on the C pointer
(the C library I am using mandates that a handle be unique within a
process), I could create multiple full userdata objects containing the
same C pointer. Here again, though, I don't know how I would finalize
the C structure when all its references are garbage-collected.

I also thought about creating the object in the global table and
returning a full userdata reference if the object has been already
created. Would Lua take care of garbage collection of the main object (I
guess when the process ends...?) and its references (when they get out
of scope)?

Thanks for any suggestions,
s

[Sainan]

I think the 'require' mechanism is kind of what you want. Each library is only required once, with subsequent calls to require simply reusing the library instance. 'package.preload' may help you easily register a named 'singleton constructor' function.

-- Sainan

[Stefano]

My situation is a bit more complicated. 

The singleton is an object inside a module, which is already guaranteed to be unique when using require. However, the object creation method  takes an identifier as an argument, and I need the method to allow creating different objects with different IDs, but avoiding the creation of two objects with the same ID.  That's why I was thinking about keeping the objects in a hash map.

For context, the object is an environment handle for an embedded DB (LMDB) and the identifier is the DB path. Only one environment must be opened for the same DB within the same process at any time.

[Stefano]

Oh, I get what you mean now. It looks like I could use the 'package.preload' table and built-in methods instead of creating my own singleton checking. I will give it a try. 

Thanks

s

[Sewbacca]

"The object should be unique per process and should span across threads."

If you mean across different lua_States then what you probably want is a pointer to your object with an added counter for how many live references exist. On __gc you decrement the counter. upon first initialization you create the object and store it elsewhere with the counter set to 1. Every subsequent singleton call either returns the previous userdata directly if it finds one in the registry of the current lua_State or if not you create a new userdata which points to your live object and increments the counter by one. If the counter hits zero during __gc you release the object.

This is basically a reference counter where Lua handles GC.

~ Sewbacca

[Stefano]

It looks like reference counting might be the way... in my case it would be a table with object IDs for keys and tables for values, holding the userdata handle to be returned for multiple calls to new(), and the number of references. 

I checked the 'module.loaded' table but it doesn't seem to work for me because my setup needs the ID associated with the singleton.

Thanks,

s

[Stefano Cossu]

I rethought the problem and ended up implementing a memorize function:

static int l_store_new (lua_State *L)
{
const VOLK_StoreType store_type = luaL_checkinteger (L, 1);
if (VOLK_store_int (store_type)->features & VOLK_STORE_EMBED)
luaL_error (
L, "Explicitly creating an embedded store is not allowed.",
2);
const char *id = luaL_optstring (L, 2, NULL);
const bool clear = lua_toboolean (L, 3);

const char *reg_key = lua_pushfstring (
L, "VOLK_store-%d-%s", store_type, id ? id : "default");
int check_type = lua_rawgetp (L, LUA_REGISTRYINDEX, reg_key);
// if check_type is not nil, the store userdata is now on top of
the stack.

if (check_type == LUA_TNIL) {
VOLK_Store **store_p = lua_newuserdatauv (L, sizeof (*store_p), 0);
luaL_setmetatable (L, "VOLK.Store");
lua_rawsetp (L, LUA_REGISTRYINDEX, reg_key);
if (clear) log_info ("Clearing old store.");
*store_p = VOLK_store_new (store_type, id, 0, clear);
LUA_NLCHECK (*store_p, "Error creating back end store.");
LOG_DEBUG ("Created first instance of store %s @%p", reg_key,
store_p);
} else LOG_DEBUG (
"Reusing store handle %s @%p",
reg_key, lua_topointer (L, -1));

return 1;
}

The changes were minimal and only to the new() function, no ref counting
or new tables needed, no changes to the finalizer.

The DB handle is tiny and used all throughout the library, and there
shouldn't be many variations of it in a normal application, so I don't
mind leaving it indefinitely until the program closes.

This seems to work as expected:

$ luap
Lua 5.4.8 Copyright (C) 1994-2025 Lua.org, PUC-Rio
luap 0.9 Copyright (C) 2012-2023 Dimitris Papavasiliou, Boris Nagaev
> store = require "volksdata.store"
> s1 = store.new(store.T_MDB)
10:37:48 INFO src/store_mdb.c:218: `VOLK_MDB_STORE_URN' environment
variable is not set. The default URN file:///tmp/mdb_store has been set
as the store ID.
10:37:48 INFO src/store_mdb.c:375: Created environment at /tmp/mdb_store
10:37:48 DEBUG src/lua_store.c:46: Created first instance of store
VOLK_store-1-default @0x5e7b60f2af00
> s2 = store.new(store.T_MDB)
10:37:52 DEBUG src/lua_store.c:48: Reusing store handle
VOLK_store-1-default @0x5e7b60f2af00
> s3 = store.new(store.T_MDB, "file:///tmp/alt_store")
10:38:12 INFO src/store_mdb.c:375: Created environment at /tmp/alt_store
10:38:12 DEBUG src/lua_store.c:46: Created first instance of store
VOLK_store-1-file:///tmp/alt_store @0x5e7b61231a90
> s4 = store.new(store.T_MDB, "file:///tmp/alt_store")
10:38:20 DEBUG src/lua_store.c:48: Reusing store handle
VOLK_store-1-file:///tmp/alt_store @0x5e7b61231a90
> <CTRL-D>

10:38:22 DEBUG src/lua_store.c:69: Garbage collecting store @0x5e7b61231ac0.
10:38:22 DEBUG src/store.c:72: Freeing store @0x5e7b61231ac0
10:38:22 INFO src/store_mdb.c:394: Closing MDB env at /tmp/alt_store.
10:38:22 DEBUG src/lua_store.c:69: Garbage collecting store @0x5e7b60f2afc0.
10:38:22 DEBUG src/store.c:72: Freeing store @0x5e7b60f2afc0
10:38:22 INFO src/store_mdb.c:394: Closing MDB env at /tmp/mdb_store.

Thanks for all the hints! Happy New Year!
s

[Stefano Cossu]

Well, not quite. I was storing the string pointer value instead of the
string itself as an index.

Also, re-reading the manual at
https://www.lua.org/manual/5.4/manual.html#4.3 , which says:

> When you create a new Lua state, its registry comes with some
predefined values.

Which seems to assume that the registry is tied to the state. I imagine
that busted creates a new Lua state for each test file, and my library
creates overlapping environments on multiple open Lua states.

However, the PIL book (for 5.4) states, under "Storing state in C
functions" -> "The registry":

> The registry is a global table that can be accessed only by C code.

Which contradicts the manual, unless I am interpreting "global" wrong. I
see, however, that the registry has a reference to the global
environment via LUA_RIDX_GLOBALS, so it should be possible to store my
memorized handles there.

So, considering that the registry is not global across Lua states, I
reworked my function thus:

static int l_store_new (lua_State *L)
{
const VOLK_StoreType store_type = luaL_checkinteger (L, 1);
if (VOLK_store_int (store_type)->features & VOLK_STORE_EMBED)
luaL_error (
L, "Explicitly creating an embedded store is not allowed.",
2);
const char *id = luaL_optstring (L, 2, NULL);
const bool clear = lua_toboolean (L, 3);

lua_rawgeti (L, LUA_REGISTRYINDEX, LUA_RIDX_GLOBALS);

const char *reg_key = lua_pushfstring (
L, "VOLK_store-%d-%s", store_type, id ? id : "default");

lua_pushvalue (L, -1); // Duplicate before popping for LUA_TNIL
condition.
int check_type = lua_rawget (L, -3);

// if check_type is not nil, the store userdata is now on top of
the stack.

if (check_type == LUA_TNIL) {

lua_remove (L, -1); // Top value was nil.
VOLK_Store **store_p = allocate_store (L);
lua_rawset (L, -3);

if (clear) log_info ("Clearing old store.");
*store_p = VOLK_store_new (store_type, id, 0, clear);
LUA_NLCHECK (*store_p, "Error creating back end store.");
LOG_DEBUG ("Created first instance of store %s @%p", reg_key,
store_p);
} else LOG_DEBUG (
"Reusing store handle %s @%p",
reg_key, lua_topointer (L, -1));

return 1;
}

s