Suggestion for Lua 5.5: search current exe for luaopen_* first

[Steven]

As far as I can tell, there's no automatic way to look into the current exe for luaopen_* functions when requiring a module. The default C package loader looks inside foo.dll for luaopen_foo, but if I'm writing an .exe, I can't just implement an external linkage function luaopen_foo and expect that loader to find it. It either has to be in foo.dll on the cpath, or I have to manually add it.

The answer in https://stackoverflow.com/questions/23391821/how-to-set-a-require-to-return-a-table-module-from-lua-c-api seems to confirm that this isn't possible, at least it wasn't 11 years ago, and the comment suggests manually adding the function to package.preload.

I'm suggesting that before Lua 5.5 launches, the default C package loader look in the current .exe for luaopen_foo when requiring "foo".

Maybe I am just too new to Lua's C API and there is either a better workaround than this, or this solution is the most ideal due to some caveats that I don't yet know about due to lack of actual experience with this problem, or perhaps my suggestion is simply not possible to implement. Or perhaps it is possible, just not ideal due to likely breaking backwards compatibility.

But if it could be made automatic, that would be convenient. Thanks for taking the time to read this.

[Scott Morgan]

On 17/07/2025 19:49, Steven wrote:
> As far as I can tell, there's no automatic way to look into the current
> exe for luaopen_* functions when requiring a module. The default C
> package loader looks inside foo.dll for luaopen_foo, but if I'm writing
> an .exe, I can't just implement an external linkage function luaopen_foo
> and expect that loader to find it. It either has to be in foo.dll on the
> cpath, or I have to manually add it.

Check out luaL_requiref

If you read through the lua.c and linit.c code, and look for
luaL_openlibs, it should give you an idea. It's how the lua executable
load the standard libs (string, io, etc. etc.)

Scott

[Sewbacca]

On 17.07.2025 20:49, Steven wrote:

As far as I can tell, there's no automatic way to look into the current exe for luaopen_* functions when requiring a module. The default C package loader looks inside foo.dll for luaopen_foo, but if I'm writing an .exe, I can't just implement an external linkage function luaopen_foo and expect that loader to find it. It either has to be in foo.dll on the cpath, or I have to manually add it.

[...]

I'm suggesting that before Lua 5.5 launches, the default C package loader look in the current .exe for luaopen_foo when requiring "foo".

You can actually write this functionality in Lua:

-- get path to executable first
local app
for i = 0, -math.huge,-1 do
	if not arg[i] then break end
	app = arg[i]
end
-- add a searcher based on package.loadlib
table.insert(package.searchers, 1, function(modname)
	local symbol = "luaopen_"..modname:gsub("%.", "_")
	local f,err = package.loadlib(app, symbol)
	return f or string.format("failed loading `%s` from executable '%s': %s", symbol, app, err:gsub("\n$", ""))
end)
-- Load for example the module luaopen_builtin_mod
require "builtin.mod".foo()

This however only works for some platforms, as dynamically loading the executable runs into problems on some other platforms. I tested this for Windows (under MinGW, didn't try msvc), Ubuntu and Alpine, and while Windows and Alpine worked fine, Ubuntu complained about dynamically loading the executable. I suspect the difference between Alpine and Ubuntu is that Alpine uses musl, while Ubuntu uses glibc. See this StackOverflow post for more info on why you seemingly can't dlopen executables.

If you plan to go cross platform at some point, I'd recommend to do what Scott says:
Keep a list of your C modules around (of your luaopen_<bla> entrypoints) and load the function pointers directly into the preload table, for example.

[ppp vvv]

with C api i guess you could do something like this:

static int requre_ref;

int luaopen_foo(L){...}

int lua_require(L){
  if (0 == strcmp(lua_tostring(L, 1), "foo")) return luaopen_foo(L);  //require("foo") == luaopen_foo

  lua_rawgeti(L, LUA_REGISTRYINDEX, require_ref);   //otherwise get "original" require()
  lua_pushvalue(L, 1);
  lua_call(L, 1, 1);
  return 1;
}

int main(){
  lua_State * L = luaL_newstate();

  luaL_openlibs(L);

  lua_getglobal(L, "require");
  require_ref = luaL_ref(L, -1);
  lua_register(L, "require", lua_require);
}

i suppose lua_register should not destroy previous luaL_ref?

пятница, 18 июля 2025 г. в 14:57:14 UTC+2, Sewbacca:

[Sewbacca]

On 18.07.2025 13:57, 'Sewbacca' via lua-l wrote:

You can actually write this functionality in Lua:

-- get path to executable first
local app
for i = 0, -math.huge,-1 do
	if not arg[i] then break end
	app = arg[i]
end
-- add a searcher based on package.loadlib
table.insert(package.searchers, 1, function(modname)
	local symbol = "luaopen_"..modname:gsub("%.", "_")
	local f,err = package.loadlib(app, symbol)
	return f or string.format("failed loading `%s` from executable '%s': %s", symbol, app, err:gsub("\n$", ""))
end)
-- Load for example the module luaopen_builtin_mod
require "builtin.mod".foo()

Little Addendum: For my example to work, the executable must be compiled with -rdynamic on Linux and -Wl,--export-all-symbols on Windows, otherwise package.loadlib won't find the symbol, even if nm reports it correctly.


[ppp vvv]

and of course 

require_ref = luaL_ref(L, LUA_REGISTRYINDEX);

not

require_ref = luaL_ref(L, 1);

пятница, 18 июля 2025 г. в 15:16:33 UTC+2, ppp vvv:

[mjmouse9999]

> On Friday, July 18, 2025 at 11:16:33 PM UTC+10 ppp vvv wrote:
> with C api i guess you could do something like this:
>
> static int requre_ref;
>
> int luaopen_foo(L){...}
>
> int lua_require(L){
>   if (0 == strcmp(lua_tostring(L, 1), "foo")) return luaopen_foo(L);  //require("foo") == luaopen_foo
>
>   lua_rawgeti(L, LUA_REGISTRYINDEX, require_ref);   //otherwise get "original" require()
>   lua_pushvalue(L, 1);
>   lua_call(L, 1, 1);
>   return 1;
> }
>
> int main(){
>   lua_State * L = luaL_newstate();
>   luaL_openlibs(L);
>
>   lua_getglobal(L, "require");
>   require_ref = luaL_ref(L, -1);
>   lua_register(L, "require", lua_require);
> }
>
> i suppose lua_register should not destroy previous luaL_ref?
>

You'd be better off just inserting `luaopen_foo` in the package.preload table rather than overriding the require function. That way you can still take advantage of all the machinery for loading and bookkeeping libraries.

For example, your new require function no longer has two returns (the default one returns the module as well as the preload data). It also doesn't populate the `package.loaded` table or scan it, so doing `require'foo'; require'foo';` will attempt to load two instances of the library.

[this is part of what I've done for packaging Lua apps into a single .exe]

int main_fill_preload (lua_State *L) {

   /* package.preload is stored in the registry as well */

   /* https://www.lua.org/manual/5.4/manual.html#pdf-package.preload */
   luaL_getsubtable(L, LUA_REGISTRYINDEX, LUA_PRELOAD_TABLE);

   /* repeat for each C library */
   lua_pushcfunction(L, luaopen_foo);
   lua_setfield(L, -2, "foo");

   /* if you have precompiled Lua libraries */
   if (luaL_loadbuffer(L, "\x1bLua...", LEN, "bar/init.lua"))
      return luaL_error(L, "failed to load bar/init.lua");
   lua_setfield(L, -2, "bar")

   /* clean the stack to what it was before */
   lua_pop(L, 1);

   return 0;

}

int main () {
   lua_State *L = luaL_newstate();
   luaL_openlibs(L);

   main_fill_preload(L);

   [...]
}