Lua for statically linked programs?

[Fengkai Sun]

Hi all,

I'm trying to link a program statically, which allows the users using lua to call its exported symbols.

However, lua is not happy since it cannot find symbols in the binary. I found some source saying exporting symbols in statically linked programs is simply not possible: https://stackoverflow.com/a/53432644/14033810

I'm wondering if there is a way for lua to work with statically linked programs? Thanks!

---

Best,

Fengkai

[Luiz Henrique de Figueiredo]

> I'm trying to link a program statically, which allows the users using lua to call its exported symbols.

See http://lua-users.org/lists/lua-l/2016-04/msg00156.html

[Fengkai Sun]

I'm afraid that won't help. I have tried `-Wl, -E` (synonym of `--export-dynamic`), but it does not work for statically linked binaries.

I'm quite new to Lua, so I will explain the use case more detailedly: I have a program that allows users to call its interface via Lua. From my understanding, this is done by exporting symbols in the binary, and then Lua can find the symbols and call it.

But the question is that there is no exported symbol in statically linked binaries.

---

Thanks,

Fengkai

On Tue, Jul 9, 2024 at 10:39 PM Luiz Henrique de Figueiredo <l...@tecgraf.puc-rio.br> wrote:

> I'm trying to link a program statically, which allows the users using lua to call its exported symbols.

See http://lua-users.org/lists/lua-l/2016-04/msg00156.html

--
You received this message because you are subscribed to a topic in the Google Groups "lua-l" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lua-l/qasVn1YX7m8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lua-l+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/CAD55k3qZdyH4g4yuwJ-htU1u0PynXo48dbqbmZ8visqWZi8nqQ%40mail.gmail.com.

[Sainan]

You seem to have a severe misunderstanding here — there is no such thing as a "statically-linked program." When it comes to static & dynamic linking, these terms refer to how a program (or a library) includes another library. Static linking means the code is directly included in the final binary, and dynamic linking means the code is loaded from a different binary — that is either shipped with your program, shipped with the OS, or shipped with another package. Most importantly, a program is not limited to use only one of these approaches. E.g. a game engine would typically statically link its scripting environment like Lua, but dynamically link low-level stuff like runtime libraries, OS APIs, and graphics card specific APIs.

[Fengkai Sun]

By "statically-linked program" to say I mean a program without any external dependency. If so, it needs an interpreter to resolve dependencies at runtime.

Consider a program using static libm and dynamic libc. It can be linked with `gcc -o main main.c /usr/lib/x86_64-linux-gnu/libm.a`. `file main` will say it's a "ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2". But if we use `gcc -static -o main main.c -lm`, it becomes "ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked", without the interpreter.

It seems that for programs without external dependencies, linker will not emit `.dynsym` section, so that Lua cannot seem to find symbols.

Please correct me if I am wrong.

On Tue, Jul 9, 2024 at 11:58 PM 'Sainan' via lua-l <lu...@googlegroups.com> wrote:

You seem to have a severe misunderstanding here — there is no such thing as a "statically-linked program." When it comes to static & dynamic linking, these terms refer to how a program (or a library) includes another library. Static linking means the code is directly included in the final binary, and dynamic linking means the code is loaded from a different binary — that is either shipped with your program, shipped with the OS, or shipped with another package. Most importantly, a program is not limited to use only one of these approaches. E.g. a game engine would typically statically link its scripting environment like Lua, but dynamically link low-level stuff like runtime libraries, OS APIs, and graphics card specific APIs.

--

You received this message because you are subscribed to a topic in the Google Groups "lua-l" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lua-l/qasVn1YX7m8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lua-l+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/LIeLXNYDX-lnGWvsTS5Nee9vWwhR0lTRZ7swTVf3T2ucSz7Vu9F0iLRMMiYDCFvrMAnT0xaNJEQHHKp1KsaRcVULeZeq7z8_1WMmFSgowcc%3D%40calamity.gg.

[Sainan]

In my limited experience with Linux, I do think you need to at least dynamically link against libc to provide C standard library functions, such as fopen. Lua is pretty minimal in that regard, being able to target all the way down to C89, but it still needs some level of C support. So, for your interesting use-case, you may look at how to statically link libc (or a different, compatible implementation of the C standard library), if it is possible.

[Warner Losh]

On Tue, Jul 9, 2024, 11:03 AM 'Sainan' via lua-l <lu...@googlegroups.com> wrote:

In my limited experience with Linux, I do think you need to at least dynamically link against libc to provide C standard library functions, such as fopen.

You can statically link libc to get these.

Lua is pretty minimal in that regard, being able to target all the way down to C89, but it still needs some level of C support. So, for your interesting use-case, you may look at how to statically link libc (or a different, compatible implementation of the C standard library), if it is possible.

The problem is dlopen used to find the symbols doesn'twork 100% static.

However, having said that... you can statically link everything except libc and often be good.

Warner

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/yA9_z74Uo6hqdBchw9HidBNRJaMSGOk1kxDRulYHu94SQNsVJ7dSyOyXW0ONl3e2a9PM3XnFpjOorCgFPlYKZ9dZoZ6cbvv69I1V3lzYG_o%3D%40calamity.gg.

[Sainan]

Well, I don't think Lua itself uses dlopen unless explicitly compiled with support for it (default with make PLAT=linux​).

[Sean Conner]

It was thus said that the Great Fengkai Sun once stated:

> By "statically-linked program" to say I mean a program without any external
> dependency. If so, it needs an interpreter to resolve dependencies at
> runtime.
>
> Consider a program using static libm and dynamic libc. It can be linked
> with `gcc -o main main.c /usr/lib/x86_64-linux-gnu/libm.a`. `file main`
> will say it's a "ELF 64-bit LSB shared object, x86-64, version 1 (SYSV),
> dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2". But if we use
> `gcc -static -o main main.c -lm`, it becomes "ELF 64-bit LSB executable,
> x86-64, version 1 (GNU/Linux), statically linked", without the interpreter.
>
> It seems that for programs without external dependencies, linker will not
> emit `.dynsym` section, so that Lua cannot seem to find symbols.
>
> Please correct me if I am wrong.

I don't think you are. I tried a static compile of my gopher server
(written in Lua) [1] and while I could get a static executable out of it,
any attempt to load a Lua module written in C that wasn't compiled in would
generate:

"BlackHole:" error loading module 'org.conman.uuid' from file '/usr/local/lib/lua/5.4/org/conman/uuid.so':
/usr/local/lib/lua/5.4/org/conman/uuid.so: undefined symbol: lua_isnumber

and eventually a core dump. As long as I restricted the code to only
loading C-based Lua modules that were compiled in, the static compile would
work. Modules written in Lua could be loaded without issue at run time.

-spc

[1] https://boston.conman.org/2023/03/22.1

[sur-behoffski]

On 2024-07-10 05:31, Sean Conner wrote:
> It was thus said that the Great Fengkai Sun once stated:
>> By "statically-linked program" to say I mean a program without any external
>> dependency. If so, it needs an interpreter to resolve dependencies at
>> runtime.

>> [...]

G'day,

I think that there's a couple of things going on here, although
my knowledge of details is a little old now.

1. I think that "lua_isnumber" was a symbol that was provided by
earlier (5.1?) versions of Lua, but the client doesn't reference it
directly: The detail comes from the ABI. In this case, you may be
compiling an external module with the wrong version of the ABI.
(In fact, this is also partially what the 5.4.7-rc3 ABI
identification issue was all about.)

2. If you're looking to load a module from Lua, using "require",
the code needs to be compatible with dl_open() at a binary level, as
well as being findable on the LUA_CPATH or LUA_PATH set of paths.

In my lglicua Assistant, I tried to cover this when I switched from
Distro-provided Lua packaging (which was originally missing
versions, or else only had older patchlevels) to downloading and
compiling Lua automagically within the installation process.

See (from lglicua-0.1-beta1):

https://sourceforge.net/p/lglicua/code/ci/master/tree/install/support/lua-generic-utils.sh#l395

The Bash function "localLGU_SourcePatch" prepares Lua for dynamic
linking:

--------

# Part 1: Add item to CFLAGS asking for position-independent
# code.
sed -i -e '/^CFLAGS=/s/$/ -fPIC/' src/Makefile


# Part 2: Add a line to the build target "libluaX.Y.a" ($LUA_A)
# to also create "libluaX.Y.so.0" as part of its recipe.
SOLibBuild="\$(CC) -shared -ldl -Wl,-soname=%s"
SOLibBuild="${SOLibBuild} -Wl,-version-script=luaver.ld"
SOLibBuild="${SOLibBuild} -o %s \$? -lm \$(MYLSFLAGS)"
# shellcheck disable=SC2059
BuildSOCmd="$(printf "$SOLibBuild" "${SOLibFile}.0" "liblua.so")"
# shellcheck disable=SC2016
BuildSOSedCmd="$(printf '/$(RANLIB) $@/a\\\t%s' "$BuildSOCmd")"
sed -i -e "$BuildSOSedCmd" src/Makefile


# Part 3: Create a simple luaver.ld script for the linker.
cat <<EOF >src/luaver.ld
LUA_${cnLuaVer:3} {
global:
*;
};
EOF

"cnLuaVer" means "Canonical Name for Lua Version", e.g. "5.4",
regardless of the patchlevel. (The caller can specify any of
5.1, 5.2, 5.3 or 5.4.)

"SOLibFile" maps to "liblua.so"; this gets used both during the
build, but especially during the library installation function,
see:

https://sourceforge.net/p/lglicua/code/ci/master/tree/install/support/family-ubuntu.sh#l316

--------

# Install Dynamically-loadable shared-object files, using install
# and ldconfig:
# install: "lib/liblua.so" -> "LIBPATH/libluaX.Y.so.0.0.0"
# manual: ( cd "LIBPATH";
# ln -fs "libluaX.Y.so.0.0.0" "libluaX.Y.so"
# )
# ldconfig.1: symlink-add re "LIBPATH/libluaX.Y.so*"
# ldconfig.2: cache-update re "LIBPATHlibluaX.Y.so*"
# ?? SECURITY: USES SUDO. (Several times.)

GenericSOLibFile="$(DBIU_FN Lua:SOLibFile)"
VersionedSOLibFile="$(DBIU_FN "$cnLuaVer" SOLibFile)"
RealSOLibFile="${VersionedSOLibFile}.0.0.0"
sudo install -m 0644 "lib/${GenericSOLibFile}" \
"${LibPath}${RealSOLibFile}"
(
cd "$LibPath" || exit 42

# Create symlink "libluaX.Y.so.0.0.0" -> "libluaX.Y.so"
sudo ln -fs "$RealSOLibFile" "$VersionedSOLibFile"

# All remaining symlinks are a by-product of ldconfig, along
# with object caching.
sudo ldconfig "${LibPath}/${VersionedSOLibFile}"
)

--------

This results in a Lua version with all symbols exported, and with
position-independent code (-fPIC), so that the dynamic loader can
do its stuff.

We use "make local", which delivers unversioned headers, libraries
and executables purely into the source tree, so that we can
control filenames, particularly adding explicit versioning, when
we install the files into the OS's filesystem. (There may also be
a version-registration step, such as Debian's "update-alternatives",
so that we can switch between versions at runtime.)

--------

cheers, b

[Warner Losh]

On Tue, Jul 9, 2024, 12:34 PM 'Sainan' via lua-l <lu...@googlegroups.com> wrote:

Well, I don't think Lua itself uses dlopen unless explicitly compiled with support for it (default with make PLAT=linux​).

Can't load modules written in c without it or a similar thing...

Warner

--
You received this message because you are subscribed to the Google Groups "lua-l" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lua-l+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/mODwOveoNRJs20bBWKQb8hWed-LGB8d8YM0On-GcRMAIEn51PryyWQGne9h3kOfDo5gavfwLiMjqOS3zzRNFxI79kywbdtjoga_ip0asYY4%3D%40calamity.gg.

[Sean Conner]

It was thus said that the Great sur-behoffski once stated:

> On 2024-07-10 05:31, Sean Conner wrote:
> > It was thus said that the Great Fengkai Sun once stated:
> >> By "statically-linked program" to say I mean a program without any external
> >> dependency. If so, it needs an interpreter to resolve dependencies at
> >> runtime.
> >> [...]
>
> G'day,

Hello.

> I think that there's a couple of things going on here, although
> my knowledge of details is a little old now.
>
> 1. I think that "lua_isnumber" was a symbol that was provided by
> earlier (5.1?) versions of Lua, but the client doesn't reference it
> directly: The detail comes from the ABI. In this case, you may be
> compiling an external module with the wrong version of the ABI.
> (In fact, this is also partially what the 5.4.7-rc3 ABI
> identification issue was all about.)

No, it's still provided by Lua 5.4, and in the case I presented:

"BlackHole:" error loading module 'org.conman.uuid' from file '/usr/local/lib/lua/5.4/org/conman/uuid.so':
/usr/local/lib/lua/5.4/org/conman/uuid.so: undefined symbol: lua_isnumber

the "client code" is directly calling the function lua_isnumber() [1], and
everything is compiled against Lua 5.4.6.

> 2. If you're looking to load a module from Lua, using "require",
> the code needs to be compatible with dl_open() at a binary level, as
> well as being findable on the LUA_CPATH or LUA_PATH set of paths.

It is. No cross-compiling was done for my example.

> In my lglicua Assistant, I tried to cover this when I switched from
> Distro-provided Lua packaging (which was originally missing
> versions, or else only had older patchlevels) to downloading and
> compiling Lua automagically within the installation process.

Okay, so I have a Makefile that compiles everything into a single
executable. It statically compiles Lua, and all the requested Lua modules
(C and Lua), but otherwise, other libraries are dynamically linked:

[spc]lucy:~/projects/gemini/misc/C>ldd port70
libz.so.1 => /usr/lib/libz.so.1 (0x00ced000)
librt.so.1 => /lib/tls/librt.so.1 (0x002a5000)
libm.so.6 => /lib/tls/libm.so.6 (0x00cc2000)
libdl.so.2 => /lib/libdl.so.2 (0x00ce7000)
libmagic.so.1 => /usr/lib/libmagic.so.1 (0x00111000)
libcrypto.so.1.1 => /usr/local/lib/libcrypto.so.1.1 (0x002b9000)
libc.so.6 => /lib/tls/libc.so.6 (0x00b90000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00cff000)
/lib/ld-linux.so.2 (0x00b76000)

And this works fine. Lua (and other C-based Lua modules) are statically
linked into the final exectuable. Any C-based Lua modules not compiled in
are loaded and things work fine. I then tweaked the Makefile to get a
purely static compile (with '-static') and while it does compile, any C-base
Lua module referenced that is not compiled in will error out and the program
will eventually get a segfault.

-spc

[1] https://github.com/spc476/SPCUUID/blob/3fc5c6d58ac9ddfb60cd60087c08e037dbdd6e5c/src/luauuid.c#L222

[Sainan]

Warner, I am aware that you need LoadLibrary/dlopen to load C modules, but the problem with that is that it may introduce portability concerns, which slightly sours the benefits of a high-level language for me.

I really like just being able to drag and drop my scripts onto a Linux machine from Windows or vice-versa and having it still work.

If this required compilation of some form, I might as well still be using C++ to begin with.

[bil til]

Can't you describe a bit more clearly your background / intended application?

(OS system etc. etc... answering such basic questions in some "please
explain the Lua world manner" ist nearly impossible, if not really
understandable, what is your target application / your intentions...).

[Fengkai Sun]

Hi bil,

I guess this is a great idea, since the discussion has gone a little too far...

Basically I was trying to compile a full statically-linked snort3: https://github.com/snort3/snort3.git.

I was able to do that, but during the application startup it threw an error that seems Lua-related, which brought me here.

snort3 allows users to customize a few rules (it's an intrusion detection system) via a Lua script, which I believe works in the following workflow:

(C++) main -> init -> lua_pcall -> (Lua) calling C++ functions according to script -> (C++) retrieve results

Again, I'm completely new to Lua, so this might be imprecise or completely wrong. Sorry about that.

Below is the related stack frames when the error occurs:

#11 0x0000000000715755 in lj_err_throw (L=0x7ffff7f8d380, errcode=2) at lj_err.c:790
#12 0x0000000000715f84 in lj_err_run (L=0x7ffff7f8d380) at lj_err.c:905
#13 0x0000000000716793 in lj_err_callermsg (L=0x7ffff7f8d380, msg=0x1603cd0 "snort: undefined symbol: snort_whitelist_append") at lj_err.c:1034
#14 0x000000000079f832 in clib_error_ (L=0x7ffff7f8d380) at lj_clib.c:38
#15 0x00000000007a0011 in lj_clib_index (L=0x7ffff7f8d380, cl=0x7ffff7f9e638, name=0x7ffff7f987f0) at lj_clib.c:383
#16 0x0000000000748c6d in ffi_clib_index (L=0x7ffff7f8d380) at lib_ffi.c:370
#17 0x0000000000748c8f in lj_cf_ffi_clib___index (L=0x7ffff7f8d380) at lib_ffi.c:375
#18 0x000000000074e1b6 in lj_BC_FUNCC ()
#19 0x0000000000729047 in lua_pcall (L=0x7ffff7f8d380, nargs=0, nresults=0, errfunc=0) at lj_api.c:1151
#20 0x00000000004eae11 in Shell::load_string (this=0x1603aa0,
    s=0xe9ce18 "\n", '-' <repeats 75 times>, "\n-- Copyright (C) 2014-2023 Cisco and/or its affiliates. All rights reserved.\n--\n-- This program is free software; you can r"..., load_in_sandbox=<optimized out>,
    message=0xe9cc44 "bootstrap") at /home/sfk/snort3/src/main/shell.cc:464

I've come through this problem before, when I forgot to add `-rdynamic`, so Lua cannot find the symbol `snort_whitelist_append`.

So I added `-rdynamic` and the world was saved.

However, that only applies to dynamically-linked snort. Since I was trying to do static linking, `-rdynamic` no longer exports symbols, and thus Lua just cannot find them.

Hope I've made things clear.

---

Thanks,

Fengkai

--
You received this message because you are subscribed to a topic in the Google Groups "lua-l" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lua-l/qasVn1YX7m8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lua-l+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/CAOnDXmEHCk%2BASC00mRSmr-tr8RC9_n6q-VACoiOSv7w8M0oYhQ%40mail.gmail.com.

[Sainan]

I'd say something about the fact that you have that kind of stack trace suggests to me your Lua error configuration is incorrect (it should use a longjump, not raise anything fatal your debugger would catch), but that looks an awful lot like LuaJIT, which is a fork of a very old Lua version, so maybe your question is better directed at the LuaJIT community?

[Fengkai Sun]

Oh, I did not know about the distinction between Lua and LuaJIT. Indeed snort is using LuaJIT. Thanks for pointing that out!

On Wed, Jul 10, 2024 at 4:31 PM 'Sainan' via lua-l <lu...@googlegroups.com> wrote:

I'd say something about the fact that you have that kind of stack trace suggests to me your Lua error configuration is incorrect (it should use a longjump, not raise anything fatal your debugger would catch), but that looks an awful lot like LuaJIT, which is a fork of a very old Lua version, so maybe your question is better directed at the LuaJIT community?

--

You received this message because you are subscribed to a topic in the Google Groups "lua-l" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lua-l/qasVn1YX7m8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lua-l+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lua-l/5dooFuk5iD1mTj5gjFd4RcWq7vJpsw46xXXkH7g-DXHu-K95KU9Q4xx8prEy9pZf9rFHs7_Kn84Mn8uARMMeuOLkQV9pWkEnUoLFDS4yV2A%3D%40calamity.gg.

[Soni "They/Them" L.]

have you checked that the static binary is exporting the necessary (lua)
symbols? (we're not sure static binaries can do this honestly, but
anyway) (alternatively, we seem to recall lua 5.4 making it possible for
the module to link its own lua, as long as it's the same
version/compiled with the same settings(?))

>
> -spc
>
> [1] https://github.com/spc476/SPCUUID/blob/3fc5c6d58ac9ddfb60cd60087c08e037dbdd6e5c/src/luauuid.c#L222
>