info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Link at runtime instead of build-time
60 views
Skip to first unread message
Frederick Virchanza Gotham
Mar 5, 2023, 12:59:55 PM3/5/23
to
Let's say I have one program and one shared library on a *nix computer.
I build my program to link dynamically with the library as follows:
g++ -o prog prog.cpp -l:libmonkey.so
So let's say that this produces an executable file that works fine.
Next I change the build command to the following:
g++ -o prog prog.cpp
And so now of course I get a linker error because of unresolved symbols (I'm missing the functions implemented in the library). To overcome this linker error, I change the build command again:
g++ -o prog prog.cpp -Wl,--unresolved-symbols=ignore-in-object-files
So now we get a viable executable file, but when we run it, it segfaults. It segfaults when it tries to access something from the shared library.
So next I make a one-line change at the beginning of 'main':
int main(void)
{
dlopen("libmonkey.so", RTLD_NOW|RTLD_GLOBAL);
// The rest of main goes here
}
So now the program works fine.
I think however, that this strategy only works when the program doesn't use global variables which are exported from the library. That is to say, it works fine for exported functions but not for exported global variables. Am I right here?
I have more testing to do on this, but I need to get this working. I need to remove the library from the 'NEEDED' section in the program's ELF file, and instead I have to use 'dlopen' at runtime to load it in.
How can I get this to work with the global variables also? I've tried declaring the global variable as a weak symbol in the program, and that seems to work, but I don't want to have to go looking for a list of all the global objects exported from the shared library.
Also I realise that things get complicated if there's a global object in the program whose constructor accesses the shared library (in which case I need to make use of __attribute__((constructor)). But I'll deal with thay conundrum later.
I build my program to link dynamically with the library as follows:
g++ -o prog prog.cpp -l:libmonkey.so
So let's say that this produces an executable file that works fine.
Next I change the build command to the following:
g++ -o prog prog.cpp
And so now of course I get a linker error because of unresolved symbols (I'm missing the functions implemented in the library). To overcome this linker error, I change the build command again:
g++ -o prog prog.cpp -Wl,--unresolved-symbols=ignore-in-object-files
So now we get a viable executable file, but when we run it, it segfaults. It segfaults when it tries to access something from the shared library.
So next I make a one-line change at the beginning of 'main':
int main(void)
{
dlopen("libmonkey.so", RTLD_NOW|RTLD_GLOBAL);
// The rest of main goes here
}
So now the program works fine.
I think however, that this strategy only works when the program doesn't use global variables which are exported from the library. That is to say, it works fine for exported functions but not for exported global variables. Am I right here?
I have more testing to do on this, but I need to get this working. I need to remove the library from the 'NEEDED' section in the program's ELF file, and instead I have to use 'dlopen' at runtime to load it in.
How can I get this to work with the global variables also? I've tried declaring the global variable as a weak symbol in the program, and that seems to work, but I don't want to have to go looking for a list of all the global objects exported from the shared library.
Also I realise that things get complicated if there's a global object in the program whose constructor accesses the shared library (in which case I need to make use of __attribute__((constructor)). But I'll deal with thay conundrum later.
Christian Gollwitzer
Mar 5, 2023, 1:12:47 PM3/5/23
to
Am 05.03.23 um 18:59 schrieb Frederick Virchanza Gotham:
required library at runtime? Do you want to adjust the path at
initialization? If it is only about packaging the library with the
application, you could use the run-path when linking; e.g. say your
library is distributed in a lib folder alongside your executable, then
-Wl,-rpath,'$ORIGIN'/lib
in the linker line would do that. Another alternative is to write a
small shell script which adjusts LD_LIBRARY_PATH before invoking the binary.
I can see the need for dlopen() only in case where the functionality is
somewhat optional, like in a plugin system.
Christian
> Let's say I have one program and one shared library on a *nix computer.
>
> I build my program to link dynamically with the library as follows:
>
> g++ -o prog prog.cpp -l:libmonkey.so
>
> So let's say that this produces an executable file that works fine.
>
> Next I change the build command to the following:
>
> g++ -o prog prog.cpp
>
> And so now of course I get a linker error because of unresolved symbols (I'm missing the functions implemented in the library). To overcome this linker error, I change the build command again:
>
> g++ -o prog prog.cpp -Wl,--unresolved-symbols=ignore-in-object-files
>
> So now we get a viable executable file, but when we run it, it segfaults. It segfaults when it tries to access something from the shared library.
>
> So next I make a one-line change at the beginning of 'main':
>
> int main(void)
> {
> dlopen("libmonkey.so", RTLD_NOW|RTLD_GLOBAL);
>
> // The rest of main goes here
> }
>
> So now the program works fine.
>
> I think however, that this strategy only works when the program doesn't use global variables which are exported from the library. That is to say, it works fine for exported functions but not for exported global variables. Am I right here?
>
> I have more testing to do on this, but I need to get this working. I need to remove the library from the 'NEEDED' section in the program's ELF file, and instead I have to use 'dlopen' at runtime to load it in.
As always, I'm questioning this NEED. Why do you want to load the
>
> I build my program to link dynamically with the library as follows:
>
> g++ -o prog prog.cpp -l:libmonkey.so
>
> So let's say that this produces an executable file that works fine.
>
> Next I change the build command to the following:
>
> g++ -o prog prog.cpp
>
> And so now of course I get a linker error because of unresolved symbols (I'm missing the functions implemented in the library). To overcome this linker error, I change the build command again:
>
> g++ -o prog prog.cpp -Wl,--unresolved-symbols=ignore-in-object-files
>
> So now we get a viable executable file, but when we run it, it segfaults. It segfaults when it tries to access something from the shared library.
>
> So next I make a one-line change at the beginning of 'main':
>
> int main(void)
> {
> dlopen("libmonkey.so", RTLD_NOW|RTLD_GLOBAL);
>
> // The rest of main goes here
> }
>
> So now the program works fine.
>
> I think however, that this strategy only works when the program doesn't use global variables which are exported from the library. That is to say, it works fine for exported functions but not for exported global variables. Am I right here?
>
> I have more testing to do on this, but I need to get this working. I need to remove the library from the 'NEEDED' section in the program's ELF file, and instead I have to use 'dlopen' at runtime to load it in.
required library at runtime? Do you want to adjust the path at
initialization? If it is only about packaging the library with the
application, you could use the run-path when linking; e.g. say your
library is distributed in a lib folder alongside your executable, then
-Wl,-rpath,'$ORIGIN'/lib
in the linker line would do that. Another alternative is to write a
small shell script which adjusts LD_LIBRARY_PATH before invoking the binary.
I can see the need for dlopen() only in case where the functionality is
somewhat optional, like in a plugin system.
Christian
Frederick Virchanza Gotham
Mar 5, 2023, 2:12:07 PM3/5/23
to
> As always, I'm questioning this NEED.
Depending upon the command-line arguments to the program, it might enter into GUI mode, and if it does, it needs to load the GUI libraries.
But if it runs in console mode, I don't want the system to need to have the Gui libraries installed.
Christian Gollwitzer
Mar 5, 2023, 4:46:04 PM3/5/23
to
Am 05.03.23 um 20:11 schrieb Frederick Virchanza Gotham:
This usually works by exporting a known entry function from the library,
load the library by dlopen() and find the function using dlsym(). Put
all the functionality into the library, so that you can invoke the GUI
stuff with a simple call. It doesn't make sense to have global variables
in the main program have influenced by loadable library.
If you need a lot of the functionality of the main program in the
library, then instead put this functionality in a library if its own
(say, libengine.so). Then the main program reduces to
if (std::string(argv[1] == "-gui") {
dlopen();
run_gui = dlsym("...");
run_gui();
} else {
run_cmdline();
}
Link both this main and the GUI library with libengine.so.
Christian
load the library by dlopen() and find the function using dlsym(). Put
all the functionality into the library, so that you can invoke the GUI
stuff with a simple call. It doesn't make sense to have global variables
in the main program have influenced by loadable library.
If you need a lot of the functionality of the main program in the
library, then instead put this functionality in a library if its own
(say, libengine.so). Then the main program reduces to
if (std::string(argv[1] == "-gui") {
dlopen();
run_gui = dlsym("...");
run_gui();
} else {
run_cmdline();
}
Link both this main and the GUI library with libengine.so.
Christian
Frederick Virchanza Gotham
Mar 5, 2023, 7:10:11 PM3/5/23
to
On Sunday, March 5, 2023 at 9:46:04 PM UTC, Christian Gollwitzer wrote:
>
> If you need a lot of the functionality of the main program in the
> library, then instead put this functionality in a library if its own
> (say, libengine.so).
My program has to be one statically-linked executable file.
>
> If you need a lot of the functionality of the main program in the
> library, then instead put this functionality in a library if its own
> (say, libengine.so).
I found the following doing a web search:
https://stackoverflow.com/questions/20658809/dynamic-loading-and-weak-symbol-resolution
If you scroll down on that page, a guy called Praxeolitic writes "Here is an example in which the executable itself has an undefined symbol that is resolved through dynamic loading", and he gives a code example. His code example is /exactly/ what I'm trying to achieve, but it doesn't work on modern Linux. It seems to have worked 9 years ago, but something has changed since then. I've tried using gcc-4.8 on modern Linux but the executable I get doesn't work.
Does anyone know what might have changed in Linux in the past 10 years to stop this working? Or does anyone have an email address for Praxeolitic? Anyone know who Praxeolitic is?
I think I'll download an old copy of Ubuntu from 10 years ago and run it in a VM and try this out.
Christian Gollwitzer
Mar 6, 2023, 3:29:58 AM3/6/23
to
Am 06.03.23 um 01:10 schrieb Frederick Virchanza Gotham:
Have you seen the comment below this message from 2018, that the dynamic
linker only respects weak symbols when you set LD_DYNAMIC_WEAK? Quoting
from the referenced page:
LD_DYNAMIC_WEAK (since glibc 2.1.91)
By default, when searching shared libraries to resolve a
symbol reference, the dynamic linker will resolve to the
first definition it finds.
Old glibc versions (before 2.2), provided a different
behavior: if the linker found a symbol that was weak, it
would remember that symbol and keep searching in the
remaining shared libraries. If it subsequently found a
strong definition of the same symbol, then it would
instead use that definition. (If no further symbol was
found, then the dynamic linker would use the weak symbol
that it initially found.)
The old glibc behavior was nonstandard. (Standard
practice is that the distinction between weak and strong
symbols should have effect only at static link time.) In
glibc 2.2, the dynamic linker was modified to provide the
current behavior (which was the behavior that was provided
by most other implementations at that time).
Defining the LD_DYNAMIC_WEAK environment variable (with
any value) provides the old (nonstandard) glibc behavior,
whereby a weak symbol in one shared library may be
overridden by a strong symbol subsequently discovered in
another shared library. (Note that even when this
variable is set, a strong symbol in a shared library will
not override a weak definition of the same symbol in the
main program.)
Since glibc 2.3.4, LD_DYNAMIC_WEAK is ignored in secure-
execution mode.
This sounds very hackish to me. Most portable (and easiest) solution
will be a shell script. The overall goal reminds me to the Tk toolkit.
In Tcl, you can do "package require Tk" which loads the Tk toolkit
dynamically. However, there are at least two files - Tk is its own
shared object, which depends on libX11 etc., while Tcl alone does not
depend on libX11. For a single file executable (called starpack) they go
through some hoops, especially attaching a load to the end of the
executable (similar to a self-extracting ZIP file), which is addressed
as a virtual file system. Are you sure it is worth to go through this
for your program? The simplest solution (besides a shell script) would
be to split the GUI part into its own library, which is installed as a
separate package.
Christian
linker only respects weak symbols when you set LD_DYNAMIC_WEAK? Quoting
from the referenced page:
LD_DYNAMIC_WEAK (since glibc 2.1.91)
By default, when searching shared libraries to resolve a
symbol reference, the dynamic linker will resolve to the
first definition it finds.
Old glibc versions (before 2.2), provided a different
behavior: if the linker found a symbol that was weak, it
would remember that symbol and keep searching in the
remaining shared libraries. If it subsequently found a
strong definition of the same symbol, then it would
instead use that definition. (If no further symbol was
found, then the dynamic linker would use the weak symbol
that it initially found.)
The old glibc behavior was nonstandard. (Standard
practice is that the distinction between weak and strong
symbols should have effect only at static link time.) In
glibc 2.2, the dynamic linker was modified to provide the
current behavior (which was the behavior that was provided
by most other implementations at that time).
Defining the LD_DYNAMIC_WEAK environment variable (with
any value) provides the old (nonstandard) glibc behavior,
whereby a weak symbol in one shared library may be
overridden by a strong symbol subsequently discovered in
another shared library. (Note that even when this
variable is set, a strong symbol in a shared library will
not override a weak definition of the same symbol in the
main program.)
Since glibc 2.3.4, LD_DYNAMIC_WEAK is ignored in secure-
execution mode.
This sounds very hackish to me. Most portable (and easiest) solution
will be a shell script. The overall goal reminds me to the Tk toolkit.
In Tcl, you can do "package require Tk" which loads the Tk toolkit
dynamically. However, there are at least two files - Tk is its own
shared object, which depends on libX11 etc., while Tcl alone does not
depend on libX11. For a single file executable (called starpack) they go
through some hoops, especially attaching a load to the end of the
executable (similar to a self-extracting ZIP file), which is addressed
as a virtual file system. Are you sure it is worth to go through this
for your program? The simplest solution (besides a shell script) would
be to split the GUI part into its own library, which is installed as a
separate package.
Christian
Frederick Virchanza Gotham
Mar 6, 2023, 5:37:12 AM3/6/23
to
On Monday, March 6, 2023 at 8:29:58 AM UTC, Christian Gollwitzer wrote:
>
> Have you seen the comment below this message from 2018, that the dynamic
> linker only respects weak symbols when you set LD_DYNAMIC_WEAK?
LD_DYNAMIC_WEAK isn't what's going on here.
>
> Have you seen the comment below this message from 2018, that the dynamic
> linker only respects weak symbols when you set LD_DYNAMIC_WEAK?
I managed to get it all working just now. When building the executable, you have to give -Wl,--export-dynamic
Now it's working as intended.
Alf P. Steinbach
Mar 6, 2023, 9:45:34 AM3/6/23
to
> where devenv
C:\Program Files\Microsoft Visual
Studio\2022\Community\Common7\IDE\devenv.com
C:\Program Files\Microsoft Visual
Studio\2022\Community\Common7\IDE\devenv.exe
Guess which one has console subsystem and which has GUI subsystem?
Well, not that hard to guess, because invoked by command should choose
the console subsystem one:
[C:\Users\Alf P. Steinbach\Desktop]
> set pathext
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.LNK
PATHEXT-SYS=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
In passing, some years ago MS sabotaged the use of `.lnk` files as
commands, as I recall it had to do with argument passing. The question
of how to differentiate between sabotage and incompetence is an unsolved
philosophical question. But I simply label it sabotage when it comes
from Microsoft b/c I don't think one should discount earlier history.
- Alf
Mut...@dastardlyhq.com
Mar 6, 2023, 12:14:45 PM3/6/23
to
On Mon, 6 Mar 2023 15:45:17 +0100
"Alf P. Steinbach" <alf.p.s...@gmail.com> wrote:
>On 2023-03-05 8:11 PM, Frederick Virchanza Gotham wrote:
>>
>>> As always, I'm questioning this NEED.
>>
>>
>> I'm combining a console program and a GUI program into one program
>> (i.e. one executable).
>>
>> Depending upon the command-line arguments to the program, it might
>> enter into GUI mode, and if it does, it needs to load the GUI
>> libraries.
>>
>> But if it runs in console mode, I don't want the system to need to
>> have the Gui libraries installed.
>
>[C:\Users\Alf P. Steinbach\Desktop]
> > where devenv
>C:\Program Files\Microsoft Visual
>Studio\2022\Community\Common7\IDE\devenv.com
>C:\Program Files\Microsoft Visual
>Studio\2022\Community\Common7\IDE\devenv.exe
>
>
>Guess which one has console subsystem and which has GUI subsystem?
>
>Well, not that hard to guess, because invoked by command should choose
>the console subsystem one:
Use a version of *nix. It doesn't have this cretinous differentiation between
"Alf P. Steinbach" <alf.p.s...@gmail.com> wrote:
>On 2023-03-05 8:11 PM, Frederick Virchanza Gotham wrote:
>>
>>> As always, I'm questioning this NEED.
>>
>>
>> I'm combining a console program and a GUI program into one program
>> (i.e. one executable).
>>
>> Depending upon the command-line arguments to the program, it might
>> enter into GUI mode, and if it does, it needs to load the GUI
>> libraries.
>>
>> But if it runs in console mode, I don't want the system to need to
>> have the Gui libraries installed.
>
>[C:\Users\Alf P. Steinbach\Desktop]
> > where devenv
>C:\Program Files\Microsoft Visual
>Studio\2022\Community\Common7\IDE\devenv.com
>C:\Program Files\Microsoft Visual
>Studio\2022\Community\Common7\IDE\devenv.exe
>
>
>Guess which one has console subsystem and which has GUI subsystem?
>
>Well, not that hard to guess, because invoked by command should choose
>the console subsystem one:
a console or GUI program. They're just programs.
Frederick Virchanza Gotham
Mar 6, 2023, 4:42:06 PM3/6/23
to
Actually instead I think I'm going to link the program dynamically as normal, but then after the executable file is built, use 'patchelf' on it to remove some of the 'NEEDED' libraries, so then I can just use 'dlopen' at runtime.
Kenny McCormack
Mar 6, 2023, 5:20:46 PM3/6/23
to
In article <fa55ce4e-39e4-45e0...@googlegroups.com>,
You do you...
--
Kenny, I'll ask you to stop using quotes of mine as taglines.
- Rick C Hodgin -
--
Kenny, I'll ask you to stop using quotes of mine as taglines.
- Rick C Hodgin -
Christian Gollwitzer
Mar 6, 2023, 5:59:01 PM3/6/23
to
Am 06.03.23 um 22:41 schrieb Frederick Virchanza Gotham:
beginning to think that you do these things for fun. I can sympathise
with this idea, hacking is fun.
Christian
>
> Actually instead I think I'm going to link the program dynamically as normal, but then after the executable file is built, use 'patchelf' on it to remove some of the 'NEEDED' libraries, so then I can just use 'dlopen' at runtime.
Who would have thought that you come up with another bizarre hack. I'm
> Actually instead I think I'm going to link the program dynamically as normal, but then after the executable file is built, use 'patchelf' on it to remove some of the 'NEEDED' libraries, so then I can just use 'dlopen' at runtime.
beginning to think that you do these things for fun. I can sympathise
with this idea, hacking is fun.
Christian
0 new messages