Statically linking mod_spatialite

[Marco Manino]

Hi everybody!

I have been trying to build mod_spatialite on Linux for few days with no luck. While the "usual" ./configure && make pattern works I wanted to build mod_spatialite.so statically (e.g. a dynamic library without dynamic dependencies), so ideally it would not depend on anything (apart from basic C/C++ libraries and system stuff like pthread, libm and so on).

While I could do that by rewriting the build system with other tools (e.g. CMake + Conan.io), I would prefer not to (we have a CI/CD build on a GitHub mirror of spatialite and we wish to use the trunk branch for our build, instead of manually merging every time).

I can see from this thread that it may not be possible to detach sqlite3 without leaving out KNN, but that's fine for me (I'm not planning to use that function).

My most successful attempts so fare have been to add "-static" to CFLAGS like:

./configure --enable-rttopo=no --enable-module-only --enable-libxml2=no --enable-examples=no CFLAGS="-static -fPIC"

Which breaks as some of the AC_CHECK_LIB calls don't list all the required library (it's ok for shared builds as those symbols are transitively brought in by shared libraries). I'm attaching what I changed as I think dependencies should be listed even in non-static builds.

After those changes it builds, but I can still see the dependencies:

> ldd src/.libs/mod_spatialite.so
        linux-vdso.so.1 (0x00007fff665de000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9726518000)
        libminizip.so.1 => /lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f972630d000)
        libgeos_c.so.1 => /lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f97262ca000)
        libfreexl.so.1 => /lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f97262bf000)
        libproj.so.15 => /lib/x86_64-linux-gnu/libproj.so.15 (0x00007f9725fd0000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9725fb4000)
        libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f9725e89000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9725d3a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9725b48000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9726e27000)
        libgeos-3.8.0.so => /lib/x86_64-linux-gnu/libgeos-3.8.0.so (0x00007f972597f000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f972579e000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9725783000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f972577b000)

Can someone point me to the right direction/documentation?

Thank you,

Marco

[Marco Manino]

And... Of course I forgot to attach the diff.

diff --git a/configure.ac b/configure.ac
index e1798407..0113eb86 100644
--- a/configure.ac
+++ b/configure.ac
@@ -115,7 +115,7 @@ AC_FUNC_STRFTIME
 AC_CHECK_FUNCS([memset sqrt strcasecmp strerror strncasecmp strstr fdatasync ftruncate getcwd gettimeofday localtime_r memmove strerror])
 
 # Checks for installed libraries
-AC_CHECK_LIB(sqlite3,sqlite3_prepare_v2,,AC_MSG_ERROR(['libsqlite3' is required but it doesn't seem to be installed on this system.]),-lm)
+AC_CHECK_LIB(sqlite3,sqlite3_prepare_v2,,AC_MSG_ERROR(['libsqlite3' is required but it doesn't seem to be installed on this system.]),-lm -lpthread -ldl)
 AC_CHECK_LIB(z,inflateInit_,,AC_MSG_ERROR(['libz' is required but it doesn't seem to be installed on this system.]),-lm)
 
 AC_CONFIG_FILES([Makefile \
@@ -251,8 +251,10 @@ AC_ARG_ENABLE(proj, [AS_HELP_STRING(
 if test x"$enable_proj" != "xno"; then
   AC_CHECK_HEADERS(proj.h, [proj_new_incl=1],
     [AC_CHECK_HEADERS(proj_api.h, [proj_new_incl=0], [AC_MSG_ERROR([cannot find proj_api.h, bailing out])])])
+  AC_LANG_PUSH([C++])
   AC_SEARCH_LIBS(proj_normalize_for_visualization, proj, [proj_new_lib=1],
-    [AC_SEARCH_LIBS(pj_init_plus, proj, [proj_new_lib=0], [AC_MSG_ERROR(['libproj' is required but it doesn't seem to be installed on this system.])], [-lm -lpthread -lsqlite3])])
+    [AC_SEARCH_LIBS(pj_init_plus, proj, [proj_new_lib=0], [AC_MSG_ERROR(['libproj' is required but it doesn't seem to be installed on this system.])], -lm -lpthread -lsqlite3)], -lm -lpthread -lsqlite3)
+  AC_LANG_POP
   if test $proj_new_incl -eq 1 && test $proj_new_lib -eq 1; then
     AC_DEFINE(PROJ_NEW)
   fi
@@ -348,11 +350,10 @@ if test x"$enable_geos" != "xno"; then
   AC_CHECK_HEADERS([geos_c.h],, [AC_MSG_ERROR([could not find geos_c.h - you may need to specify the directory of a geos-config file using --with-geosconfig])])
   CPPFLAGS="$CPPFLAGS_SAVE"   
   # Ensure we can link against libgeos_c
-  LIBS_SAVE="$LIBS"
-  LIBS="$GEOS_LDFLAGS"
-  AC_SEARCH_LIBS(GEOSCoveredBy,geos_c,,AC_MSG_ERROR([could not find libgeos_c (or obsolete 'libgeos_c' < v.3.3.0 found) - you may need to specify the directory of a geos-config file using --with-geosconfig]))
-  LIBS="$LIBS_SAVE"
-  LIBS="$LIBS $GEOS_LDFLAGS -lgeos_c"
+  LDFLAGS="$LDFLAGS $GEOS_LDFLAGS"
+  AC_LANG_PUSH([C++])
+  AC_SEARCH_LIBS(GEOSCoveredBy,geos_c,,AC_MSG_ERROR([could not find libgeos_c (or obsolete 'libgeos_c' < v.3.3.0 found) - you may need to specify the directory of a geos-config file using --with-geosconfig]), -lgeos)
+  AC_LANG_POP
 
   #-----------------------------------------------------------------------
   #   --enable-controlpoints
@@ -371,8 +372,10 @@ if test x"$enable_geos" != "xno"; then
       [--enable-geosadvanced], [enables GEOS advanced features [default=yes]])],
       [], [enable_geosadvanced=yes])
   if test x"$enable_geosadvanced" != "xno"; then
-      AC_SEARCH_LIBS(GEOSDelaunayTriangulation,geos_c,,AC_MSG_ERROR(['libgeos_c' (>= v.3.4.0) is required but it doesn't seem to be installed on this system. You may need to try re-running configure with a --disable-geosadvanced parameter.]))
+      AC_LANG_PUSH([C++])
+      AC_SEARCH_LIBS(GEOSDelaunayTriangulation,geos_c,,AC_MSG_ERROR(['libgeos_c' (>= v.3.4.0) is required but it doesn't seem to be installed on this system. You may need to try re-running configure with a --disable-geosadvanced parameter.]), -lgeos)
       AC_DEFINE(GEOS_ADVANCED)
+      AC_LANG_POP
   fi
 
   #-----------------------------------------------------------------------
@@ -382,8 +385,10 @@ if test x"$enable_geos" != "xno"; then
       [--enable-geosreentrant], [enables GEOS reentrant (fully thread safe) [default=yes]])],
       [], [enable_geosreentrant=yes])
   if test x"$enable_geosreentrant" != "xno"; then
-      AC_SEARCH_LIBS(GEOSContext_setErrorMessageHandler_r,geos_c,,AC_MSG_ERROR(['libgeos_c' (>= v.3.5.0) is required but it doesn't seem to be installed on this system. You may need to try re-running configure with a --disable-geosreentrant parameter.]))
+      AC_LANG_PUSH([C++])
+      AC_SEARCH_LIBS(GEOSContext_setErrorMessageHandler_r,geos_c,,AC_MSG_ERROR(['libgeos_c' (>= v.3.5.0) is required but it doesn't seem to be installed on this system. You may need to try re-running configure with a --disable-geosreentrant parameter.]), -lgeos)
       AC_DEFINE(GEOS_REENTRANT)
+      AC_LANG_POP
   fi
 
   #-----------------------------------------------------------------------
@@ -403,8 +408,10 @@ if test x"$enable_geos" != "xno"; then
       [--enable-geos370], [enables GEOS 3.7.0 features [default=yes]])],
       [], [enable_geos370=yes])
   if test x"$enable_geos370" != "xno"; then
-      AC_SEARCH_LIBS(GEOSFrechetDistance_r,geos_c,,AC_MSG_ERROR(['libgeos_c' (>= v.3.7.0) is required but it doesn't seem to be installed on this system. You may need to try re-running configure with a --disable-geos370 parameter.]))
+      AC_LANG_PUSH([C++])
+      AC_SEARCH_LIBS(GEOSFrechetDistance_r,geos_c,,AC_MSG_ERROR(['libgeos_c' (>= v.3.7.0) is required but it doesn't seem to be installed on this system. You may need to try re-running configure with a --disable-geos370 parameter.]), -lgeos)
       AC_DEFINE(GEOS_370)
+      AC_LANG_POP
   fi
 
   #-----------------------------------------------------------------------

[a.fu...@lqt.it]

Hi Marco,

attempting to build a version of mod_spatialite statically
linked to all its dependencies (such as libproj, geos and so
on) is a very bad idea; really very, very bad.

if this is true in a general way on any operating system, it
becomes a real absurdity on Linux, because this way you'll
completely forbid any opportunity to automatically update your
libraries each time that some critical bug has been identified
and resolved to be then distributed by the packaging system of
your distro..
definitely not a wise option, that could pose at serious risk the
safety and stability of your system.

that said, if you are really convinced that it's the ideal solution
for your needs, building an extravagant oddity such as a statically
linked mod_spatialite is not at all technically impossible.

don't waste your time with the -static flag, because you'll soon
discover that too many libraries will still remain dynamically
linked.

as per my personal experience the easiest way is to simply
write a trivial Makefile carefully avoiding to link any
depending library by its symbolic name but using instead
the absolute path of the static library.
such a Makefile will simply be intended just for doing
the static linkage _after_ successfully building using
the most canonic approach.

so, not: -lproj -lgeos -lm -lpthread

but instead:
/usr/lib/libproj.a \
/usr/lib/libgeos_c.a \
/usr/lib/libgeos.a \
-lm -lpthread

NOTE: we'll still continue using -lm -lpthread because these
are common system libraries so we'll use dynamic linkeage
for them.

you can surely get some useful inspiration by studing my
personal recipes for statically linking on Windows (that's
not really different from Linux, you can easily adapt).

http://www.gaia-gis.it/gaia-sins/mingw32_how_to.html

look at the recipe for statically building tbe spatilite-tools
and study how Makefile-static-mingw32 is internally structured
(you'll find this Makefile within the source tarball).

bye Sandro

[Marco Manino]

Hi Sandro,

first of all, thank you for your quick response.

I am well aware of the (many) downsides of static linking (which, on Windows are even worse for c++ projects like proj). I probably should have made some context first.

We are trying to use spatialite in a C# project that supports both Linux and Windows. The main problem with that is that relying on the system-wide package manager becomes very difficult for many reasons that are not linked to the linux environment, but on how C# handles "native" dependencies.

We have many options we can choose, among which I try to list the "best" three we came up with:

 - just include .dll and .so in the project - very bad (and our current solution, unfortunately). It also has problems (see below).

 - use the system-wide package manager on linux and just call LoadExtension hoping that mod_spatialite will be in the PATH. This is very hard to maintain and easily breaks. On Windows, there is no package manager, so it is even worse. Also, there is no way for the code to track which version it needs so if some C# code calls a newer function and a user has an old one, it will break not during the "LoadExtension" phase but later on, if the code path that uses the function is reached. Very very bad.

 - use a NuGet package. This seemed the only way to sanely use spatialite in C#, but of course it needs to bring in the build *all* the files it needs (it would be very very bad to install a package just to discover later that you also need to install a system-wide one).

To add on top of those problems there is also another one: we use System.Data.Sqlite package from sqlite.org guys, but they are building in a different way w.t.r. the "usual" way (= the way you would expect to find on any system-wide package). They bundle a kind-of amalgamation sqlite with some wrapper method to be used by C# in... well the way C# needs. They named this module "SQLite.Interop.dll" (they called that way on Linux, too), so if the module requires any symbol from libsqlite.{so,dll,dylib} it will just fail to load.

With this in mind, we think that there are just two ways to go ahead:

 - build mod_spatialite dynamically and bring in all the .so/.dll it needs;

 - build mod_spatialite statically.

The first one has a discoverability problem though: we won't know which libraries it will really need to bundle as it depends as it depends on how the maintainer of the libraries bult the dependencies in the first place. Moreover, we would really love to have a CI/CD pipeline build and publish this package for us, so that we can manage security upgrades and bug fixing. Either we use a chroot-ed environment to isolate the dependencies (which is very hard to setup in a CI/CD and it works on linux alone) or we hardcode dependencies by hand. The first option is bad, the second is worse (as any update from your part will break the build).

This is why we want to build it statically: it just seemed right in our context. It is also what most of the libraries are doing (System.Data.Sqlite is among them btw), so it is right within the dotnet context.

Ideally, we wanted a script that just builds the library on each of your release, without having us check if any new file has been added/updated/moved and this points to the configure script on linux and on the nmake one on windows.

However, if I understood correctly what you are saying, it seems just not possible, so maybe we should just fall back to using a side-script and handle things manually - which is bad. Given the context, do you think that would be the only way ahead? Are we missing some important detail that would help us achieve our goal?

Thank you for your time,

Marco Manino

[a.fu...@lqt.it]

On Thu, 24 Jun 2021 03:44:01 -0700 (PDT), Marco Manino wrote:
> Hi Sandro,
>
> first of all, thank you for your quick response.
>
> I am well aware of the (many) downsides of static linking (which, on
> Windows are even worse for c++ projects like proj). I probably should
> have made some context first.
>
> We are trying to use spatialite in a C# project that supports both
> Linux and Windows. The main problem with that is that relying on the
> system-wide package manager becomes very difficult for many reasons
> that are not linked to the linux environment, but on how C# handles
> "native" dependencies.
>
> We have many options we can choose, among which I try to list the
> "best" three we came up with:
> - just include .dll and .so in the project - very bad (and our
> current solution, unfortunately). It also has problems (see below).
>
> - use the system-wide package manager on linux and just call

> _LoadExtension_ hoping that mod_spatialite will be in the PATH. This
> is _very_ hard to maintain and easily breaks. On Windows, there is no

> package manager, so it is even worse. Also, there is no way for the

> code to track which version it _needs_ so if some C# code calls a

> newer function and a user has an old one, it will break not during
> the
> "LoadExtension" phase but later on, if the code path that uses the

> function is reached. _Very very_ bad.

>
> - use a NuGet package. This seemed the only way to sanely use
> spatialite in C#, but of course it needs to bring in the build *all*

> the files it needs (it would be _very very bad _to install a package
> just to discover later that you also need to install a _system-wide
> _one).

>
> To add on top of those problems there is also another one: we use

> _System.Data.Sqlite_ package from sqlite.org guys, but they are

> building in a different way w.t.r. the "usual" way (= the way you
> would expect to find on any system-wide package). They bundle a
> kind-of amalgamation sqlite with some wrapper method to be used by C#
> in... well the way C# needs. They named this module
> "SQLite.Interop.dll" (they called that way on Linux, too), so if the
> module requires any symbol from libsqlite.{so,dll,dylib} it will just
> fail to load.
>
> With this in mind, we think that there are just two ways to go ahead:
>
> - build mod_spatialite dynamically and bring in all the .so/.dll it
> needs;
> - build mod_spatialite statically.
>
> The first one has a discoverability problem though: we won't know

> which libraries it will really need to bundle as it depends_ as it
> depends on how the maintainer of the libraries bult the dependencies_

> in the first place. Moreover, we would really love to have a CI/CD
> pipeline build and publish this package for us, so that we can manage
> security upgrades and bug fixing. Either we use a chroot-ed
> environment to isolate the dependencies (which is very hard to setup
> in a CI/CD and it works on linux alone) or we hardcode dependencies
> by
> hand. The first option is bad, the second is worse (as any update
> from
> your part will break the build).
>
> This is why we want to build it statically: it just seemed right in
> our context. It is also what most of the libraries are doing

> (_System.Data.Sqlite _is among them btw), so it is right within the

> dotnet context.
>
> Ideally, we wanted a script that just builds the library on each of
> your release, without having us check if any new file has been
> added/updated/moved and this points to the configure script on linux
> and on the nmake one on windows.
>
> However, if I understood correctly what you are saying, it seems just
> not possible, so maybe we should just fall back to using a
> side-script
> and handle things manually - which is bad. Given the context, do you
> think that would be the only way ahead? Are we missing some important
> detail that would help us achieve our goal?
>

Hi Marco,

now I understand better the problem you are attempting to resolve.

I absolutely agree with you: when the Windows "DLL Hell" is
involved adopting the static linkage is often the only reasonable
solution.

dynamic linkage is great on systems like Linux having a serious
packaging system, but on Windows it could quickly become a real
nightmare.

me first when I have to distribute pre-built packages for
Windows always choose a static linking approach, because
effectively greatly simplifies your life.

and I agree with you on a second point too: programming languages
such as Java, Python, PHP and C# apparently seems good because you
can run your apps on different platforms without recompiling.

but reality is completely different when your app extensively
depends on external libraries. you'll soon discover that
having the "right" native libraries on each platform is an
almost impossible mission.

that said: I can simply offer my personal experience about
maintaining several Makefiles supporting static linkage
on MinGW and NMAKE builds on MSVC.

I usually completely ignore all them until the last days
preceding a new relase.
then in few hours I fix all the Makefiles (mostly blindly
proceding by trial and error until all compile and link
errors are resolved).

it surely is a very crude and unelegant approach, but
it's a very economic one: it just costs me few hours every
sevaral months, and I'm reasonably happy with this

bye Sandro

[Duncan]

I also have problems with some projects loading mod_spatialite dependencies and looking 

at the opposite approach. 

Many of my projects use a spatialite database, but have very modest requirements.

Sometimes this might be as little as accessing the geometry and querying the spatial index.

I'm wondering if it would be practical, or even possible to compile mod_spatialite for various platforms

with no dependencies, to make the basic built in functions available

If so, how would that best be approached (for the C challenged)?

Currently, if I'm unable to load spatialite, I have to write my own lib for reading and writing spatialite 

geometry, (and with the best will in the world it would not be as efficient and robust as spatialite 

functions) amongst other wheels which need reinventing

Cheers

Duncan