Easiest way to use Spatialite with Rust

[Nolan Darilek]

Trying to use Spatialite in a Rust binary. I read in the
installation/usage instructions that it is enough to link libspatialite
against the binary from which you wish to use it, and its SQL functions
are then available to you if libsqlite3 is also present and linked. Is
that true, or am I expected to call any functions to initialize it first?


Just curious--I don't see libspatialite.so in output of running ldd on
my binary. So for now I'm thinking this is a rustc issue and am
preparing to follow up there, but I want to be sure that all I need to
do is link the library if I only want to use the SQL functions.


Thanks.

[mj10777]

On Friday, 19 January 2018 00:08:29 UTC+1, Nolan Darilek wrote:

Trying to use Spatialite in a Rust binary.

My answer's are based on the fact that I have no idea what a 'Rust binary' is 

- I will assume that it a normal binary that wishes to use the spatialite Dynamic-Library

I read in the
installation/usage instructions that it is enough to link libspatialite

This is only true if you wish to use the Spatialite API. 

against the binary from which you wish to use it, and its SQL functions

If you wish to use the SQL functions, you need not link to libspatialite 

- using a sqlite3 Database connection, you issue a 'SELECT mod_spatialite', after which all spatialite specific Sql-Functions are available

-- in most cases the use of the Spatialite API should not be needed

are then available to you if libsqlite3 is also present and linked. Is

Linking to and using the libsqlite3 API is, in my opinion, the most effective way.

For that you must learn how to use the Sqlite3-API properly:

https://www.sqlite.org/c3ref/open.html

https://www.sqlite.org/c3ref/funclist.html

This pseudo code will open and with the proper flags, create a new Sqlite3-Database

  bool bCreateSpatialiteDb=true;
  int i_rc = sqlite3_open_v2( filename, psqlite_handle, flags, zVfs );

  if ( i_rc == SQLITE_OK )

  {    // avoid 'not authorized' error

    sqlite3_enable_load_extension( sqlite_handle, 1 ); // always returns SQLITE_OK 

    // activating Foreign Key constraints [needed for proper internal spatialite support for admin tables (CASCADE)]

    ( void )sqlite3_exec( sqlite_handle, "PRAGMA foreign_keys = 1", nullptr, 0, nullptr );

    QString sExtension = QStringLiteral( "mod_spatialite" );

         i_rc = sqlite3_load_extension( sqlite_handle, sExtension.toUtf8().constData(), nullptr, &errMsg );

    if ( i_rc == SQLITE_OK )

    { // Spatialite is now active and usable [libspatialite.so/mod_libspatialite.so has been found]
           if (bCreateSpatialiteDb) 
           {
             char *errMsg = nullptr;

             if ( sqlite3_exec( dbSqliteHandle(), "InitSpatialMetaData(1)", nullptr, nullptr, &errMsg ) == SQLITE_OK )

             {
       // The needed Spatialite Database Metadata has been created
             }

           }

    }

  }

The rest would be to learn how to use the Spatialite Sql-Functions:

http://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html#p16

You must invest the time to learn the Sqlite3-Api and the Spatialite Sql-Functions properly.

that true, or am I expected to call any functions to initialize it first?

See the above pseudo code.

Just curious--I don't see libspatialite.so in output of running ldd on
my binary. So for now I'm thinking this is a rustc issue and am
preparing to follow up there, but I want to be sure that all I need to
do is link the library if I only want to use the SQL functions.

Using the Sqlite3-API approach, ldd will show libsqlite3.so, but not libspatialite.so/mod_libspatialite.so.

mod_libspatialite.so must, however, be found on the system when/where your program is being started.

You do not need to include spatialite *.h files or link to libspatialite.so when using the Sqlite3-API approach with Spatialite SQL functions.

If you only wish to use the Spatialite SQL functions (not the Spatialite C-API) , this is the best approach.

Hope this helps

Mark

Thanks.

[mj10777]

Correction: 'SELECT InitSpatialMetaData(1)' must be used (forgot the 'SELECT ')

[Evgeniy Dushistov]

Hi, Nolan.

пятница, 19 января 2018 г., 2:08:29 UTC+3 пользователь Nolan Darilek написал:

I use libspatialite from Rust.

There are two variants: as sqlite extension - you do not need link directly,

but you have to execute SQL statementes to initialize spatialite extension.

rusqlite supports sqlite extensions.

And link directly, then you have to call libspatialite code directly, and obviously

you have to link with it.

Personally I prefer the second variant, because of deploying issues.

To simplify things for me I build libspatialite, libsqlite and all their dependencies as static

libraries and link with my spatialite module.

And then:

```rust

        let conn = rusqlite::Connection::open(path_to_db_file)?;

        let cache = unsafe { spatialite::spatialite_alloc_connection() };

        if cache.is_null() {

            return Err(BaseError::UnderlyingDbError(

                "spatialite initialization error".to_string(),

            ));

        }

        unsafe {

            spatialite::spatialite_init_ex(conn.handle() as *mut spatialite::sqlite3, cache, 1)

        };

```

[Nolan Darilek]

Thanks! My interest is in linking with libspatialite rather than using an extension. It looks like, in that case, you do have to call a function to initialize the library. I thought that linking would somehow make the spatial SQL functions available to the embedded SQLite. I just want to use enough of the API to make the SQL functions available, then use the functions from SQL.

A Rust binary is a binary of an application written in the Rust programming language. https://www.rust-lang.org

--
You received this message because you are subscribed to the Google Groups "SpatiaLite Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spatialite-use...@googlegroups.com.
To post to this group, send email to spatiali...@googlegroups.com.
Visit this group at https://groups.google.com/group/spatialite-users.
For more options, visit https://groups.google.com/d/optout.

[Nolan Darilek]

Nice. I'd prefer the link option for easier deployment as well. So it looks like you do have to call a function in the Spatialite API to make the functions available to queries.

Is the source of your spatialite crate available anywhere? I was thinking of building a spatialite-sys crate for this purpose, though I had no idea how large the API surface is.

Thanks.

[Evgeniy Dushistov]

пятница, 19 января 2018 г., 16:35:17 UTC+3 пользователь Nolan Darilek написал:

Nice. I'd prefer the link option for easier deployment as well. So it looks like you do have to call a function in the Spatialite API to make the functions available to queries.

Also you have to use libspatialite API if one of parameters of your query is "geometry" for "geometry" -> "blob" conversion.

 

Is the source of your spatialite crate available anywhere? I was thinking of building a spatialite-sys crate for this purpose, though I had no idea how large the API surface is.

I've never thought that Rust part will be interesting to anyone (it is rather simple: ~40 lines of code in

build.rs with help of bindgen + cmake crates and voila - I can call directly spatialite functions).

The traces of my efforts can be found here:

https://github.com/Dushistov/libspatialite (develop)

https://github.com/Dushistov/geos (master)

I added cmake build system to libspatialite (+travis CI integration) to use one build system

for all platforms of interest (mac os x, iOS, Android),

fixed cmake's geos build system to build one static library with C API.

By default geos builds two libraries - one with C++ API, another one that calls the first one with C API.

Geos library is the hard dependency of libspatialite.

I have "spatialite-sys" as module (99% autogerated by bindgen) in crate rather than separate libspatialite-sys crate because of sqlite dependency. I build sqlite as static library,

and from one hand I need to setup environment variable to link this "home made" sqlite with

rusqlite crate (https://github.com/jgallagher/rusqlite),

on the other hand libspatialite also depends on sqlite.

So, if you want to create  "spatialite-sys" and add "bundle" option to build libspatialite embedded into crate,

you have to tell rusliqte to link with crate rather than with C library.

[a.fu...@lqt.it]

Hi Nolan and Evgeniy,

I personally completely ignored Rust.
After reading this [1] I finally discovered that it's yet
another innovative programming language strongly sponsored
by Mozilla and internally adopted by Firefox.

[1] https://en.wikipedia.org/wiki/Rust_(programming_language)

my abysmal ignorance about Rust obviously forbids me to enter
in specific details, but I suppose to be able to give you few
may be useful general order hints based on my long experience
about deploying spatialite.

1. avoid as far as possible the very dangerous hidden pitfalls
deriving from adopting a static linkage model.
pros: it apparently strongly simplifies the distribution
and installation of your binaries.
cons: in the long term you'll sadly discover that it could
probably cause many unexpected troubles and puzzling
conflicts (as it happened to the very first versions
of spatialite itself).
using a static linkage approach could (may be) reasonably
justified on Windows, once considered its messy and clumsy
support to dynamic libraries (the infamous "dll hell").
but it surely is a very (very) bad idea on Linux, MacOS,
Android and on any other Unix-like platform, because it
will negate all the many benefits deriving from the
centralized distribution of standard packages (that is
almost fully based on dynamic libraries).

2. loading spatialite as a dynamic extension to SQLite (via
the "load_extension" SQL call) surely is the cleanest,
simplest and safest mechanism you can adopt in order to
achieve a real and effective cross-platform portability.
please, don't overlook the many advantages deriving
from adopting a dynamically loadable external module
just because it will possibly add few extra complexity
(practically nil on Unix-like platforms: the problem
is mainly related to Windows installations).

if you are not yet convinced, pleas read this post from the
SpatiaLite's User mailing list about the sad history of
"pyspatialite", a data connector module for Python (now
happily discontinued).

https://groups.google.com/forum/#!searchin/spatialite-users/pyspatialite%7Csort:date/spatialite-users/o0jUwMUqx_g/AQghbaQcCAAJ

bye Sandro

[Evgeniy Dushistov]

Hi  Sandro.

пятница, 19 января 2018 г., 19:30:24 UTC+3 пользователь sandro furieri написал:

    but it surely is a very (very) bad idea on 

... 

    Android and on any other Unix-like platform, because it
    will negate all the many benefits deriving from the
    centralized distribution of standard packages (that is
    almost fully based on dynamic libraries).

 

...

 

2. loading spatialite as a dynamic extension to SQLite (via
    the "load_extension" SQL call) surely is the cleanest,
    simplest and safest mechanism you can adopt in order to
    achieve a real and effective cross-platform portability.
   

I am afraid but I don't understand you,

how extensions and shared library in portable way can work on Android and iOS?

Android dlopen:

https://stackoverflow.com/questions/15719149/how-to-integrate-native-runtime-library-with-dlopen-on-ndk

> The main problem with implementing an equivalent of System.LoadLibrary() in C, is that the location /data/data/<your.package>/lib is a non-documented implementation detail 

iOS dlopen:

https://stackoverflow.com/questions/6530701/is-the-function-dlopen-private-api

> as of iOS 8, extensions and shared frameworks are prohibited from using dlopen

[Nolan Darilek]

Did I miss your build.rs? Couldn't find it, though I'm still a bit coffee-deprived this morning. :)

In any case, I knocked this together:

https://gitlab.com/ndarilek/spatialite-sys

Links directly against libspatialite dynamically, with a pkg-config fallback where present. Is that basically what you did as well? I'm wondering if I need any extra munging in build.rs.

I understand there's still higher level work to do. My intent with this is to package away the spatialite linking/dependency in a separate crate so higher-level apps don't need to concern themselves with that. Even though it is only a few lines of integration work, it probably makes sense to standardize on these lower-level building blocks where possible, then build higher-level adapters to rusqlite and others as needed.

Thanks.

[Evgeniy Dushistov]

Plus in iOS and Android applications run in sandboxes.

And sandbox become more restrictive every realease.

Yes at least on Android it is possible to share some libraries,

but almost any package do  not share any code in form of shared libaries.

So for these OSes shared and static difference almost in -fPIE/-fPIC compile options.

[Evgeniy Dushistov]

пятница, 19 января 2018 г., 19:58:05 UTC+3 пользователь Nolan Darilek написал:

Did I miss your build.rs? Couldn't find it, though I'm still a bit coffee-deprived this morning. :)

No, google groups render just thinks that 'build dot rs' is URL

 

In any case, I knocked this together:

https://gitlab.com/ndarilek/spatialite-sys

Links directly against libspatialite dynamically, with a pkg-config fallback where present. Is that basically what you did as well? I'm wondering if I need any extra munging in build.rs.

You can find my build.rs and high level cmake file here:

 https://gist.github.com/Dushistov/b924c8c31a956ad4da74d7765009ed35 

I understand there's still higher level work to do. My intent with this is to package away the spatialite linking/dependency in a separate crate so higher-level apps don't need to concern themselves with that. Even though it is only a few lines of integration work, it probably makes sense to standardize on these lower-level building blocks where possible, then build higher-level adapters to rusqlite and others as needed.

The problem is "bundle" mode (I name it so, because of such name of feature in rusqlite crate).

For me any probable `libspatialite-sys` crate is useless if it is not support static linking.

But if it is support static linking it is almost impossible to use ruqslite crate with it.

Because of you need export C symbols from C->Rust->Rust->C.

It is not clear how to do it.

[Nolan Darilek]

Thanks for this.


At the moment I'm linking dynamically just to get something going. My
Spatialite interface theoretically uses pkg-config, so if a system uses
that, then Cargo (Rust's build interface and package manager) makes it
pretty easy to switch to static linking by setting a few environment
variables. Likewise, it would be pretty easy for me to migrate to an
extension by changing the initialization mechanism in my app. All that
is to say that I'm using dynamic linking for now, but appreciate your
advice on the potential pitfalls and have a good migration path if I
need it.


For anyone interested, I just published this:


https://crates.io/crates/spatialite-sys


No docs because it's a pure API wrapper, but it plus Evgeniy's help got
me up and running. Contributions welcome. To be extra useful, it should
probably be depended on by higher-level wrappers like rusqlite, but for
the moment I'm just glad to have something that seems to work.
Collaboration welcome.