go-sqlite3 loading extensions error

[Peter V.]

Hi go-nuts

Mattn's excellent sqlite3 wrapper library supports now loading sqite3 extensions.

I am trying to use the spatialite sqlite3 extension, which enables to use sqlite3 as a spatial database for GIS stuff.

Loading the extansion is straight forward:

_, err = db.Exec("SELECT load_extension('libspatialite')")

This works and I am able to use spatialite function in the sql queries.

The problem is when closing the database with defer db.Close() I get the following error:

*** Error in `/home/peter/.tmp/go-build467732638/command-line-arguments/_obj/exe/main': double free or corruption (!prev): 0x0000000000babc80 ***
======= Backtrace: =========
/usr/lib/libc.so.6(+0x72ecf)[0x7fccaf9c0ecf]
/usr/lib/libc.so.6(+0x7869e)[0x7fccaf9c669e]
/usr/lib/libc.so.6(+0x79377)[0x7fccaf9c7377]
/usr/lib/libsqlite3.so.0(sqlite3_free+0x6e)[0x7fccaff2e32e]
/usr/lib/libsqlite3.so.0(+0x2329c)[0x7fccaff3929c]
/usr/lib/libsqlite3.so.0(+0x2331c)[0x7fccaff3931c]
/usr/lib/libsqlite3.so.0(+0x59d36)[0x7fccaff6fd36]
/usr/lib/libsqlite3.so.0(sqlite3_finalize+0x27)[0x7fccaff6fd77]
/usr/lib/libsqlite3.so.0(+0x5a5f4)[0x7fccaff705f4]
/usr/lib/libsqlite3.so.0(+0x5a659)[0x7fccaff70659]
/usr/lib/libsqlite3.so.0(+0x21091)[0x7fccaff37091]
/usr/lib/libsqlite3.so.0(+0x58de1)[0x7fccaff6ede1]
/home/peter/.tmp/go-build467732638/command-line-arguments/_obj/exe/main(_cgo_604110260f67_Cfunc_sqlite3_close+0xc)[0x40210c]
/home/peter/.tmp/go-build467732638/command-line-arguments/_obj/exe/main[0x420b2f]
======= Memory map: ========
00400000-00547000 r-xp 00000000 00:20 11814031                           /home/peter/.tmp/go-build467732638/command-line-arguments/_obj/exe/main
00747000-0075b000 rw-p 00147000 00:20 11814031                           /home/peter/.tmp/go-build467732638/command-line-arguments/_obj/exe/main
0075b000-0076d000 rw-p 00000000 00:00 0 
00ad7000-00bbc000 rw-p 00000000 00:00 0                                  [heap]
c1ffff0000-c200100000 rw-p 00000000 00:00 0 
7fcc90000000-7fcc90021000 rw-p 00000000 00:00 0 
7fcc90021000-7fcc94000000 ---p 00000000 00:00 0 
7fcc98000000-7fcc98021000 rw-p 00000000 00:00 0 
7fcc98021000-7fcc9c000000 ---p 00000000 00:00 0 
7fcc9c2f8000-7fcc9c30d000 r-xp 00000000 08:01 2025308                    /usr/lib/libgcc_s.so.1
7fcc9c30d000-7fcc9c50d000 ---p 00015000 08:01 2025308                    /usr/lib/libgcc_s.so.1
7fcc9c50d000-7fcc9c50e000 rw-p 00015000 08:01 2025308                    /usr/lib/libgcc_s.so.1
7fcc9c50e000-7fcc9c5f4000 r-xp 00000000 08:01 1976278                    /usr/lib/libstdc++.so.6.0.18
7fcc9c5f4000-7fcc9c7f3000 ---p 000e6000 08:01 1976278                    /usr/lib/libstdc++.so.6.0.18
7fcc9c7f3000-7fcc9c7fb000 r--p 000e5000 08:01 1976278                    /usr/lib/libstdc++.so.6.0.18
7fcc9c7fb000-7fcc9c7fd000 rw-p 000ed000 08:01 1976278                    /usr/lib/libstdc++.so.6.0.18
7fcc9c7fd000-7fcc9c812000 rw-p 00000000 00:00 0 
7fcc9c812000-7fcc9c987000 r-xp 00000000 08:01 2021510                    /usr/lib/libgeos-3.3.8.so
7fcc9c987000-7fcc9cb87000 ---p 00175000 08:01 2021510                    /usr/lib/libgeos-3.3.8.so
7fcc9cb87000-7fcc9cb94000 r--p 00175000 08:01 2021510                    /usr/lib/libgeos-3.3.8.so
7fcc9cb94000-7fcc9cb98000 rw-p 00182000 08:01 2021510                    /usr/lib/libgeos-3.3.8.so
7fcc9cb98000-7fcc9cb99000 rw-p 00000000 00:00 0 
7fcc9cb99000-7fcc9cc9b000 r-xp 00000000 08:01 1973269                    /usr/lib/libm-2.18.so
7fcc9cc9b000-7fcc9ce9a000 ---p 00102000 08:01 1973269                    /usr/lib/libm-2.18.so
7fcc9ce9a000-7fcc9ce9b000 r--p 00101000 08:01 1973269                    /usr/lib/libm-2.18.so
7fcc9ce9b000-7fcc9ce9c000 rw-p 00102000 08:01 1973269                    /usr/lib/libm-2.18.so
7fcc9ce9c000-7fcc9cec0000 r-xp 00000000 08:01 2021512                    /usr/lib/libgeos_c.so.1.7.8
7fcc9cec0000-7fcc9d0bf000 ---p 00024000 08:01 2021512                    /usr/lib/libgeos_c.so.1.7.8
7fcc9d0bf000-7fcc9d0c0000 r--p 00023000 08:01 2021512                    /usr/lib/libgeos_c.so.1.7.8
7fcc9d0c0000-7fcc9d0c1000 rw-p 00024000 08:01 2021512                    /usr/lib/libgeos_c.so.1.7.8
7fcc9d0c1000-7fcc9d0d6000 r-xp 00000000 08:01 1976310                    /usr/lib/libz.so.1.2.8
7fcc9d0d6000-7fcc9d2d5000 ---p 00015000 08:01 1976310                    /usr/lib/libz.so.1.2.8
7fcc9d2d5000-7fcc9d2d6000 r--p 00014000 08:01 1976310                    /usr/lib/libz.so.1.2.8
7fcc9d2d6000-7fcc9d2d7000 rw-p 00015000 08:01 1976310                    /usr/lib/libz.so.1.2.8
7fcc9d2d7000-7fcc9d326000 r-xp 00000000 08:01 2021526                    /usr/lib/libproj.so.0.7.0
7fcc9d326000-7fcc9d526000 ---p 0004f000 08:01 2021526                    /usr/lib/libproj.so.0.7.0
7fcc9d526000-7fcc9d527000 r--p 0004f000 08:01 2021526                    /usr/lib/libproj.so.0.7.0
7fcc9d527000-7fcc9d52a000 rw-p 00050000 08:01 2021526                    /usr/lib/libproj.so.0.7.0
7fcc9d52a000-7fcc9d532000 r-xp 00000000 08:01 2021535                    /usr/lib/libfreexl.so.1.0.0
7fcc9d532000-7fcc9d731000 ---p 00008000 08:01 2021535                    /usr/lib/libfreexl.so.1.0.0
7fcc9d731000-7fcc9d732000 r--p 00007000 08:01 2021535                    /usr/lib/libfreexl.so.1.0.0
7fcc9d732000-7fcc9d733000 rw-p 00008000 08:01 2021535                    /usr/lib/libfreexl.so.1.0.0
7fcc9d733000-7fcc9db12000 r-xp 00000000 08:01 2012479                    /usr/lib/libspatialite.so.5.1.0
7fcc9db12000-7fcc9dd11000 ---p 003df000 08:01 2012479                    /usr/lib/libspatialite.so.5.1.0
7fcc9dd11000-7fcc9dd12000 r--p 003de000 08:01 2012479                    /usr/lib/libspatialite.so.5.1.0
7fcc9dd12000-7fcc9dd15000 rw-p 003df000 08:01 2012479                    /usr/lib/libspatialite.so.5.1.0
7fcc9dd4c000-7fcc9dd4d000 ---p 00000000 00:00 0 
7fcc9dd4d000-7fcc9e59d000 rw-p 00000000 00:00 0                          [stack:24291]
7fcc9e59d000-7fcc9e59e000 ---p 00000000 00:00 0 
7fcc9e59e000-7fcc9eeee000 rw-p 00000000 00:00 0 
7fcc9eeee000-7fcc9eeef000 ---p 00000000 00:00 0 
7fcc9eeef000-7fccaf74a000 rw-p 00000000 00:00 0                          [stack:24289]
7fccaf74a000-7fccaf74d000 r-xp 00000000 08:01 1973302                    /usr/lib/libdl-2.18.so
7fccaf74d000-7fccaf94c000 ---p 00003000 08:01 1973302                    /usr/lib/libdl-2.18.so
7fccaf94c000-7fccaf94d000 r--p 00002000 08:01 1973302                    /usr/lib/libdl-2.18.so
7fccaf94d000-7fccaf94e000 rw-p 00003000 08:01 1973302                    /usr/lib/libdl-2.18.so
7fccaf94e000-7fccafaef000 r-xp 00000000 08:01 1973270                    /usr/lib/libc-2.18.so
7fccafaef000-7fccafcee000 ---p 001a1000 08:01 1973270                    /usr/lib/libc-2.18.so
7fccafcee000-7fccafcf2000 r--p 001a0000 08:01 1973270                    /usr/lib/libc-2.18.so
7fccafcf2000-7fccafcf4000 rw-p 001a4000 08:01 1973270                    /usr/lib/libc-2.18.so
7fccafcf4000-7fccafcf8000 rw-p 00000000 00:00 0 
7fccafcf8000-7fccafd10000 r-xp 00000000 08:01 1973229                    /usr/lib/libpthread-2.18.so
7fccafd10000-7fccaff10000 ---p 00018000 08:01 1973229                    /usr/lib/libpthread-2.18.so
7fccaff10000-7fccaff11000 r--p 00018000 08:01 1973229                    /usr/lib/libpthread-2.18.so
7fccaff11000-7fccaff12000 rw-p 00019000 08:01 1973229                    /usr/lib/libpthread-2.18.so
7fccaff12000-7fccaff16000 rw-p 00000000 00:00 0 
7fccaff16000-7fccaffc5000 r-xp 00000000 08:01 1995992                    /usr/lib/libsqlite3.so.0.8.6
7fccaffc5000-7fccb01c5000 ---p 000af000 08:01 1995992                    /usr/lib/libsqlite3.so.0.8.6
7fccb01c5000-7fccb01c7000 r--p 000af000 08:01 1995992                    /usr/lib/libsqlite3.so.0.8.6
7fccb01c7000-7fccb01ca000 rw-p 000b1000 08:01 1995992                    /usr/lib/libsqlite3.so.0.8.6
7fccb01ca000-7fccb01ea000 r-xp 00000000 08:01 1973253                    /usr/lib/ld-2.18.so
7fccb01fd000-7fccb03b2000 rw-p 00000000 00:00 0                          [stack:24290]
7fccb03b7000-7fccb03e9000 rw-p 00000000 00:00 0 
7fccb03e9000-7fccb03ea000 r--p 0001f000 08:01 1973253                    /usr/lib/ld-2.18.so
7fccb03ea000-7fccb03eb000 rw-p 00020000 08:01 1973253                    /usr/lib/ld-2.18.so
7fccb03eb000-7fccb03ec000 rw-p 00000000 00:00 0 
7fff98a31000-7fff98a52000 rw-p 00000000 00:00 0                          [stack]
7fff98b51000-7fff98b53000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
SIGABRT: abort
PC=0x7fccaf9833d9
signal arrived during cgo execution

github.com/ptrv/go-sqlite3._Cfunc_sqlite3_close(0xad71c8, 0x0)
    github.com/ptrv/go-sqlite3/_obj/_cgo_defun.c:152 +0x2f
github.com/ptrv/go-sqlite3.(*SQLiteConn).Close(0xc200000020, 0x0, 0x7fccb035ddb8)
    github.com/ptrv/go-sqlite3/_obj/sqlite3.cgo1.go:171 +0x7f
database/sql.(*driverConn).finalClose(0xc20006d060, 0xc200058300, 0xc20003b230)
    /usr/local/go/src/pkg/database/sql/sql.go:289 +0xcb
database/sql.func·002(0xc20006d000, 0xc2000691b0)
    /usr/local/go/src/pkg/database/sql/sql.go:372 +0x2c
database/sql.(*driverConn).closeDBLocked(0xc20006d060, 0x412974, 0x4f3f70)
    /usr/local/go/src/pkg/database/sql/sql.go:261 +0x133
database/sql.(*DB).Close(0xc20006d000, 0x0, 0x0)
    /usr/local/go/src/pkg/database/sql/sql.go:421 +0xb3
main.main()
    /home/peter/gocode/src/github.com/ptrv/go-spatialite/spatialite_example/main.go:39 +0xf1

goroutine 2 [syscall]:
rax     0x0
rbx     0x92
rcx     0xffffffffffffffff
rdx     0x6
rdi     0x5ee0
rsi     0x5ee0
rbp     0x7fff98a4ff70
rsp     0x7fff98a4fbd8
r8      0x0
r9      0x401fe4
r10     0x8
r11     0x202
r12     0x7fff98a4fd80
r13     0x7
r14     0x92
r15     0x7
rip     0x7fccaf9833d9
rflags  0x202
cs      0x33
fs      0x0
gs      0x0
exit status 2

When I don't close the db I do not get the message.

Does anyone has an idea what the problem might be?

Thanks in advance,

Peter

[Carlos Castillo]

Code?

You've given one line, which looking at mattn/go-sqlite is probably already wrong (it only affects a single connection in a pool of connections). Although this won't cause the problem you are seeing, there's nothing else to work with.

The change to mattn/go-sqlite3 is pretty young, so there could be a bug there. You also don't even seem to be using mattn's pkg (according to the posted trace you are using ptrv/go-sqlite3), and I can't see the code of packages that you do use.

I have not had any problems loading the libspatialite library using mattn's sqlite3. Attached is a sample program that works with the latest mattn/go-sqlite3 and libspatialite-4.1.1 on darwin/amd64. Caveat: It's not an extensive test, and I have no idea of how your code attempts to use these libraries.

[Peter V.]

Hi Carlos,

thanks for the reply and your example code.

On Sunday, August 25, 2013 12:34:07 PM UTC+2, Carlos Castillo wrote:

Code?

You've given one line, which looking at mattn/go-sqlite is probably already wrong (it only affects a single connection in a pool of connections). Although this won't cause the problem you are seeing, there's nothing else to work with.

Here is my example code: https://gist.github.com/ptrv/6335248
The program crashes if the the queries include "SELECT
CreateSpatialIndex('testtable', 'geom');"
When I comment the line and running the program without the spatial

index it does not crash. 

 

The change to mattn/go-sqlite3 is pretty young, so there could be a bug there. You also don't even seem to be using mattn's pkg (according to the posted trace you are using ptrv/go-sqlite3), and I can't see the code of packages that you do use.

ptrv/go-sqlite3 was my temporary fork before mattn implemented the
feature to load extensions, which loaded the extension also only on
open and not in the hook.

 

I have not had any problems loading the libspatialite library using mattn's sqlite3. Attached is a sample program that works with the latest mattn/go-sqlite3 and libspatialite-4.1.1 on darwin/amd64. Caveat: It's not an extensive test, and I have no idea of how your code attempts to use these libraries.

I work now also with the latest mattn/go-sqlite3 version and
libspatialite 4.1.1 on linux/amd64 and I get the crash when I use the

database with spatial index.

Peter

[Carlos Castillo]

I've tried several things, and haven't come up with a satisfactory solution or answer to your question.

1. On my machine, your code either:
1. Crashes with a SIGSEGV during the final database close (low chance)
2. Hangs forever during the final database close
2. Rewriting the code to use the Raw db driver (instead of using database/sql) makes no difference (so it's not db/sql's fault)
3. Rewriting the code to use the code.google.com/p/go-sqlite driver doesn't work (no extension loading)
1. Trying to modify the driver to support extension loading failed... probably because it's compiling it's own version of the driver
2. The modified version SIGSEGVs during extension load

I don't see anything egregiously wrong with the code, although you may want to check the return value of stmt.Close() instead of just defering it (and ignoring the error).

 

It seems odd that removing the 1 command (CreateSpatialIndex) fixes the problems, It must be doing something that doesn't jive well with how mattn/go-sqlite3 operates, since I was able to write a C app that works (attached). I think that CreateSpatialIndex does a bunch of VirtualTable work, which might be where the go-driver is hiccuping.

[Maxim Khitrov]

On Mon, Aug 26, 2013 at 6:46 PM, Carlos Castillo <cook...@gmail.com> wrote:
> Rewriting the code to use the code.google.com/p/go-sqlite driver doesn't
> work (no extension loading)
>
> Trying to modify the driver to support extension loading failed... probably
> because it's compiling it's own version of the driver
> The modified version SIGSEGVs during extension load

I take it that you tried commenting out "#cgo CFLAGS:
-DSQLITE_OMIT_LOAD_EXTENSION=1" in sqlite3.go and it still didn't
work? I don't think there is any technical reason why my package
wouldn't work with extensions, I just never tested this feature and
decided to leave it disabled until someone asked for it.

I'm going to update SQLite to 3.8.0 later this week, perhaps I'll take
a look at extension loading as well. If someone manages to get it
working before then, please send me a patch.

[Carlos Castillo]

Yes, I disabled the OMIT_EXTENSION directive, but found in gdb that when the libspatialite library was loaded it was using the external sqlite3 shared-lib functions, not the code of the local static compiled version, and thus was trying to insert/find functions in the wrong place.

[Carlos Castillo]

As it turns out, the modifications I made to your library does work for regular straightforward extensions. By using the info at http://www.sqlite.org/loadext.html, I was able to build a module that could be loaded by code.google.com/p/go-sqlite. I have attached the code for the example. The modifications also work with the sample extension http://www.sqlite.org/contrib/download//download/extension-functions.c?get=25

I'm not sure how you'd like patches, so I've attached the result of hg diff. I basically added a LoadExtension method to sqlite3.Conn (if you use the package directly), and added an Extensions field to sqlite3.Driver (if you want to use database/sql instead), mirroring the behaviour I helped add to mattn/go-sqlite3.

On Monday, August 26, 2013 4:02:07 PM UTC-7, Maxim Khitrov wrote:

[Carlos Castillo]

Unfortunately, libspatialite seems to link to and directly call the functions in the sqlite dll, as opposed to using the extension API, so it bugs out with the static-linked version of SQLite3 used in code.google.com/p/go-sqlite. At least this is the case with the homebrew version of libspatialite on darwin.

[Carlos Castillo]

Further developments: By modifying the building of my C program (see above) to be linked to sqlite3 statically instead of via the dylib I get a similar crash when trying to load libspatialite (Segmentation fault: 11). Therefore extension loading is in general possible with a static build of sqlite, but libspatialite doesn't work because of it's not using the sqlite3 extension API correctly.

[Peter Vasil]

Hi Carlos,

thank you for your extensive testing and for making clear what causes
the crash when using libspatialite as extension. This should be
reported probably on the spatialite mailing list.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[dan...@skrach.at]

Hello,

I just came across this thread because I also ran into problems using spatialite using mattn's go-slite3 library. Is there any solution which would allow me to use spatialite from within go? (E.g. by using a amalgamated version of spatialite?)

Thank you for your help!

[Daniel Skrach]

Update: I found a solution (which at least works on MacOS X 10.9).

Steps:

Install the HEAD version of spatialite (remove a possible previous version first):

brew install --HEAD libspatialite

patch sqlite3.c file in mattn/go-sqlite3/ to enable RTREE Support:

+#define SQLITE_ENABLE_RTREE 1

Install the new version with go install

Change the test go extension loading from

Extensions: []string{"libspatialite"},

to

Extensions: []string{"mod_spatialite"}

I hope this information helps someone.

[Jan Weitz]

Spatialite as of version 4.2 has 2 build targets:

libspatialite (uses sqlite3 directly. Use this for stand-alone apps.)

mod_spatialite (uses sqlite3 extension api)

If you want to use it as an extension, you have to use mod_spatialite.

If you have an older version than 4.2 you might be able to load libspatialite as an extension.

See: https://www.gaia-gis.it/fossil/libspatialite/wiki?name=mod_spatialite

[Jan Weitz]

Now if someone can point me to an Android/ARM version of mod_spatialite, I would be happy.

[Jan Weitz]

Finally, a working extension:

https://github.com/weitzj/libspatialite/tree/master/libs

Be aware how to load this:

- When doing an ndk-build, it is only possible to name the files as `libMODULENAME.so`

- But the default entrypoint of the mod_sqlite extension is called `sqlite3_modspatialite_init`

- Sqlite3 load_extension() method tries to --guess-- the entrypoint based on the library filename (https://www.sqlite.org/loadext.html)

So in this case it tries to load `sqlite3_spatialite_init`, which will fail.

Solution:

You have to --override-- the entrypoint when loading the extension. Therefore you have to use the 2-argument-method:

SELECT load_extension('FILENAME', 'ENTRYPOINT')

This is:

SELECT load_extension('libspatialite.so','sqlite3_modspatialite_init')

I am not sure, yet, if mattn supports the overriding of the entrypoint.