static sqlite on linux & windows

[Brad Fitzpatrick]

Any Windows and/or SQLite and/or linker experts here?

I've been working with a friend trying to get SQLite to compile statically with Go (using cgo, 6g, and 8g) on both Linux and Windows.

The basic problem (as I vaguely understand it) is that the 6l/8l linkers don't support linker scripts, and glibc uses them, so we have to avoid using SQLite's os_unix vfs, as it assumes it can use libc.

Instead, we implemented our own SQLite vfs that calls back into Go using cgo and we implement the SQLite vfs using portable Go APIs.

So far so good.  It now runs on Linux:

    http://code.google.com/p/camlistore/source/browse/#git%2Fmisc%2Fsqlite

    $ git clone https://code.google.com/p/camlistore/ camlistore

    $ cd camlistore/misc/sqlite

    $ gotest -test.v

(we haven't finished implementing all the vfs calls, but nothing hard remains)

Windows confuses me though.  I've banged my head against the wall and searched the Googles as much as I could, and I'm down to one linker error:

rm -rf *.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo* *.cgo[12].* test.out build.out

CGOPKGPATH=camdev cgo --  vfs.go sqlite.go 

touch _obj/_cgo_run

8g  -o _go_.8  _obj/vfs.cgo1.go _obj/sqlite.cgo1.go _obj/_cgo_gotypes.go

8c -FVw -I/c/go//pkg/windows_386 -I . -o "_cgo_defun.8" _obj/_cgo_defun.c

gcc -m32 -I . -g -fPIC -O2 -o _cgo_main.o -c  -D_GNU_SOURCE _obj/_cgo_main.c

_obj/_cgo_main.c:1:0: warning: -fPIC ignored for target (all code is position independent)

gcc -m32 -g -fPIC -O2 -o sqlite3_obj.o -c  -D_GNU_SOURCE sqlite3_obj.c

sqlite3_obj.c:1:0: warning: -fPIC ignored for target (all code is position independent)

gcc -m32 -g -fPIC -O2 -o vfs.o -c  -D_GNU_SOURCE vfs.c

vfs.c:1:0: warning: -fPIC ignored for target (all code is position independent)

vfs.c: In function 'go_file_write':

vfs.c:31:3: warning: passing argument 2 of 'GoFileWrite' discards qualifiers from pointer target type

_cgo_export.h:21:12: note: expected 'schar *' but argument is of type 'const void *'

vfs.c: In function 'go_vfs_delete':

vfs.c:120:3: warning: passing argument 1 of 'GoVFSDelete' discards qualifiers from pointer target type

_cgo_export.h:33:12: note: expected 'schar *' but argument is of type 'const char *'

vfs.c: In function 'go_vfs_access':

vfs.c:127:3: warning: passing argument 1 of 'GoVFSAccess' discards qualifiers from pointer target type

_cgo_export.h:35:12: note: expected 'schar *' but argument is of type 'const char *'

vfs.c: In function 'go_init_vfs':

vfs.c:203:14: warning: assignment from incompatible pointer type

gcc -m32 -I . -g -fPIC -O2 -o vfs.cgo2.o -c  -D_GNU_SOURCE _obj/vfs.cgo2.c

_obj/vfs.cgo2.c:1:0: warning: -fPIC ignored for target (all code is position independent)

gcc -m32 -I . -g -fPIC -O2 -o sqlite.cgo2.o -c  -D_GNU_SOURCE _obj/sqlite.cgo2.c

_obj/sqlite.cgo2.c:1:0: warning: -fPIC ignored for target (all code is position independent)

sqlite.go: In function '_cgo_7802683b6eaa_Cfunc_sqlite3_prepare_v2':

sqlite.go:79:2: warning: passing argument 5 of 'sqlite3_prepare_v2' from incompatible pointer type

./sqlite3.h:2842:16: note: expected 'const char **' but argument is of type 'char **'

gcc -m32 -I . -g -fPIC -O2 -o _cgo_export.o -c  -D_GNU_SOURCE _obj/_cgo_export.c

_obj/_cgo_export.c:1:0: warning: -fPIC ignored for target (all code is position independent)

gcc -m32 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o sqlite3_obj.o vfs.o vfs.cgo2.o sqlite.cgo2.o _cgo_export.o  -lpthread -lgcc_s -lmingwex -lmsvcrt

cgo -dynimport _cgo1_.o >_obj/_cgo_import.c_ && mv -f _obj/_cgo_import.c_ _obj/_cgo_import.c

8c -FVw -I . -o "_cgo_import.8" _obj/_cgo_import.c

rm -f _obj/camdev/sqlite.a

gopack grc _obj/camdev/sqlite.a _go_.8  _cgo_defun.8 _cgo_import.8 sqlite3_obj.o vfs.o  vfs.cgo2.o sqlite.cgo2.o _cgo_export.o

rm -f _test/camdev/sqlite.a

8g  -o _gotest_.8  _obj/vfs.cgo1.go _obj/sqlite.cgo1.go _obj/_cgo_gotypes.go sqlite_test.go

rm -f _test/camdev/sqlite.a

gopack grc _test/camdev/sqlite.a _gotest_.8  _cgo_defun.8 _cgo_import.8 sqlite3_obj.o vfs.o  vfs.cgo2.o sqlite.cgo2.o _cgo_export.o

_test/camdev/sqlite.a(sqlite3_obj.o)(.text): winSysInfo: not defined

_test/camdev/sqlite.a(sqlite3_obj.o)(.text): winSysInfo: not defined

_test/camdev/sqlite.a(sqlite3_obj.o)(.text): winSysInfo: not defined

winSysInfo(0): not defined

gotest: "c:\\go\\bin\\\\8l.exe -L _test _testmain.8" failed: exit status 1

In the sqlite3.c amalgamation, I see winSysInfo, though:

SYSTEM_INFO winSysInfo;

And it's being compiled.  (it's not #ifdef'd out).  And <windows.h> is included, which includes <winbase.h> where SYSTEM_INFO is defined.

Why is this this not linking?

I've solved the previous dozen linker errors on Windows by adding new -lfoo LDFLAGS, now up to:

     #cgo windows LDFLAGS: -lpthread -lgcc_s -lmingwex -lmsvcrt                                            

I guess I'm missing one for Windows?  But the symbol I need is winSysInfo in the project's own .c file, so I'm not sure why I'd have to link in something else to get it.

Any pointers would be much loved.

Thanks!

[Ian Lance Taylor]

Brad Fitzpatrick <brad...@golang.org> writes:

> The basic problem (as I vaguely understand it) is that the 6l/8l linkers
> don't support linker scripts, and glibc uses them, so we have to avoid using
> SQLite's os_unix vfs, as it assumes it can use libc.

Note that glibc only uses linker scripts when linking dynamically (you
can see the linker script in question if you look at /usr/lib/libc.so).
Glibc does not use linker scripts when linking statically. While there
may well be a problem with glibc, the problem is not linker scripts.

(I'm not going to try to tackle your question about Windows, though.)

Ian

[brainman]

Hey. I know nothing about gcc but if I remove your line of

#cgo windows LDFLAGS: -lpthread -lgcc_s -lmingwex -lmsvcrt

I could build your package

make clean
make all

The test

make test

fails:

gotest

rm -f _test/camdev/sqlite.a
8g  -o _gotest_.8  _obj/vfs.cgo1.go _obj/sqlite.cgo1.go _obj/_cgo_gotypes.go sqlite_test.go
rm -f _test/camdev/sqlite.a
gopack grc _test/camdev/sqlite.a _gotest_.8  _cgo_defun.8 _cgo_import.8 sqlite3_obj.o vfs.o  vfs.cgo2.o sqlite.cgo2.o _cgo_export.o

_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __moddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __moddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __moddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __moddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __umoddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __udivdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __umoddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __udivdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __umoddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __moddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
too many errors
gotest: "c:\\MinGW\\go\\bin\\8l.exe -L _test _testmain.8" failed: exit status 1
make: *** [test] Error 2

These are all using mingw directory blob that we all use to build go.

On the other hand, if winSysInfo is your only problem, then you should be able to get rid of it all together. It calls:

GetSystemInfo(&winSysInfo) // http://msdn.microsoft.com/en-us/library/ms724381%28v=vs.85%29.aspx

and then uses winSysInfo.dwAllocationGranularity // http://msdn.microsoft.com/en-us/library/ms724958%28v=vs.85%29.aspx

I suspect winSysInfo.dwAllocationGranularity is always 4k. You could write small go program to call GetSystemInfo to find out for sure.

Alex

[Brad Fitzpatrick]

On Thu, Jul 21, 2011 at 5:09 PM, brainman <alex.b...@gmail.com> wrote:

Hey. I know nothing about gcc but if I remove your line of

#cgo windows LDFLAGS: -lpthread -lgcc_s -lmingwex -lmsvcrt

I could build your package

make clean
make all

The test

make test

fails:

gotest

rm -f _test/camdev/sqlite.a
8g  -o _gotest_.8  _obj/vfs.cgo1.go _obj/sqlite.cgo1.go _obj/_cgo_gotypes.go sqlite_test.go
rm -f _test/camdev/sqlite.a
gopack grc _test/camdev/sqlite.a _gotest_.8  _cgo_defun.8 _cgo_import.8 sqlite3_obj.o vfs.o  vfs.cgo2.o sqlite.cgo2.o _cgo_export.o

_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __moddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __moddi3: not defined
_test/camdev/sqlite.a(sqlite3_obj.o)(.text): __divdi3: not defined

Those are in -lgcc_s.

I'll look into removing winSysInfo usage altogether.  Good tip!

[Brad Fitzpatrick]

Oh, I should mention:  just running "make" is never the problem.  It's when I run "gotest" (which does the final link) that I hit all the problems.

On Thu, Jul 21, 2011 at 5:09 PM, brainman <alex.b...@gmail.com> wrote:

[mattn]

try to add "static".

static SYSTEM_INFO winSysInfo;

[brainman]

I lied. It is 64k, not 4k.

package main

import (
    "fmt"
    "log"
    "syscall"
    "unsafe"
)

type SYSTEM_INFO struct {
    OemId                     uint32
    PageSize                  uint32
    MinimumApplicationAddress uintptr
    MaximumApplicationAddress uintptr
    ActiveProcessorMask       *uint32
    NumberOfProcessors        uint32
    ProcessorType             uint32
    AllocationGranularity     uint32
    ProcessorLevel            uint16
    ProcessorRevision         uint16
}

func main() {
    h, err := syscall.LoadLibrary("kernel32.dll")
    if err != 0 {
        log.Fatalf("LoadLibrary: %s", syscall.Errstr(err))
    }
    defer syscall.FreeLibrary(h)
    proc, err := syscall.GetProcAddress(h, "GetSystemInfo")
    if err != 0 {
        log.Fatalf("GetProcAddress: %s", syscall.Errstr(err))
    }
    var si SYSTEM_INFO
    syscall.Syscall(uintptr(proc), 1, uintptr(unsafe.Pointer(&si)), 0, 0)
    fmt.Printf("SYSTEM_INFO=%+v\n", si)
}

prints

SYSTEM_INFO={OemId:0 PageSize:4096 MinimumApplicationAddress:65536 MaximumApplic
ationAddress:2147418111 ActiveProcessorMask:0x1 NumberOfProcessors:1 ProcessorTy
pe:586 AllocationGranularity:65536 ProcessorLevel:15 ProcessorRevision:258}

on my computer.

Alex

[Wei guangjing]

2011/7/22 mattn <matt...@gmail.com>:

> try to add "static".
> static SYSTEM_INFO winSysInfo;

yes, static works. ldpe has bug link global values, I fixed it, please
apply < http://codereview.appspot.com/4794052> and try again.

--
Wei guangjing

[Brad Fitzpatrick]

On Thu, Jul 21, 2011 at 10:59 PM, Wei guangjing <vcc...@gmail.com> wrote:

2011/7/22 mattn <mattn.jp@gmail.com>:

> try to add "static".
> static SYSTEM_INFO winSysInfo;

yes, static works. ldpe has bug link global values, I fixed it, please
apply < http://codereview.appspot.com/4794052> and try again.

Whoa, thanks!  :-)

I look forward to trying this!  Do the tests pass now?  (my Windows machine at home is off, and my Amazon EC2 instance doesn't have enough stuff on it...)

[Wei guangjing]

2011/7/22 Brad Fitzpatrick <brad...@golang.org>:

> On Thu, Jul 21, 2011 at 10:59 PM, Wei guangjing <vcc...@gmail.com> wrote:
>>

>> 2011/7/22 mattn <matt...@gmail.com>:

>> > try to add "static".
>> > static SYSTEM_INFO winSysInfo;
>>
>> yes, static works. ldpe has bug link global values, I fixed it, please
>> apply < http://codereview.appspot.com/4794052> and try again.
>
> Whoa, thanks!  :-)
>
> I look forward to trying this!  Do the tests pass now?  (my Windows machine
> at home is off, and my Amazon EC2 instance doesn't have enough stuff on
> it...)

test works now, need apply CL <http://codereview.appspot.com/4794056>,
fixes another bug in ldpe.

--
Wei guangjing

[minux]

On Sat, Jun 30, 2012 at 5:30 PM, BeMasher <bema...@gmail.com> wrote:

I've been trying to build sqlite statically with mattn's go-sqlite3 and this post helped a lot with resolving linker issues, but I've got one last undefined reference that I haven't been able to find any information about.

This is being built on go1.0.2 with. On compiling the package all goes well, but compiling a program using the package produces:

# command-line-arguments

C:\fakepath\Source\GoLang\pkg\windows_386/github.com/mattn/go-sqlite3.a(sqlite3.o)(.text): __chkstk_ms: not defined

__chkstk_ms(0): not defined

try adding "-lmingwex -lmingw32" to cgo LDFLAGS.

I've been looking for a solution for a while now and haven't found anything about it. Any help is appreciated.

imo, a more general way would be to use the bundled cc to compile sqlite3.c so that cgo is not involved.

I've tried that, and found a way to persuade 5c/6c/8c to accept sqlite3.c but I still lacked a sqlite3 vfs implementation,

now that I found Brad's implementation in http://code.google.com/p/camlistore/source/browse/misc/#misc%2Fsqlite

I might be able to complete a real static sqlite3 binding without using cgo.

[Jingcheng Zhang]

Hello minux,

Any progress on this work? I'm looking for a way to link SQLite3
statically into go package too.

Thanks

--
Best regards,
Jingcheng Zhang
Beijing, P.R.China

[BeMasher]

I finally got it working on windows with the 1.0.2 amd64 package of go and tdm-gcc. I got copies of the sqlite amalgamation source and put those into the same directory as the package: $GOPATH/src/github.com/mattn/go-sqlite3 and changed the LDFLAGs and CFLAGS in sqlite3.go to:

#cgo windows LDFLAGS: -lmingwex -lmingw32

#cgo windows CFLAGS: -fno-stack-check -fno-stack-protector -mno-stack-arg-probe

The CFLAGS remove the warning about ___chkstk_ms being undefined. Once that's done and the package compiles, everything works flawlessly.