SQLite3 Support without CGo

[s...@infitialis.com]

Hello,

I love the pure-Go compiler & tool-chain, especially how effortless it makes cross-compiling from our Linux development server for our estate of Windows PCs.

It even made me feel like it was worth the effort to create a DLL wrapper for SQLite3, so we can use it without CGo by bundling the pre-build DLL.

Plus I already have to bundle several other DLLs anyway, so this deployment method wasn't as much of an issue as the build process.

I've just put an early test version online here: https://github.com/iamacarpet/go-sqlite3-win64

Ideally I'd like to tweak it to worth with 32bit Windows too (should be fairly easy), plus Linux (which seems a bit more difficult).

From what I've seen, dlopen is only supported via CGo (e.g. https://github.com/rainycape/dl).

Does anyone have advice on calling C shared libraries on Linux using pure-Go, like we can call DLLs on Windows?

Regards,

Samuel Melrose.

[Konstantin Khomoutov]

AFAIK, dlopen() requires cgo simply due to the fact it's not just a
thin wrapper around some syscall (like, say, open() is) but it's
actually implemented in libc. So I think what you can do is to
start with [1] and chase the code paths from there.

Note that surely other libc implementations exist, such as muzzle and
ulibc, and they could have simpler implementations. (Maybe, I'm not
sure.)

1. https://github.com/lattera/glibc/blob/master/dlfcn/dlopen.c

[s...@infitialis.com]

Thanks Konstantin,

That was very helpful.

I'll do some more looking into it, as taking your suggestion, the ulibc version (here) seems more simple than glibc.

I'm not going to kid myself though, it looks well beyond my current capabilities.

Is there anyone else with an interest in this kind of functionality (dlopen/dlsym to call C shared libs on Linux and other *nix) from pure Go, or at least, without CGo?

If so, maybe we could get someone from the main Go team to put it on the agenda?

I've heard the idea spoken of before and the support for it in Windows (DLLs) is fantastic!

Although I understand that's because most of the syscalls require calls to core DLLs, unlike Linux it seems.

Regards,

Samuel Melrose

[brainman]

I was toying with this idea myself.

Maybe even going all the way with "pure" Go version - with something similar to https://github.com/nkbai/go-memorydll :-)

Alex

[s...@infitialis.com]

If anyone is interested, I've opened a ticket on GitHub (https://github.com/golang/go/issues/18296).

Please drop by and show your support.

[sup...@lieferpool.de]

If you want sqlite3 support without cgo, you might check out

https://github.com/cvilsmeier/sqinn-go

[Jan Mercl]

On Sat, Jan 23, 2021 at 7:02 PM sup...@lieferpool.de
<sup...@lieferpool.de> wrote:

> If you want sqlite3 support without cgo, you might check out
> https://github.com/cvilsmeier/sqinn-go

Shameless plug: https://pkg.go.dev/modernc.org/sqlite

[ben...@gmail.com]

Jan, is there any write-up about modernc.org/sqlite? I've dabbled in automated conversion, and I'm very curious if there's more information about how it works, any pitfalls / unsupported stuff, etc, but having trouble finding anything like that in the repos.

[Jan Mercl]

On Tue, Feb 16, 2021 at 10:52 PM ben...@gmail.com <ben...@gmail.com> wrote:

> Jan, is there any write-up about modernc.org/sqlite? I've dabbled in automated conversion, and I'm very curious if there's more information about how it works, any pitfalls / unsupported stuff, etc, but having trouble finding anything like that in the repos.

The transpiller:
C Front end: https://pkg.go.dev/modernc.org/cc/v3
Go back end: https://pkg.go.dev/modernc.org/ccgo/v3

How it works:
The front end produces type annotated ASTs for the translation units,
the back end links the ASTs and renders them as Go code.

Pitfalls:
The linker is memory only and memory hungry. ccgo has not yet any
object file format for separating the compilation and linking phases.

Unsupported:
Most gcc extensions. Many C99-specific things. They are being added as
needed when I run into a blocking issue with some C -> Go project.
Recently for example: https://gitlab.com/cznic/tk

Feel free to ask more specific questions. Perhaps off-list or at the
issue tracker(s). There are also Slack channels #sqlite and
#modernc_cc at gophers.slack.com.

[ben...@gmail.com]

Thanks a lot for that -- I appreciate the links.