CGO static linking support status

[Ciprian Dorin Craciun]

Hello all!

After skimming over the mailing list and the issues, I've -- like
some other poster in March -- concluded that: currently it is
impossible to statically link anything touching `import "C"`.

Although someone suggested that if we "embed" the C code directly
into the `.go` file it should work. Unfortunately it is not the case.
Even the simplest code:
~~~~
package main

import "C"

func main () {
}
~~~~

When built with `go build -ldflags -d ./cgo-static-3.go`, fails with:
~~~~
# command-line-arguments
.../go/pkg/tool/linux_386/8l:
$WORK/command-line-arguments.a(_cgo_import.8): cannot use dynamic
imports with -d flag
.../go/pkg/linux_386/runtime/cgo.a(_cgo_import.8): cannot use dynamic
imports with -d flag
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text): .got: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text):
pthread_attr_init: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text):
pthread_attr_getstacksize: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text):
pthread_attr_destroy: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text): .got: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text): free: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text): .got: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text):
sigfillset: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text):
sigprocmask: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text):
pthread_attr_init: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text):
pthread_attr_getstacksize: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text):
pthread_create: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text):
sigprocmask: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text): strerror:
not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text): stderr: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text): fprintf: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_linux_386.o)(.text): abort: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_util.o)(.text): .got: not defined
.../go/pkg/linux_386/runtime/cgo.a(gcc_util.o)(.text): malloc: not defined
too many errors
~~~~

Now my question is more related to the "road-map" of the static
linking feature, thus I would like to know:

A) Is it safe to assume that in an acceptable period of time such
a feature would be implemented? (An acceptable period would be about
one year or so.)

B) Which will there be any serious limitations? (Like would it
work for C++ libraries?)

Thanks all,
Ciprian.

[André Moraes]

On Mon, May 28, 2012 at 5:43 AM, Ciprian Dorin Craciun
<ciprian...@gmail.com> wrote:
>    Hello all!
>
>    After skimming over the mailing list and the issues, I've -- like
> some other poster in March -- concluded that: currently it is
> impossible to statically link anything touching `import "C"`.

There this project wich uses the golang C compilers to compile the Lua
interpreter and link that with Go,

https://github.com/akavel/goluago

I don't know if this make any difference in allowing static linking the C code.

--
André Moraes
http://amoraes.info

[minux]

On Mon, May 28, 2012 at 4:43 PM, Ciprian Dorin Craciun <ciprian...@gmail.com> wrote:

   After skimming over the mailing list and the issues, I've -- like
some other poster in March -- concluded that: currently it is
impossible to statically link anything touching `import "C"`.

static linking libc and libpthread is not possible now, but if you're willing to use tip version

of the Go tree, and one extra CL (CL 5822049), you can static link custom library, just

use this syntax in #cgo LDFLAGS:

instead of -lsomelib, use libsomelib.a (that is, spell the whole filename of the library, make

sure the library is compiled with -fPIC [note normally, static library are not compiled with

-fPIC, so you might need to recompile your custom library])

   Although someone suggested that if we "embed" the C code directly
into the `.go` file it should work. Unfortunately it is not the case.
Even the simplest code:
~~~~
package main

import "C"

func main () {
}
~~~~

   When built with `go build -ldflags -d ./cgo-static-3.go`, fails with:

-d flag to ld means disable dynamic linking.

but the current cgo implementation doesn't support static linking, if you disable dynamic linking,

you can't use cgo at all (even with the aforementioned CL).

    A) Is it safe to assume that in an acceptable period of time such
a feature would be implemented? (An acceptable period would be about
one year or so.)

it depends on issue 3591

please note it doesn't have high priority, although I'm trying, but I can't make any guarantees.

   B) Which will there be any serious limitations? (Like would it
work for C++ libraries?)

Linking C++ library using cgo is not possible right now. Again, after issue 3591 is solved, this is possible.

You could try SWIG for linking C++ libraries.

[minux]

Yes, this is another option. But, you must make sure your code is almost C89

and compilable by Plan 9 cc. Also, it can't use much external library (libc included),

as you must emulate them all in Go.

[Ciprian Dorin Craciun]

On Mon, May 28, 2012 at 4:45 PM, minux <minu...@gmail.com> wrote:
>> There this project wich uses the golang C compilers to compile the Lua
>> interpreter and link that with Go,
>>
>> https://github.com/akavel/goluago
>>
>> I don't know if this make any difference in allowing static linking the C
>> code.
>
> Yes, this is another option. But, you must make sure your code is almost C89
> and compilable by Plan 9 cc. Also, it can't use much external library (libc
> included),
> as you must emulate them all in Go.

Hmmm... This could work for me. (The library I try to statically
embed is very small and simple and uses only a few syscalls which I
could implement in ASM)...

Where would I find some documentation on how I could do this? (Or
where exactly in the `luago`, as I've not found it in a quick search?)

Thanks,
Ciprian.

P.S.: What does `CL` stand for? (I've beaten my head and I
couldn't figure it out? "Code ?something?"?)

[minux]

On Tue, May 29, 2012 at 1:20 AM, Ciprian Dorin Craciun <ciprian...@gmail.com> wrote:

On Mon, May 28, 2012 at 4:45 PM, minux <minu...@gmail.com> wrote:
>> There this project wich uses the golang C compilers to compile the Lua
>> interpreter and link that with Go,
>>
>> https://github.com/akavel/goluago
>>
>> I don't know if this make any difference in allowing static linking the C
>> code.
>
> Yes, this is another option. But, you must make sure your code is almost C89
> and compilable by Plan 9 cc. Also, it can't use much external library (libc
> included),
> as you must emulate them all in Go.

   Hmmm... This could work for me. (The library I try to statically
embed is very small and simple and uses only a few syscalls which I
could implement in ASM)...

   Where would I find some documentation on how I could do this? (Or
where exactly in the `luago`, as I've not found it in a quick search?)

My recommendation is https://github.com/akavel/gostdc

this is a standalone libc implementation suitable for this kind of projects,

from it you can learn how to interface C and Go.

Note: you might need to learn something about the Plan 9 C Compiler,

at http://plan9.bell-labs.com/sys/doc/compiler.html and

http://plan9.bell-labs.com/sys/doc/comp.html

most notably, it doesn't support full preprocessor, e.g. #if (but #ifndef/#ifdef

/#else and #endif are supported)

if you think rewrite all the preprocessing steps in your C library code is

too annoying, you can use empty header files and 'gcc -E' to preprocess them

before feed them to cc.

   P.S.: What does `CL` stand for? (I've beaten my head and I
couldn't figure it out? "Code ?something?"?)

I think it means change list, think it as source code patch, but uploaded

to http://codereview.appspot.com for peer review (you can learn about

details about this process at: golang.org/doc/contribute.html)

[peterGo]

Ciprian,

CL = ChangeList

Changelists
http://www.subversion.org.cn/svnbook/nightly/svn.advanced.changelists.html

Peter

On May 28, 1:20 pm, Ciprian Dorin Craciun <ciprian.crac...@gmail.com>
wrote:

[Ciprian Dorin Craciun]

On Mon, May 28, 2012 at 8:47 PM, peterGo <go.pe...@gmail.com> wrote:
> Ciprian,
>
> CL = ChangeList
>
> Changelists
> http://www.subversion.org.cn/svnbook/nightly/svn.advanced.changelists.html
>
> Peter

Aha! Thanks! (This could go in the "Contribute" page, as there is
only the acronym mentioned.)

[Ciprian Dorin Craciun]

Thanks for the feedback. (The library is actually the `tinycdb`
implementation which is pretty small (I only need a fraction of it). I
think I'll even rewrite it in pure Go maybe.)

I've made a simple experiment and it works:
https://github.com/cipriancraciun/volution-do/tree/5612165ec7aed7eb9857e8dff43c6ec3eed1d404/sources/experiments/c-static

Although the call semantic took me some time to track down.
(Thanks to an older email of yours "Linking in a small C object" I've
nailed it.)

Maybe this should be documented somewhere?

Ciprian.

[minux]

On Tue, May 29, 2012 at 3:47 AM, Ciprian Dorin Craciun <ciprian...@gmail.com> wrote:

   Thanks for the feedback. (The library is actually the `tinycdb`

implementation which is pretty small (I only need a fraction of it). I
think I'll even rewrite it in pure Go maybe.)

   I've made a simple experiment and it works:
     https://github.com/cipriancraciun/volution-do/tree/5612165ec7aed7eb9857e8dff43c6ec3eed1d404/sources/experiments/c-static

one correction, the #define should read:

#define FLUSH(x) USED(x)

my other post is wrong regarding this, in fact, you can see this define in pkg/runtime/runtime.h.

in case you wonder what's USED(x), it's a builtin macro that means we need x, please don't

optimize x away.

   Although the call semantic took me some time to track down.
(Thanks to an older email of yours "Linking in a small C object" I've
nailed it.)

   Maybe this should be documented somewhere?

I'm not aware of any such documentation, and I'm sure this is too low level. 

Maybe I could write an article about it when I have enough time :-)

just one note, in C, you can't access normal Go function's return value, if you need

to call Go function from C, you must write a special wrapper function in Go, for example:

package somepkg

import "math"

func mySin(in float64, out *float64) {

   *out = math.Sin(in)

}

in C, you can use:

double out;

extern void somepkg·mySin(double, double *); // don't forget this declaration!

somepkg·mySin(0.1, &out);

Another note, C could call every Go function, even those unexported and/or in another

package, and this is a good way to hide your wrapper functions from the outside (Go) world.

[Dmitry Chestnykh]

On Monday, May 28, 2012 9:47:50 PM UTC+2, ciprian wrote:

    Thanks for the feedback. (The library is actually the `tinycdb`
implementation which is pretty small (I only need a fraction of it). I
think I'll even rewrite it in pure Go maybe.)

There's a pure Go implementation of cdb:

https://github.com/jbarham/go-cdb

I haven't tried it though.

 

[Ian Lance Taylor]

Ciprian Dorin Craciun <ciprian...@gmail.com> writes:

> A) Is it safe to assume that in an acceptable period of time such
> a feature would be implemented? (An acceptable period would be about
> one year or so.)
>
> B) Which will there be any serious limitations? (Like would it
> work for C++ libraries?)

The long-term plan is to make it possible for 6l/8l/5l to generate an
object file containing all the Go code. Then the system linker can link
that object file with the C/C++ code to generate the final binary. So
the answers are A) yes; B) no.

Ian

[Ciprian Dorin Craciun]

Yay! This is great news!

Thus I can safely program in Go, and hope that by the time I
finish my project (which could even be "never") such a feature will be
implemented.

And something just hit me right now. You imply that I could just
program a part in Go (like a logic as a library) and then "embed" it
into a C application, with the main entry-point (i.e. ELF `_init`?)
not under Go's control? This would be even greater!

Thanks,
Ciprian.

[Kyle Lemons]

You can already write part of your application in C, but the go runtime still needs to be the one to set things in motion.

 

   Thanks,
   Ciprian.