Cross-linking cgo

729 views
Skip to first unread message

Rob Napier

unread,
Apr 27, 2014, 2:05:38 PM4/27/14
to golan...@googlegroups.com
I work on an appliance that includes a cross-platform Go client distributed by the appliance. I'd like to embed appliance-specific configuration into the client executable. It occurred to me that I could just ship the client source code and cross-compile it on the appliance passing -X linker flags to set various defaults. While that has worked well in basic experiments (and is insanely fast and simple to implement), the problem is that it interferes with cgo. While my app doesn't have any cgo of its own, it relies on the native network resolver and http certificate verifiers (Darwin) that require cgo. Even if I could work around those specifics, I don't want to box myself into a situation where I can never ship cgo.

(I've looked at https://code.google.com/p/go/source/detail?r=6d3bdbd27761, but I assume this would require a full C cross-compilation setup? This seems a lot of complexity, but perhaps I'm overestimating? My appliance is unix, and I need to build Mac and Windows.)

Is there an effective way to relink an already-built executable with new -X flags in a cross-compiling way? I assume that I can ship a pkg directory of all the compiled packages and 6l/8l it on demand, but is that going to work with cross-linking cgo that requires dynamic libraries that won't be on the linking system? I assume I would also need to include pkg objects for all the stdlib things I use. Is there a good way to work everything that is required? Or is there a simpler way to just change the -X values without relinking the object files?

I'm open to other approaches (like attaching configuration data to the end of the executable), but -X settings seemed such a clean solution until I ran into the cgo issues... :)

-Rob

minux

unread,
Apr 27, 2014, 3:42:27 PM4/27/14
to Rob Napier, golang-nuts
If the only cgo-dependent packages are in the standard library, then you do have workaround
without setting up a full cross compilation environment.

Download the binary distribution for the target platform, extract the pkg/$GOOS_$GOARCH directory
into your $GOROOT/pkg (your go installation must have the same version as the binary distribution,
and your go installation must support the architecture you're building), then do this to build your application:

GOOS=target_OS GOARCH=target_ARCH go build -ldflags '-X main.something "value" -linkmode internal' importpath

NOTE: 
1. make sure the timestamp of all the source files in your $GOROOT is older than any of the
*.a file in $GOROOT/pkg/$GOOS_$GOARCH that you extracted from the binary distribution,
otherwise the above command will try to rebuild the foreign packages.
2. you probably also need to copy a few generate source file from the binary distribution. all
of them are in the runtime package, and with the prefix "z".
cp -p /path/to/extracted/go/src/pkg/runtime/z*_$GOOS_$GOARCH.* $GOROOT/src/pkg/runtime

minux

unread,
Apr 30, 2014, 11:12:01 PM4/30/14
to Alan Shreve, golang-nuts, Rob Napier


On Apr 30, 2014 10:02 PM, "Alan Shreve" <al...@ngrok.com> wrote:
> I’ve automated this approach for anyone to use:
>
> https://github.com/inconshreveable/gonative
that's great! thanks for sharing.
> And it works! Almost!
>
> I’ve tested linux/darwin/windows on 386/amd64 (not freebsd or linux/arm yet) and everything seems to work great except . . .
>
> When I try to run linux_386 binaries on linux_amd64 machines. A simple hello world works, but if I try to use something that requires a native library like this:
>
> // test.go
> package main
>
> import (
>     "fmt"
>     "os/user"
> )
> func main() {
>     u, err := user.Current()
>     fmt.Printf("%v %v\n", u, err)
> }
>
> I get a really weird error:
>
> alan@inconshreveable:~$ ./test 
> -bash: ./test: No such file or directory
>
> Any ideas on what could possibly be going on? 
probably it can't find the elf interpreter.

what does "readelf -l test" output?

Alan Shreve

unread,
Apr 30, 2014, 11:26:46 PM4/30/14
to minux, golang-nuts, Rob Napier
Elf file type is EXEC (Executable file)
Entry point 0x8066a50
There are 9 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x08048034 0x08048034 0x00120 0x00120 R E 0x1000
  INTERP         0x000bed 0x08048bed 0x08048bed 0x00013 0x00013 R   0x1
      [Requesting program interpreter: /lib/ld-linux.so.2]
  LOAD           0x000000 0x08048000 0x08048000 0x68e70 0x68e70 R E 0x1000
  LOAD           0x069000 0x080b1000 0x080b1000 0xc9403 0xc9403 R   0x1000
  LOAD           0x133000 0x0817b000 0x0817b000 0x10580 0x23554 RW  0x1000
  DYNAMIC        0x133080 0x0817b080 0x0817b080 0x00098 0x00098 RW  0x4
  TLS            0x000000 0x00000000 0x00000000 0x00000 0x00008 R   0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x4
  LOOS+5041580   0x000000 0x00000000 0x00000000 0x00000 0x00000     0x4

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .text .plt .interp 
   03     .rodata .typelink .gosymtab .gopclntab .rel .gnu.version .gnu.version_r .hash .rel.plt .dynstr .dynsym .shstrtab 
   04     .got .got.plt .dynamic .noptrdata .data .bss .noptrbss 
   05     .dynamic 
   06     .tbss 
   07     
   08

minux

unread,
Apr 30, 2014, 11:39:21 PM4/30/14
to Alan Shreve, golang-nuts, Rob Napier


On Apr 30, 2014 11:26 PM, "Alan Shreve" <al...@inconshreveable.com> wrote:
> Elf file type is EXEC (Executable file)
> Entry point 0x8066a50
> There are 9 program headers, starting at offset 52
>
> Program Headers:
>   Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
>   PHDR           0x000034 0x08048034 0x08048034 0x00120 0x00120 R E 0x1000
>   INTERP         0x000bed 0x08048bed 0x08048bed 0x00013 0x00013 R   0x1
>       [Requesting program interpreter: /lib/ld-linux.so.2]

do you have this file?
if you have, then take a look at "ldd test".
the program header looks correct to me.

Alan Shreve

unread,
May 1, 2014, 12:17:56 AM5/1/14
to minux, golang-nuts, Rob Napier
You’re right, I don’t have the 32-bit ELF interpreter. Thank you! I now realize this has nothing to do with the cross-compilation approach.

This is stretching my knowledge, so my apologies if my questions don’t make sense. Why is it that binaries that are cross compiled to linux_386 *without Cgo* will run fine on a system without the ELF interpreter? What is necessary about linking with the natively-bulid 386 libraries that requires the ELF interpreter be present at runtime?

On Apr 30, 2014, at 8:39 PM, minux <minu...@gmail.com> wrote:

/lib/ld-linux.so.2

Alan Shreve

unread,
May 1, 2014, 12:29:08 AM5/1/14
to minux, golang-nuts, Rob Napier
Sorry, one last question. If I’m understanding correctly, the ELF interpreter only needs to be present for executables that are dynamically linked.

ldd test says:
“not a dynamic executable”

Although "readelf -d test” says:
Dynamic section at offset 0x133080 contains 19 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
more …

What is the distinction? Or what is good reference material to understand the distinction?

- alan
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Alan Shreve

unread,
May 1, 2014, 12:34:30 AM5/1/14
to minux, golang-nuts, Rob Napier
And just to answer my own question: it’s because I’m analyzing it with an amd64 ldd. Thanks for the help.

minux

unread,
May 1, 2014, 1:07:19 AM5/1/14
to Alan Shreve, golang-nuts, Rob Napier


On May 1, 2014 12:17 AM, "Alan Shreve" <al...@inconshreveable.com> wrote:
>
> You’re right, I don’t have the 32-bit ELF interpreter. Thank you! I now realize this has nothing to do with the cross-compilation approach.
>
> This is stretching my knowledge, so my apologies if my questions don’t make sense. Why is it that binaries that are cross compiled to linux_386 *without Cgo* will run fine on a system without the ELF interpreter? What is necessary about linking with the natively-bulid 386 libraries that requires the ELF interpreter be present at runtime?

the elf interpreter is needed for dynamic linked executables.

when not using cgo, the program will be statically linked, so they don't need the elf interpreter.

minux

unread,
May 1, 2014, 1:14:02 AM5/1/14
to Alan Shreve, golang-nuts, Rob Napier


On May 1, 2014 12:29 AM, "Alan Shreve" <al...@inconshreveable.com> wrote:
>
> Sorry, one last question. If I’m understanding correctly, the ELF interpreter only needs to be present for executables that are dynamically linked.
>
> ldd test says:
> “not a dynamic executable”

actually if you take a look at ldd's source code you will find that it's a script and just set some magic environment variable and then execute the program, essentially it's asking the elf interpreter to dump needed shared library.

so if you don't have the elf interpreter, the program will fail to execute and ldd mistake that as the program not being dynamically linked.

(that's the reason why i first ask you to verify you do have the elf interpreter and then use ldd)

PS: not all ldds work like this (because this is kinda a security hole to execute the program when you're only querying its dependencies, on certain systems, the ldd will actually read and parse the .dynamic section to get the needed shared libraries).

Alan Shreve

unread,
Apr 30, 2014, 10:02:48 PM4/30/14
to minux, Rob Napier, golang-nuts
I’ve automated this approach for anyone to use:


And it works! Almost!

I’ve tested linux/darwin/windows on 386/amd64 (not freebsd or linux/arm yet) and everything seems to work great except . . .

When I try to run linux_386 binaries on linux_amd64 machines. A simple hello world works, but if I try to use something that requires a native library like this:

// test.go
package main

import (
    "fmt"
    "os/user"
)
func main() {
    u, err := user.Current()
    fmt.Printf("%v %v\n", u, err)
}

I get a really weird error:

alan@inconshreveable:~$ ./test 
-bash: ./test: No such file or directory

Any ideas on what could possibly be going on? 

- alan

Message has been deleted
Message has been deleted

Alan Shreve

unread,
May 5, 2014, 3:06:49 PM5/5/14
to Robert, golan...@googlegroups.com
Kinda -

You don’t really want gox to rebuild the toolchain, it’s already built for you. Although it technically shouldn’t break anything if it doesn’t force a rebuild of the libraries. You also don’t need to set any env variables in your gonative call. Something like this should do the trick:

$ cd /your/project
$ gonative
$ PATH=$PWD/go/bin/:$PATH gox 

You’ll get errors for platforms that gonative doesn’t support (openbsd/freebsd/plan9), but the rest should work just fine.

- alan

On May 4, 2014, at 3:54 PM, Robert <robert...@gmail.com> wrote:

After some playing around, I'm pretty sure this does the business:

$ GOROOT=$PWD/go PATH=$PWD/go/bin:$PATH gonative
$ GOROOT=$PWD/go PATH=$PWD/go/bin:$PATH gox -build-toolchain
$ GOROOT=$PWD/go PATH=$PWD/go/bin:$PATH gox

On Sunday, May 4, 2014 2:32:42 PM UTC-7, Robert wrote:
Can you post an example of how to use gonative with gox to do an actual cross-compilation?

gonative builds the toolchain, but does not actually cross-compile...

Dave Mazzoni

unread,
Jan 9, 2017, 4:16:07 PM1/9/17
to golang-nuts, robert...@gmail.com, al...@inconshreveable.com
This looked really nice, but I'm having problems with it: 
(no problem)
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:67: cannot use "" (type string) as type bool in field value
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:67: cannot use nil as type string in field value
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:67: too few values in struct initializer
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:68: cannot use "" (type string) as type bool in field value
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:68: cannot use nil as type string in field value
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:68: too few values in struct initializer
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:69: cannot use "" (type string) as type bool in field value
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:69: cannot use nil as type string in field value
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:69: too few values in struct initializer
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:70: cannot use "" (type string) as type bool in field value
Projects/Golang/src/github.com/inconshreveable/gonative/gonative.go:70: too many errors

Any idea what's wrong?

Alan Shreve

unread,
Jan 9, 2017, 4:59:04 PM1/9/17
to Dave Mazzoni, golang-nuts, robert...@gmail.com, Alan Shreve
Ah, I should update that I don't really maintain this any longer since there are better tools these days. The best of which I've found is xgo (which supports cgo): https://github.com/karalabe/xgo
Reply all
Reply to author
Forward
0 new messages