Set up tools for cross compilation with c code

[Ain]

Hi

I would like to set up go tools in a way that allows cross-compilation even when c code is used. Ie I'd like to use "github.com/mattn/go-sqlite3" package (and others which depend on c) in a project which I want to cross compile for windows and linux. Is this is possible and if yes then how?

I found this

https://www.limitlessfx.com/cross-compile-golang-app-for-windows-from-linux.html

but it is pre "go is written in go" so I now need some older go version installed before I can compile the native toolchain. But is the general approach still viable?

Any pointers on how to set up such an environment are welcome. I probably going to set up VirtualBox VM with Ubuntu 16 desktop to experiment with.

TIA

ain

[Michael Hudson-Doyle]

On 26 May 2016 at 23:09, Ain <ain.v...@gmail.com> wrote:
> Hi
>
> I would like to set up go tools in a way that allows cross-compilation even
> when c code is used. Ie I'd like to use "github.com/mattn/go-sqlite3"
> package (and others which depend on c) in a project which I want to cross
> compile for windows and linux. Is this is possible and if yes then how?

In principle you just need to point CC at the cross compiler when
building and set CGO_ENABLED=1, e.g.

CGO_ENABLED=1 CC=powerpc64le-linux-gnu-gcc GOARCH=ppc64 go build -o
go.ppc64 cmd/go

> I found this
> https://www.limitlessfx.com/cross-compile-golang-app-for-windows-from-linux.html
> but it is pre "go is written in go" so I now need some older go version
> installed before I can compile the native toolchain. But is the general
> approach still viable?

It's much much easier now, no set up steps required.

> Any pointers on how to set up such an environment are welcome. I probably
> going to set up VirtualBox VM with Ubuntu 16 desktop to experiment with.

I've only done cross-arch compiling as above, not cross-OS as you are
talking about. mingw seems to be packaged for 16.04 but I don't know
exactly which package you need. You may also have some fun installing
the development headers for sqlite in a way that works for the cross
compiler, not sure.

Cheers,
mwh

[Ain]

neljapäev, 26. mai 2016 14:15.50 UTC+3 kirjutas Michael Hudson-Doyle:

On 26 May 2016 at 23:09, Ain <ain.v...@gmail.com> wrote:
> Hi
>
> I would like to set up go tools in a way that allows cross-compilation even
> when c code is used. Ie I'd like to use "github.com/mattn/go-sqlite3"
> package (and others which depend on c) in a project which I want to cross
> compile for windows and linux. Is this is possible and if yes then how?

In principle you just need to point CC at the cross compiler when
building and set CGO_ENABLED=1, e.g.

CGO_ENABLED=1 CC=powerpc64le-linux-gnu-gcc GOARCH=ppc64 go build -o
go.ppc64 cmd/go

Thanks!

I guess I now need some tips as to what c compilers to install and how to use them with CC parameter (which OS and ARCH combo requires which version of the compiler).

 

 

I've only done cross-arch compiling as above, not cross-OS as you are
talking about. mingw seems to be packaged for 16.04 but I don't know
exactly which package you need. You may also have some fun installing
the development headers for sqlite in a way that works for the cross
compiler, not sure.

I'd expect the headers not to be problem, but the libs to bundle with binary might be...?

 

TIA

ain 

[Tamás Gulácsi]

Take a look at goxc. It's been created just for this.

[Ain]

neljapäev, 26. mai 2016 18:33.02 UTC+3 kirjutas Tamás Gulácsi:

Take a look at goxc. It's been created just for this.

But I still have to worry about installing right c compilers, right?

BTW, the installing instructions are bit confusing - it says that one should use go source install, presumably because toolchain must be built for each target platform. But with the latest go releases, this is not requirement anymore, so one could actually use binary installation of go, correct?

Anyway, meanwhile I tryed following:

$ GOOS=windows GOARC=386 CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc go build github.com/mattn/go-sqlite3/_example/simple

and this produced working win64 exe. Note that I mistakenly used GOARC=386 while it should have been GOARC=amd64. I later used GOARC=amd64 and that too resulted in working win64 exe.

Didn't have luck with win32 version:

$ GOOS=windows GOARC=386 CGO_ENABLED=1 CC=i686-w64-mingw32-gcc go build github.com/mattn/go-sqlite3/_example/simple

# runtime/cgo

cc1: sorry, unimplemented: 64-bit mode not compiled in

The gcc compiler names are really confusing and I'm not sure I actually use correct ones, especially since I'm on 64bit Ubuntu but compiler is named "mingw32" (ie not mingw64).

Is there any tutorial for the beginner as to what packages to install and what combination of CC, GOOS and GOARC to use for different (cross-compiled) target platform?

TIA

ain

[Dave Cheney]

You want GOARCH, not GOARC. The latter will be ignored and the default will match your current CPU.

Can I convince you to abandon the idea of cross compilation _with_ cgo. Cross compilation without cgo is trivial, you get it for free with every Go install, but _with_ cgo requires you to have a complete working C compiler for the target environment and a precompiled C library to link against. This might be static or dynamic, usually whichever is the least convenient.

Compare that to building in a VM, which is super simple, requires no oddball cross compilation C compilers, and is super well tested.

Please consider it, you'll have a better and less frustrating experience.

Finally, you might want to consider a paid service such as https://equinox.io/ (disclaimer, other than knowing that Alan knows his stuff, I have no experience with the service)

Thanks

Dave

[Ain]

reede, 27. mai 2016 12:57.14 UTC+3 kirjutas Dave Cheney:

You want GOARCH, not GOARC. The latter will be ignored and the default will match your current CPU.

(blush)

Fixed that and win32 version compiled too.

 

Can I convince you to abandon the idea of cross compilation _with_ cgo. Cross compilation without cgo is trivial, you get it for free with every Go install, but _with_ cgo requires you to have a complete working C compiler for the target environment and a precompiled C library to link against. This might be static or dynamic, usually whichever is the least convenient.

Compare that to building in a VM, which is super simple, requires no oddball cross compilation C compilers, and is super well tested.

Do I understand right, that you suggest me to set up win32, win64, linux32 etc VMs with go installed and then build my project on each of them, to get binary for corresponding platform?

That looks like a lot of VMs to manage... unless I only need one win and one linux setup (IOW can go build win64 exe on win32 enviromnent without cgo?).

 

Please consider it, you'll have a better and less frustrating experience.

"less frustrating experience" is exactly what I'm looking for. However I thought that managing (at least) four VMs with go and thirdparty libs is going to be frustrating, that's why I started looking for how to produce all target binaries from single setup.

 

TIA

ain

[Dave Cheney]

On Friday, 27 May 2016 21:22:39 UTC+10, Ain wrote:

reede, 27. mai 2016 12:57.14 UTC+3 kirjutas Dave Cheney:

You want GOARCH, not GOARC. The latter will be ignored and the default will match your current CPU.

(blush)

Fixed that and win32 version compiled too.

 

Can I convince you to abandon the idea of cross compilation _with_ cgo. Cross compilation without cgo is trivial, you get it for free with every Go install, but _with_ cgo requires you to have a complete working C compiler for the target environment and a precompiled C library to link against. This might be static or dynamic, usually whichever is the least convenient.

Compare that to building in a VM, which is super simple, requires no oddball cross compilation C compilers, and is super well tested.

Do I understand right, that you suggest me to set up win32, win64, linux32 etc VMs with go installed and then build my project on each of them, to get binary for corresponding platform?

That looks like a lot of VMs to manage... unless I only need one win and one linux setup (IOW can go build win64 exe on win32 enviromnent without cgo?).

Yup, that's what i'm pitching. You can probably get away with a linux64 machine to build for linux32 providing your distro gives you the right compilers and binary blobs. Maybe win64 offers a sensible win32 compile environment, I wouldn't know.

 

 

Please consider it, you'll have a better and less frustrating experience.

"less frustrating experience" is exactly what I'm looking for. However I thought that managing (at least) four VMs with go and thirdparty libs is going to be frustrating, that's why I started looking for how to produce all target binaries from single setup.

This is the bargain you make when you make your Go program depend on C code, it's no longer a simple cross platform Go program, the C dependency reduces it to the same level as cross compiling any other C program. I wrote about the tradeoffs I see in basing your product on a cgo dependency earlier this year. http://dave.cheney.net/2016/01/18/cgo-is-not-go

Thanks

Dave

[Konstantin Khomoutov]

On Fri, 27 May 2016 04:22:38 -0700 (PDT)
Ain <ain.v...@gmail.com> wrote:

[...]

> > Compare that to building in a VM, which is super simple, requires
> > no oddball cross compilation C compilers, and is super well tested.
> Do I understand right, that you suggest me to set up win32, win64,
> linux32 etc VMs with go installed and then build my project on each
> of them, to get binary for corresponding platform?
> That looks like a lot of VMs to manage... unless I only need one win
> and one linux setup (IOW can go build win64 exe on win32 enviromnent
> without cgo?).

[...]

Without cgo, Go on any platform is able to build for any other
combination of GOOS+GOARCH.

The real problem is that once you have cross-compiled, you will need to
actually test the compiled executable. Say, look at the code in [1]: it
uses a bunch of linker and compiler flags to make the code compile
and link correctly for Windows, and for Windows on GOARCH=386
specifically. So that's not an easy business and so even if you
managed to cross-compile a Windows/amd64 or Windows/x86 binary you have
no idea whether it will really pass all your tests unless you run it on
the target system.

1. https://github.com/mattn/go-sqlite3/blob/master/sqlite3_windows.go

[Ain]

reede, 27. mai 2016 14:42.47 UTC+3 kirjutas Konstantin Khomoutov:

On Fri, 27 May 2016 04:22:38 -0700 (PDT)
Ain <ain.v...@gmail.com> wrote:

[...]
> > Compare that to building in a VM, which is super simple, requires
> > no oddball cross compilation C compilers, and is super well tested.
> Do I understand right, that you suggest me to set up win32, win64,
> linux32 etc VMs with go installed and then build my project on each
> of them, to get binary for corresponding platform?
> That looks like a lot of VMs to manage... unless I only need one win
> and one linux setup (IOW can go build win64 exe on win32 enviromnent
> without cgo?).
[...]

Without cgo, Go on any platform is able to build for any other
combination of GOOS+GOARCH. 

With "Without cgo" You mean "when your code doesn't depend on c code", right?

Yes, that's really nice! Unfortunately there is still lot of useful stuff which is not available in pure go.

 

 

The real problem is that once you have cross-compiled, you will need to
actually test the compiled executable.

I think having to have VMs to test your final binary on different OS-es is a lot easier than managing whole toolchains on those OSes in order to build your project. But yes, building software for multiple target OSes is a difficult problem and go does a great job with pure go code.

ain 

[Konstantin Khomoutov]

On Fri, 27 May 2016 05:27:52 -0700 (PDT)
Ain <ain.v...@gmail.com> wrote:

[...]

> > Without cgo, Go on any platform is able to build for any other
> > combination of GOOS+GOARCH.
> With "Without cgo" You mean "when your code doesn't depend on c
> code", right?

Yes. "cgo" is a subsystem in the Go toolchain which interfaces both
worlds -- Go and C -- with each other and manages to glue your Go code
with external C libraries. That is, when you're calling out into C code
and/or use your own C code in your Go project you're using cgo.

[Lars Tørnes Hansen]

In a project I decided to let a Go program and a C program run in a separate OS process.
I used the "os/exec" package. Package "gopkg.in/pipe.v2": http://gopkg.in/pipe.v2 could also be useful.

Den fredag den 27. maj 2016 kl. 14.27.52 UTC+2 skrev Ain:

 ...
 

 Unfortunately there is still lot of useful stuff which is not available in pure go.

 

ain