[ANN] xgo: my take on cross compiling... CGO enabled ;)

[Péter Szilágyi]

Hi all,

  I've been a bit frustrated with cross compiling my project to various platforms, since it contained a few short, but essential code snippets in C (getting the CPU load on win and osx). And, since I'm way too lazy to maintain 4 separate dev environments, I've just wasted 2 days of my life not to :D

  I'd like to introduce xgo, my take on cross compiling CGO enabled Go programs to various platforms (Linux, Windows, OS X) and architectures (386, amd64, arm). It is based on lightweight Linux containers, specifically a Docker container containing all the necessary tooling and compilers to do their magic.

The result?

  $ docker pull karalabe/xgo

  $ go get github.com/karalabe/xgo

And voila:

  $ xgo github.com/project-iris/iris

  ...

  $ ls -al

  -rwxr-xr-x  1 root     root  3086860 Aug  7 10:01 iris-darwin-386

  -rwxr-xr-x  1 root     root  3941068 Aug  7 10:01 iris-darwin-amd64

  -rwxr-xr-x  1 root     root  4185144 Aug  7 10:01 iris-linux-386

  -rwxr-xr-x  1 root     root  5196784 Aug  7 10:01 iris-linux-amd64

  -rwxr-xr-x  1 root     root  4151688 Aug  7 10:01 iris-linux-arm

  -rwxr-xr-x  1 root     root  4228608 Aug  7 10:01 iris-windows-386.exe

  -rwxr-xr-x  1 root     root  5243904 Aug  7 10:01 iris-windows-amd64.exe

  I'm sure there are rough edges, open ended questions and possibilities to optimize, but it does what I needed from it, and most importantly, does it trivially. Hope you'll find it useful too :)

Cheers,

  Peter

PS: Yes, Iris does contain C: here and here.

[Ian Davis]

 

On Thu, Aug 7, 2014, at 08:39 AM, Péter Szilágyi wrote:

 

  I'd like to introduce xgo, my take on cross compiling CGO enabled Go programs to various platforms (Linux, Windows, OS X) and architectures (386, amd64, arm). It is based on lightweight Linux containers, specifically a Docker container containing all the necessary tooling and compilers to do their magic.

 

 

This looks like something that's going to make my life a lot easier. It's not explained in the documentation how you would inject c headers and libraries for dependent packages (e.g. for SDL) - is that easy to do?

 

Ian

[Péter Szilágyi]

Hey Ian,

  I've only focused in getting the tool-chains and cross compilers operational. Injecting C dependencies is for the moment beyond what I need so I don't think I personally will have the capacity to do it. One possible and easy-ish way I see doing this is to create a docker container based on xgo (i.e. additional layers on top) with all the required dependencies added (or maybe some mechanism to load dependencies from a mounted volume).

  C library injection is something that would definitely be valuable to investigate, but I think some experimentation is in order before a solution will arise. But since I don't have C dependencies beyond the system libraries, I'm not the best person to do it. I will however gladly accept pull requests :D

Cheers,

  Peter

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Péter Szilágyi]

Hey hey,

  Yes, actually it was - among others - Alan's gonative package (and the mailing list discussion with minux) that made me think about taking it a step further and enabling cgo for external dependencies too and not just the system libs. I've discussed Dave's golang-crosscompile and Alan's gonative - and their shortcomings - in the xgo readme :)

Cheers,

  Peter

On Thu, Aug 7, 2014 at 12:47 PM, Dobrosław Żybort <matr...@gmail.com> wrote:

Did you check https://github.com/inconshreveable/gonative earlier?

--

[Am Laher]

Péter, this is a great development. 

I've been wondering how to achieve this in 'goxc', and I think your Docker approach is a beautiful solution.

For now I'll just to point people towards xgo, but in the [distant] future I think I'd like to borrow/steal your idea, if you don't mind.

I was thinking Docker could download a Go tool (in my case another copy of goxc), which could then package up the executables and distribute them as necessary.

Anyway, well done.

[Péter Szilágyi]

Glad you like it :)

  As I mentioned previously, my goal with xgo was to quickly and easily cross compile for my particular use case. Because of this, it doesn't contain anything too fancy. However, the reason I released is publicly is because I thought it contains solutions to problems not yet solved, and maybe in the right hands it can grow out to be a valuable tool.

  So bottom line, xgo is perfect for me, but you're more than welcome to use it and incorporate it (as is, or whatever derivation) into goxc :) Just ping me if/when you do, so I can brag about it :D

Cheers,

  Peter

[Péter Szilágyi]

I've just noticed that the docker image contained the race binaries too, which aren't used by xgo. Dropping them reduced the container by 60MB download and 112MB final size. If anyone sees further unused packages that can be binned, shoot a mail :)

[Péter Szilágyi]

Hey all,

  I've reworked the xgo internals a bit, now it's possible to select which Go release should build the package through the -go flag (though for now only the 1.3.0 release is integrated... adding new ones is easy, but I need Dave Cheney to build his arm package before I can add the 1.3.1 release O:) ).

  $ xgo -go 1.3.0 github.com/project-iris/iris

  

In addition, I've also introduced two wildcard release strings: "latest" and "1.3.x". I guess it's pretty obvious what they do.

Cheers,

  Peter

[Camilo Aguilar]

I have spent so many hours in this problem too. This is a good step ahead, thanks Péter! 

Camilo Aguilar

Software Engineer

http://github.com/c4milo

[Péter Szilágyi]

Hi all,

  Just wanted to announce that my xgo - cgo enabled cross compiler - was updated to use Go 1.3.3 thanks to the amazing Go team and Dave Cheney for providing the arm packages! Currently it only buils ARM v5, but this will be extended soon to build all three versions (5/6/7), just needs a docker hierarchy reorganization.

  And for a friendly usage example:

$ docker pull karalabe/xgo-latest (optional, but it displays a progress report this way)

$ go get github.com/karalabe/xgo

$ xgo -out iris-v0.3.1 github.com/project-iris/iris

And voila:

$ ls -al

-rwxr-xr-x 1 root     root  3103916 Oct  3 16:56 iris-v0.3.1-darwin-386

-rwxr-xr-x 1 root     root  3962284 Oct  3 16:56 iris-v0.3.1-darwin-amd64

-rwxr-xr-x 1 root     root  4201688 Oct  3 16:56 iris-v0.3.1-linux-386

-rwxr-xr-x 1 root     root  5222248 Oct  3 16:56 iris-v0.3.1-linux-amd64

-rwxr-xr-x 1 root     root  4171936 Oct  3 16:56 iris-v0.3.1-linux-arm

-rwxr-xr-x 1 root     root  4248064 Oct  3 16:56 iris-v0.3.1-windows-386.exe

-rwxr-xr-x 1 root     root  5268992 Oct  3 16:56 iris-v0.3.1-windows-amd64.exe

  For more infos, command options and other goodies, head over to the project's intro page.

Try it, code it, share it :)

  Peter

[Sebastien Binet]

hi,

On Fri, Oct 3, 2014 at 4:03 PM, Péter Szilágyi <pet...@gmail.com> wrote:
> Hi all,
>
> Just wanted to announce that my xgo - cgo enabled cross compiler - was
> updated to use Go 1.3.3 thanks to the amazing Go team and Dave Cheney for
> providing the arm packages! Currently it only buils ARM v5, but this will be
> extended soon to build all three versions (5/6/7), just needs a docker
> hierarchy reorganization.
>
> And for a friendly usage example:
>
> $ docker pull karalabe/xgo-latest (optional, but it displays a progress
> report this way)
> $ go get github.com/karalabe/xgo
> $ xgo -out iris-v0.3.1 github.com/project-iris/iris
>
> And voila:
> $ ls -al
> -rwxr-xr-x 1 root root 3103916 Oct 3 16:56 iris-v0.3.1-darwin-386
> -rwxr-xr-x 1 root root 3962284 Oct 3 16:56 iris-v0.3.1-darwin-amd64
> -rwxr-xr-x 1 root root 4201688 Oct 3 16:56 iris-v0.3.1-linux-386
> -rwxr-xr-x 1 root root 5222248 Oct 3 16:56 iris-v0.3.1-linux-amd64
> -rwxr-xr-x 1 root root 4171936 Oct 3 16:56 iris-v0.3.1-linux-arm
> -rwxr-xr-x 1 root root 4248064 Oct 3 16:56 iris-v0.3.1-windows-386.exe
> -rwxr-xr-x 1 root root 5268992 Oct 3 16:56
> iris-v0.3.1-windows-amd64.exe
>
> For more infos, command options and other goodies, head over to the
> project's intro page.

just wondering:
any reason why you didn't reuse the cross-compilation enabled official
containers ?
https://registry.hub.docker.com/_/golang/

-s

[Péter Szilágyi]

Hi,

Two reasons:

• xgo is older than the official Go docker images
  
• xgo can compile CGO enabled code too (i.e. which contains C snippets)

Additionally, as far as I can tell by looking at your linked Dockerfiles, they use the classical cross compilation of building the tool-chain locally. This however means that OS dependent parts of the standard library (anything to do with certificates, among others) will fail to work.

Cheers,

  Peter

PS: But if there would be interest, I'd gladly merge my work into the official Go docker images. So addressing the Go team, if you're thinking along these lines, send me a ping and we can figure out a way to do it.

[Jérôme Champion]

Does that mean that it is possible with xgo to crosscompile the examples from github.com/veandco/go-sdl2 ?

If yes, how would you install the sdl2 lib in the containers? (sorry I don't really know how docker works )

[Péter Szilágyi]

Well, life's not that easy :D

  Cross compiling with C/C++ dependencies is a way tougher not to crack and hasn't really been solved yet. Consider that xgo takes Go out of the equation and reverts the original complex problem of cross compiling C embedded Go code to the still very complex problem of cross compiling C code.

  Since I myself don't really rely on any external libraries, only the OS APIs, my goal with xgo was to make cross compilation possible, as long as you stick to the standard APIs. Whether or not cross compiling with external dependencies is feasible is up to the challenger who actually needs it and is willing to put the effort in :)

Cheers,

  Peter

On Fri, Oct 3, 2014 at 6:01 PM, Jérôme Champion <cham...@gmail.com> wrote:

Does that mean that it is possible with xgo to crosscompile the examples from github.com/veandco/go-sdl2 ?

If yes, how would you install the sdl2 lib in the containers? (sorry I don't really know how docker works )

--

[Drew Wells]

This looks very compelling.  I have some C++ dependencies, which is really making cross compiling difficult.  Is there anyway to prebuild (universall) the C++ lib for use by cgo/xgo?

[Péter Szilágyi]

Nothing directly supported I'm afraid. As a mentioned in a few discussions, up till now nobody managed to solve C++ cross compilation is a completely portable and generic way which would work for even most libraries. Currently xgo supports only the core OS libraries, which were enough for my needs with Iris.

  I am quite sure, that as long as you don't want to be able to compile every possible C++ code someone could throw at it, just a few specific libraries of your choosing, it could be automated fairly easily, however as I had no need for such features, I cannot give any worthy guidelines. 

[Péter Szilágyi]

Hey all,

  Thanks to the PRs of Hasty Granbery and the packages of Dave Cheney and the Go team, the xgo cross compiler has been updated to Go 1.4, and in addition also features a few arm compatibility and dependency fixes :)

  As always, you can fetch the cross compiler via

$ go get github.com/karalabe/xgo

$ go install github.com/karalabe/xgo

  After which cross compiling can be done via:

$ xgo github.com/project-iris/iris

...

$ ls -al

-rwxr-xr-x 1 root     root   5891684 Dec 17 17:27 iris-darwin-386

-rwxr-xr-x 1 root     root   7501668 Dec 17 17:27 iris-darwin-amd64

-rwxr-xr-x 1 root     root   8116600 Dec 17 17:27 iris-linux-386

-rwxr-xr-x 1 root     root  10035096 Dec 17 17:27 iris-linux-amd64

-rwxr-xr-x 1 root     root   8053736 Dec 17 17:27 iris-linux-arm

-rwxr-xr-x 1 root     root   8194560 Dec 17 17:27 iris-windows-386.exe

-rwxr-xr-x 1 root     root  10113536 Dec 17 17:27 iris-windows-amd64.exe

The compiler will use Go 1.4 by default. For other versions see the `-go` flag.

Thanks and cheers,

  Peter

[Vivekanand S V]

Peter,

    I am trying to use your tool to compile my go application. My main.go resides in "domain.com/level1/main" package path. When I try to run

xgo -v domain.com/level1/main from src directory I get an error saying "package domain.com/level1/main: unrecognized import path "domain.com/level1/main""

How do I resolve this ? My source is not initialised as a repo, btw.

Thanks,

Vivek.

[Péter Szilágyi]

Hey Vivekanand,

  Are you trying to compile a local only package? I haven't added support yet to pull in packages from your local disk, so all packages and dependencies need to be available as public go get-able resources. I am considering sharing the local GOPATH to xgo and working with local packages too, but that requires a bit of thought (I don't want to overwrite any local stuff, so I need a copy... but I don't want to copy everything, only what is essential... which is not that trivial to figure out, since different platforms might have different deps).

  So all in all, if it's a public thing, please share the exact package path so I can check. If it's private, then at the moment it's not possible to pull it in, but I'll sort it out soon hopefully :)

Cheers,

  Peter

[Vivekanand S V]

Thanks Peter, yes its a local package. I'll fork it and see if I can do some magic and meet my requirements :)