[ANN] Next xgo release: CGO compiling to Android, C++ support, local builds

[Péter Szilágyi]

Hey all,

It's been a ling while since I've updated my xgo project, and given that a lot of things have been happening, I thought it's high time to make a new release :)

For those who haven't heard about the project before, xgo is a CGO enabled cross compiler suite. Its goal is to allow building Go projects with embedded C/C++ code (or simpler external C/C++ dependencies) into various target platforms without the need to configure anything on your host machine (nor have it litter all over the place). It is based on docker containers and a lightweight Go wrapper tool around them, so with Docker and Go installed, cross compiling boils down to:

$ go get -u github.com/karalabe/xgo
$ xgo github.com/project-iris/iris
... 

$ ls -al
-rwxr-xr-x  1 root  root  10899488 Sep 14 18:05 iris-android-21-arm
-rwxr-xr-x  1 root  root   6442188 Sep 14 18:05 iris-darwin-386
-rwxr-xr-x  1 root  root   8228756 Sep 14 18:05 iris-darwin-amd64
-rwxr-xr-x  1 root  root   9532568 Sep 14 18:05 iris-linux-386
-rwxr-xr-x  1 root  root  11776368 Sep 14 18:05 iris-linux-amd64
-rwxr-xr-x  1 root  root   9408928 Sep 14 18:05 iris-linux-arm
-rwxr-xr-x  1 root  root   7131477 Sep 14 18:05 iris-windows-386.exe
-rwxr-xr-x  1 root  root   8963900 Sep 14 18:05 iris-windows-amd64.exe

I don't want to go into a detailed presentation of xgo, as that's way better done in the project's readme, but thought I'd just mention the novelties introduces over the last 1-2 weeks.

The main novelty is the Go version bump to 1.5 (and 1.5.1, default), which allowed the introduction of Android cross compilation targets. Xgo is now able to build CGO enabled (snippets or lighter dependencies) to Android too, while at the same time taking care of all the NDK linker quirks needed (e.g. Position Independent Executables, etc). The build outputs are the plain binaries, so packaging them up is left for the user for now (PRs welcome ;) ). Without any build targets specified, xgo will build for Android Lollipop (platform version 21). But you can easily change this by manually requesting platforms, versions and architectures:

$ xgo --targets=android-16/arm,android-21/arm ...

 

Beside supporting C dependencies, this new release introduces C++ cross compilation support, so any code interfacing with C++ can also be successfully compiled from now on (initial code credits to Robert Carlsen) . E.g. building the sample C++ hello world code: 

$ xgo github.com/karalabe/xgo/tests/embedded_cpp
... 

 

$ ls -al
-rwxr-xr-x  1 root  root  5052412 Sep 22 12:23 embedded_cpp-android-21-arm
-rwxr-xr-x  1 root  root   617936 Sep 22 12:23 embedded_cpp-darwin-386
-rwxr-xr-x  1 root  root   782032 Sep 22 12:23 embedded_cpp-darwin-amd64
-rwxr-xr-x  1 root  root   954313 Sep 22 12:23 embedded_cpp-linux-386
-rwxr-xr-x  1 root  root  1200355 Sep 22 12:23 embedded_cpp-linux-amd64
-rwxr-xr-x  1 root  root  1017956 Sep 22 12:23 embedded_cpp-linux-arm
-rwxr-xr-x  1 root  root   829376 Sep 22 12:23 embedded_cpp-windows-386.exe
-rwxr-xr-x  1 root  root  1020352 Sep 22 12:23 embedded_cpp-windows-amd64.exe

The last big feature included in this release is something requested for a long time now, cross compilation from the local GOPATH. Up until now xgo only accepted canonical import paths, which it then retrieved from the internet. This lead to two different issues: building from private repositories was impossible, since xgo didn't have access to them; and debugging was again a pain since you needed to push any WIP code to a public repo before xgo could retrieve it. The current release finally addresses this issue, by allowing you to point xgo to a local folder instead of a canonical import path (i.e. start with '.' or '/'). In those cases xgo will mount your entire GOPATH (multiple paths, symlinks allowed) into the cross compilation container (read only), and builds using that code instead go get-ing from the network (initial code credits to Kevin Darlington).

A shout out is also called for to Tudor Golubenco, who kindly sent in a few polishing PRs as well as a feature PR to support using custom docker images (instead of the official ones) so that anyone could create their own extensions on top of the base images and have xgo work with them.

All in all I'd like to thank everyone their valuable contributions, and hope other too will find this small docker based cross compilation suite useful :). There are probably a lot of rough edges still, missing features, bugs, etc. so I ask kindly anyone bumping in to them to feel free and file an issue at the project's tracker, and maybe we can work out a solution for them.

Happy coding,

  Peter

[Vivekanand S V]

Hi Peter,

     Is there any examples on cross compiling using local GOPATH ? I am actually not able to get it working, even the example given by kdar here is not working for me as it ends up in a panic on line no 182. I had to run the command from $GOPATH to avoid the panic as it was looking for src folder in the current directory

-> So, should we be running the command from $GOPATH ? 

1) If not, please give me an example.

2) If yes, what should be done in-case we have multiple paths set in $GOPATH ? And after avoiding the panic by running from $GOPATH, I still get the following error when I run kdar's example.

COMMAND = sudo /home/test/go_cross_compiler/bin/xgo --targets=windows/* -out "build/test" ./   (I use sudo as I didnt add permission to the normal user yet)

GOPATH = /home/test/xgo_test (it contains the src/test folders and main.go file as in kdar's example)

### COMMAND OUTPUT START ###

Checking docker installation...

Client:

 Version:      1.8.2

 API version:  1.20

 Go version:   go1.4.2

 Git commit:   0a8c2e3

 Built:        Thu Sep 10 19:19:00 UTC 2015

 OS/Arch:      linux/amd64

Server:

 Version:      1.8.2

 API version:  1.20

 Go version:   go1.4.2

 Git commit:   0a8c2e3

 Built:        Thu Sep 10 19:19:00 UTC 2015

 OS/Arch:      linux/amd64

Checking for required docker image karalabe/xgo-latest... found.

Cross compiling ....

Building locally ....

Compiling for windows/amd64...

can't load package: package .: no buildable Go source files in /

2015/09/24 17:26:26 Failed to cross compile package: exit status 1.

### COMMAND OUTPUT END ###

Can you please shed some light on this ? Am I using it in a wrong way ?

Thanks,

Vivek.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Péter Szilágyi]

Hey Vivek,

1. So, first up xgo requires you to have a valid $GOPATH work structure. If you can do `go get github.com/karalabe/xgo`, then it should already be properly configured, and it should also put xgo into a runnable place (if you haven't added your $GOPATH/bin folder to $PATH, it might be a good idea as it saves you a lot of trouble. You should be able to run `xgo --help` from anywhere.
   
2. Xgo currently assumes, that if you want to cross compile something, it is located within your $GOPATH, at a proper import path (e.g. $GOPATH/src/my/import/path). The reason I require this is because most projects will have multiple internal dependencies (and have the command located in a project/cmd/... folder), so xgo needs to be able to locate all the dependencies, which can be done only if they are on the correct paths.
3. If you have $GOPATH set up and you have a project within it that you wish to compile, just point xgo to it (`xgo /some/path/to/package`). You don't have to run xgo from within $GOPATH, just the sources you wish to compile should be in there. Xgo will use the $GOPATH env var to figure out the dependencies and to mount the correct folders for cross compilation.

A few remarks: the --out flag is used to specify an output prefix to use for the build results, not the build destination folder. You can set the destination folder via --dest. The example command you quoted is not the one supported by xgo. A proper example to build that would be:

$ cd $GOPATH/src       // If you have multiple (I have at least 3 in there), change into any, doesn't matter
$ mkdir xgo_test
$ cd xgo_test
$ echo -e 'package main\n import "fmt"\n func main(){\nfmt.Println("Hello")\n}' > main.go
$ xgo .
...
Compiling for android-21/arm...
Compiling for linux/amd64...
Compiling for linux/386...
Compiling for linux/arm...
Compiling for windows/amd64...
Compiling for windows/386...
Compiling for darwin/amd64...
Compiling for darwin/386...

The main difference is, that you need to put the file into a package (I placed it into a top level xgo_test). Doesn't matter what the name is or how complex the path to it is, but it cannot be a dangling file in the root src folder. That is not valid Go path as far as I know.

Hope this helps, but if not, please ask and I'll clarify :)

[Vivekanand S V]

So it was my mistake, as I was running as root user (using sudo) the GOPATH was not being forwarded to the xgo command, I added GOPATH while running that command as sudo and things work fine as expected.

Thanks for your time Peter. This was a nice lesson to me :)

Vivek.

[Péter Szilágyi]

Glad it worked out. Though xgo could probably be made a bit smarter to detect this and at least provide a hint instead of plain crashing. Will do on Monday :) Cheers

[Péter Szilágyi]

Just a follow up to last week's issue, I've pushed a bugfix version of xgo out:

• If no GOPATH is set or forwarded, report an error and bail out.
• If certain parts of GOPATH are not accessible (e.g. permission denied), report but continue.
• Fix an issue where Godeps weren't properly detected if a sub-package was built.
• Fix an issue where branch/remote selection didn't work if a sub-package was built.
• Fix a regression where workspace containers weren't cleaned up after cross compilation.

I'd like to shout out to Vivekanand for reporting the GOPATH issues and Anthony Cros for reporting the sub-package build issues.

To update to the latest xgo:

  go get -u github.com/karalabe/xgo

And to clean up old container (and update to new ones):

  docker rmi `docker images | grep karalabe/xgo-* | awk '{print $3}'`

Cheers,

  Peter