Modules and cross-compilation

[geoff.j...@gmail.com]

Hello.

I’m getting rather confused by Go modules and cross-compilation. I think that I have fundamentally misunderstood some module-related concept in the way that I’ve structured my project.

Scenario: My project consists of some individual executables that share some common code. The project is (or should be) a coherent, self-contained whole: the executables don’t make much sense individually, and I want to work on / version the supporting code and executables together.

In GOPATH times I had something like:

$ find . -type f
./src/cmd/foo/main.go
./src/cmd/bar/main.go
./src/qux/qux.go

$ cat src/qux/qux.go
package qux

const Qux = `qux`

$ src/cmd/foo/main.go     # bar/main.go is essentially the same
package main

import "qux"

func main() {
    println(`foo: ` + qux.Qux)
}

and the following works as expected, following my understanding that cross-compilation just means setting GOOS and/or GOARCH:

$ uname -sm
Darwin x86_64

$ GOPATH=$PWD go install ./...
$ GOOS=linux GOARCH=arm GOPATH=$PWD go install ./...

$ find bin -type f
bin/linux_arm/foo
bin/linux_arm/bar
bin/foo
bin/bar

I’ve tried to migrate this to modules. At a logical level, these binaries plus support code are one unit: if Go doesn’t require me to spread them across separate modules then I’d rather not.

First I removed the src directory, then added a go.mod:

$ find . -type f
./cmd/foo/main.go
./cmd/bar/main.go
./qux/qux.go

$ go mod init foo.com/foobar

This baffles me:

$ go build foo.com/foobar
can't load package: package foo.com/foobar: cannot find module providing package foo.com/foobar

$ head -1 go.mod
module foo.com/foobar

$ go list -m
foo.com/foobar      # but this finds it perfectly well?

although this seems to work:

$ GOBIN=/some/path go install ./...
$ find /some/path -type f
/some/path/foo
/some/path/bar

But: now I want to cross-compile:

$ GOOS=linux GOBIN=/some/path go install ./...
go install: cannot install cross-compiled binaries when GOBIN is set
go install: cannot install cross-compiled binaries when GOBIN is set

(Why not? If I set GOBIN then I’m saying “please put the binaries over there”. I know that I don’t want them appearing on my Mac’s $PATH: that’s what I’m setting GOBIN. I want to cross-compile some binaries and have them appear in some suitable output directory, so that I can then put them into some disk image or whatever. In any case, if this runs in CI then, cross-compilation or not, I don’t want them appear on the CI box’s PATH at all.)

The Internet says that I should use go build instead of go install, and use -o to write the results somewhere:

$ GOOS=linux go build -o /some/path ./...
go build: cannot write multiple packages to non-directory /some/path

This seems to work, but I’ve never had to create the output directory before, and it doesn't do the handy creation of per-OS/ARCH subdirectories for me:

$ mkdir /tmp/bin
$ GOOS=linux go build -o /some/path ./...

So:

1. I don’t understand why go build foo.com/foobar fails to find the module if it doesn’t contain some top-level .go file, but go list -m finds it perfectly fine.

2. [How] can I have a module X that provides packages X/Y and X/Z, with no source files in the top-level package X? Or am I not supposed to do this under Go modules?

3. Go modules appear to break my assumption that “I just set GOOS and/or GOARCH to cross-compile, and everything else just works”. The is (was?) one of the biggest benefits of Go for me. I need to cross-compile for Linux (ARM, AArch64 and AMD64) and Windows (AMD64) all the time.

4. Cross-compiled go install to some non-global output directory worked fine in GOPATH days: creating $GOPATH/bin/$GOOS-$GOARCH/*, which seemed perfectly sensible to me. I don’t understand why it’s broken under modules, particularly when I set GOBIN, which I take to mean “Put the binaries here please: I know what I’m doing; please don’t argue”. Or at least have some sort of -yes-really flag to stop the toolchain arguing.

At the moment I can see two solutions:

1. Split up my project into individual modules for the shared code (or even every package therein), and each executable. Since none of those components, in this context, makes sense individually, this seems like the tail wagging the dog.

2. Revert to using GOPATH and hope that it never gets removed. History teaches me that this is unwise.

but I can’t help thinking that I’m doing something fundamentally wrong. I’ve read the docs, help pages, Wiki et cetera, and found nothing that has helped me. Can someone please put me right?

Many thanks,

Geoff.

[Volker Dobler]

On Wednesday, 26 February 2020 20:53:52 UTC+1, geoff.j...@gmail.com wrote:

This baffles me:

$ go build foo.com/foobar
can't load package: package foo.com/foobar: cannot find module providing package foo.com/foobar

You simply cannot build a module. Only packages can be built.

and there is no package at the root of your module. Try

   cd cmd/foo; go build

or go build foo.com/foobar/cmd/foo

 

$ head -1 go.mod
module foo.com/foobar

$ go list -m
foo.com/foobar      # but this finds it perfectly well?

Yes. You are asking questions about a module and this

question is answered.

 

although this seems to work:

$ GOBIN=/some/path go install ./...
$ find /some/path -type f
/some/path/foo
/some/path/bar

Well sure. Asked to build every package and that was done.

 

[... larger rant on the tooling ...]

 

So:

1. I don’t understand why go build foo.com/foobar fails to find the module if it doesn’t contain some top-level .go file, but go list -m finds it perfectly fine.

The error message clearly is suboptimal and I think there is

an issue already for that. Just pick it up and fix it.

Note that what you asked the go tool to do is not

something it can do. The go tool operates on packages

during build, install, test, etc. and not on modules.

 

1. [How] can I have a module X that provides packages X/Y and X/Z, with no source files in the top-level package X? Or am I not supposed to do this under Go modules?

Just like you did. Everything works just fine, you just

cannot ask the go tool to build or install the module.

You noticed that module stuff works properly without

*.go files in the module root.

Remember: A  module is a collection of packages versioned

together and their dependencies.

1. Go modules appear to break my assumption that “I just set GOOS and/or GOARCH to cross-compile, and everything else just works”. The is (was?) one of the biggest benefits of Go for me. I need to cross-compile for Linux (ARM, AArch64 and AMD64) and Windows (AMD64) all the time.

I do not see how modules infer with cross-compilation.

I doubt you could set GOBIN in GOPATH mode either

1. Cross-compiled go install to some non-global output directory worked fine in GOPATH days: creating $GOPATH/bin/$GOOS-$GOARCH/*, which seemed perfectly sensible to me. I don’t understand why it’s broken under modules, particularly when I set GOBIN, which I take to mean “Put the binaries here please: I know what I’m doing; please don’t argue”. Or at least have some sort of -yes-really flag to stop the toolchain arguing.

You should probably address this on the issue tracker.

 

At the moment I can see two solutions:

1. Split up my project into individual modules for the shared code (or even every package therein), and each executable. Since none of those components, in this context, makes sense individually, this seems like the tail wagging the dog.

2. Revert to using GOPATH and hope that it never gets removed. History teaches me that this is unwise.

but I can’t help thinking that I’m doing something fundamentally wrong. I’ve read the docs, help pages, Wiki et cetera, and found nothing that has helped me. Can someone please put me right?

I do not see why doing a simple mkdir before running go install

would be a no-go. It might take a handful of more commands

to crosscompile multiple executables from a module but nothing

rocket science. If typing these command annoys you: put the in 

a shell script.

V.

[Manlio Perillo]

On Wednesday, February 26, 2020 at 8:53:52 PM UTC+1, geoff.j...@gmail.com wrote:

Hello.

> [...]

So:

1. I don’t understand why go build foo.com/foobar fails to find the module if it doesn’t contain some top-level .go file, but go list -m finds it perfectly fine.

This fails with GOPATH mode, too:

can't load package: package foo.com/foobar: no Go files in .../foo.com/foobar

The problem is that the error message is incorrect/confusing.

I have tested with go1.14 and it is even more confusing:

can't load package: cannot find module providing package github.com/perillo/gocmd: invalid github.com/ import path "github.com/perillo"

1. [How] can I have a module X that provides packages X/Y and X/Z, with no source files in the top-level package X? Or am I not supposed to do this under Go modules?

Of course you can work with it.

Just do go build -o build. ./...

Note that this will fail if there are no main packages in the main module.

When cross compiling you can create a Makefile or shell script, that do:

mkdir -p build/$GOOS-$GOARCH

go build -o build/$GOOS-$GOARCH/ ./...

1. Go modules appear to break my assumption that “I just set GOOS and/or GOARCH to cross-compile, and everything else just works”. The is (was?) one of the biggest benefits of Go for me. I need to cross-compile for Linux (ARM, AArch64 and AMD64) and Windows (AMD64) all the time.

I have to admit that I never noticed that go build installed executables in $GOBIN/$GOOS-$GOARCH.  I don't know the reason for the change.

> [...]

Manlio