How are we meant to build Go binaries using build systems?

[Peter Bourgon]

The canonical way of publishing Go code is providing a repo which is
"go get"-able. OK, fine. So, now I want to build some well-behaved
canonical Go code, and do something with the artifact, like ship it
somewhere.

I have a build infrastructure, like Travis, which can build any repo.
The build system takes as input a repository (location) and a
revision. It will check out that repo at that revision, and then do
<something>, which (if successful) will produce my build artifact in a
known location.

What is <something>?

It's not "make", because my canonical Go code doesn't have a Makefile.

It's not "go build", because that doesn't produce an artifact.

It could be "go install", but my checked-out repo isn't necessarily in
a $GOPATH. In fact, I don't even have a $GOPATH set in my build
environment. That's fine, I can set up an arbitrary $GOPATH as part of
my <something>, and create $GOPATH/src/myrepo as a symlink to
/path/to/checked/out/myrepo. Except, whoops, none of my dependencies
are there. How does the Go toolchain resolve dependencies? Right --
"go get" automatically fetches, builds, and installs them. I should
probably leverage that.

So it could be "go get ." (hat tip to Alex Surma for discovering this
AFAIK). I've actually been using this method for some time in another
context, but I've now hit a wall trying to help get Go support in
Travis. When I create the symlink and cd to $GOPATH/src/myrepo, the go
tool gets confused about where it is and says:

/arbitrary/gopath/src/myrepo$ go get -v .
go install: no install location for /path/to/checked/out/myrepo

The exact problem is at [0]. On advice, I tried dropping the symlink
and copying the .go source files to the $GOPATH/src directory
directly, and that worked[1]. So there appears to be some buggy
behavior in "go get". But that's really not my point, which is rather
that this all seems so fragile, so poorly-thought-out. (Is "go get ."
even officially supported behavior, or is it working by accident?) In
any case I can't go back to my contact at Travis and suggest the
default behavior for Go projects is to cp -r the folder to another
location; he's already generally mystified by the GOPATH requirements,
especially given that no other supported language (of a dozen+)
demands anything like this level of specificity.

I digress. I must be missing something: given that you're sitting in a
directory which contains exactly the Go code you want to build, *with
no other assumptions about your environment*, how are you meant to
convert that into an artifact (ie. binary), canonically?

_____
[0] http://travis-ci.org/#!/peterbourgon/g2g/builds/1820924
[1] http://travis-ci.org/#!/peterbourgon/g2g/builds/1821107

[Rob 'Commander' Pike]

% cat Makefile

all:

  go build

%

-rob

[Peter Bourgon]

On Tue, Jul 10, 2012 at 1:14 AM, Rob 'Commander' Pike <r...@golang.org> wrote:
> % cat Makefile
> all:
> go build
> %

$ go build
foo.go:8:2: import "github.com/bar/baz": cannot find package
foo.go:8:3: import "github.com/bing/bop": cannot find package

[Ian Davis]

On Tue, Jul 10, 2012 at 12:19 AM, Peter Bourgon <pe...@bourgon.org> wrote:
> On Tue, Jul 10, 2012 at 1:14 AM, Rob 'Commander' Pike <r...@golang.org> wrote:
>> % cat Makefile
>> all:
>> go build
>> %
>
> $ go build
> foo.go:8:2: import "github.com/bar/baz": cannot find package
> foo.go:8:3: import "github.com/bing/bop": cannot find package
>
>

How does your build infrastructure normally fetch dependencies such as
shared libraries or jar files? Can you hook into that workflow to go
get these package dependencies?

Ian

[Peter Bourgon]

Sure, there's probably a mechanism. What's the canonical way to
enumerate dependencies for a Go project? As far as I'm aware, it
doesn't exist, because "go get" deduces them.

remy_o on IRC suggested [0], from which you could probably derive [1].
Hat tip to him for that: it passes the "does it work?" test, but I
think fails the "can you recommend it to your build team with a
straight face?" test, in the same way that "cp -r the directory to an
arbitrary location" fails.

[0] go list -f "{{range .Imports}}{{ . }} {{end}}" .
[1] for D in `go list -f "{{range .Imports}}{{ . }} {{end}}" .` ; do
go get -v $D ; done

[Ian Davis]

Just thinking out loud, perhaps the way to go is to read the .go files
in your directory (either parse with Go or probably a regex will
suffice), set GOPATH environment variable to a temporary directory, go
get each dependency and then go build the source.

Ian

[Ian Davis]

Parsing the current package for dependencies, installing them and then
building the package could be a good candidate addition for the go
tool.

Ian

[Peter Bourgon]

> Parsing the current package for dependencies, installing them and then
> building the package could be a good candidate addition for the go
> tool.

+1 :)

[DisposaBoy]

On Tuesday, July 10, 2012 12:29:14 AM UTC+1, Peter Bourgon wrote:

[...]

Sure, there's probably a mechanism. What's the canonical way to
enumerate dependencies for a Go project? As far as I'm aware, it
doesn't exist, because "go get" deduces them.

remy_o on IRC suggested [0], from which you could probably derive [1].
Hat tip to him for that: it passes the "does it work?" test, but I
think fails the "can you recommend it to your build team with a
straight face?" test, in the same way that "cp -r the directory to an
arbitrary location" fails.

[0] go list -f "{{range .Imports}}{{ . }} {{end}}" .
[1] for D in `go list -f "{{range .Imports}}{{ . }} {{end}}" .` ; do
go get -v $D ; done

why the need to loop? why not `go get $deps` where $deps is your list of dependencies and if you're going to use go get anyway, for what purpose do you need the list of dependencies

 

[DisposaBoy]

isn't that exactly what `go get` does?

 

[Rob 'Commander' Pike]

% cat Makefile

all:

  go get -d && go build

%

-rob

[Jesse McNelis]

On Tue, Jul 10, 2012 at 8:35 AM, Peter Bourgon <pe...@bourgon.org> wrote:
> especially given that no other supported language (of a dozen+)
> demands anything like this level of specificity.

C wants it's dependencies in /usr/lib/
Python wants it's dependencies in /usr/lib/python/
Java wants it's dependencies in it's classpath.

Putting dependencies in a specific location seems pretty common to me.


--
=====================
http://jessta.id.au

[Cole Mickens]

If you're looking for complexity and/or finer control, could you not invoke the `go tool` commands manually? Instead of relying on `go {get,install,build}` that expect the canonical environment setup?

[Andrew Wilkins]

On Tuesday, 10 July 2012 06:35:18 UTC+8, Peter Bourgon wrote:

I digress. I must be missing something: given that you're sitting in a
directory which contains exactly the Go code you want to build, *with
no other assumptions about your environment*, how are you meant to
convert that into an artifact (ie. binary), canonically?

Why not just do this? (for package "github.com/peterbourgon/g2g")

mkdir -p /tmp/gopath/src/github.com/peterbourgon

ln -s /path/to/peterbourgon/g2g /tmp/gopath/src/github.com/peterbourgon/g2g

export GOPATH=/tmp/gopath

go get github.com/peterbourgon/g2g

Cheers,

Andrew

[Rob 'Commander' Pike]

go get installs the target, which you don't want here.

-rob

[Rémy Oudompheng]

On 2012/7/10 Peter Bourgon <pe...@bourgon.org> wrote:
> Sure, there's probably a mechanism. What's the canonical way to
> enumerate dependencies for a Go project? As far as I'm aware, it
> doesn't exist, because "go get" deduces them.
>
> remy_o on IRC suggested [0], from which you could probably derive [1].
> Hat tip to him for that: it passes the "does it work?" test, but I
> think fails the "can you recommend it to your build team with a
> straight face?" test, in the same way that "cp -r the directory to an
> arbitrary location" fails.
>
> [0] go list -f "{{range .Imports}}{{ . }} {{end}}" .

I'm not sure why you think this command was a bad idea. You said it
was for an automatic build system, and I don't see it becoming better
except if you start implementing subcommands like "go doWhatIWant" "go
doWhatHeWants" "go dowhattheywant" etc.

You seem to have a better syntax in mind, so feel free to suggest it.

> [1] for D in `go list -f "{{range .Imports}}{{ . }} {{end}}" .` ; do
> go get -v $D ; done

Rémy.

[Peter Bourgon]

On Tue, Jul 10, 2012 at 1:42 AM, Rob 'Commander' Pike <r...@golang.org> wrote:
> % cat Makefile
> all:
> go get -d && go build
> %

I was under the impression that simply downloading dependencies was
somehow not enough, because "go build" would just look for the .a's.
But I see now that's not true. So, I rather like this solution:

export GOPATH=/tmp/gopath
mkdir -p $GOPATH
go get -d
go build

The "go get -d" populates $GOPATH/src, and "go build" looks there for
dependency source + builds it as necessary. The output is a binary in
pwd, or nothing if it's a lib.

I'm building and testing this now. Any reason I shouldn't do "go build
./..." by default?

[Naitik Shah]

A bit late to the party, but was very happy to see Go support in

Travis. I played with it over the weekend and have some thoughts about

how it could be made more robust and support a bigger variety of use

cases. I'll try to summarize my thoughts in a few sections. Note, I've

also filed an issue on Github with much of the same stuff:

https://github.com/travis-ci/travis-build/issues/25

--

1. The current setup only runs tests for the top level package in the

repository.

For simple projects that only define a single package, this works

well. It is common though to have multiple packages in the same

repository. For example one of my repository defines a few packages:

https://github.com/daaku/go.grace -- notably the valid import paths (in

go get compatible terms) for packages (and a demo command) for this

repository are:

github.com/daaku/go.grace

github.com/daaku/go.grace/gracehttp

github.com/daaku/go.grace/gracedemo (a command)

The current setup on Travis won't end up running the tests contained in

the gracehttp package since it only runs them in the top level directory

(which in this case does not contain tests).

Go provides the "..." syntax to help with this, described in the section

here: http://golang.org/cmd/go/#Description_of_package_lists

--

2. The current setup will not handle inter package dependencies correctly.

This is slightly tricky and does not surface in the current setup since

multiple packages in a repository are not handled anyways. But one way

to solve the mult package issue from (1) is to replace the line here:

https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/job/test/go.rb#L21

with this:

  "go get -d -v ./... && go build -v ./..."

This would "mostly" work. The case this would not work in is when one of

the contained packages in the repository references another package in

the same repositry. Using the same example from earlier, the gracehttp

package has such a inter repository reference:

https://github.com/daaku/go.grace/blob/master/gracehttp/http.go#L9. With

the command I mentioned above, and the current Travis setup, this would

end up downloading the current version of the package from Github into

#{gopath}/src/github.com/daaku/go.grace instead of using the already

checked out copy. This is because the above go get command does a

"local" build, and Go doesn't know that the package is in fact

"github.com/daaku/go.grace". In order to fix this, the checked out

repository needs to be setup into the gopath as part of the build

process.

--

3. The current build does not account for test/xtest only dependencies

(such as an assertion library).

This one is easier to fix, Go provides the go list command which can be

used to extract these dependencies and its output can be used as input

to go get and the dependencies can be fetched. For example:

  TEST_DEPS=$(go list -f '{{.TestImports}} {{.XTestImports}}' github.com/$GITHUB_USER/$GITHUB_REPO/... | sed -e 's/\[//g' | sed -e 's/\]//g')

  if [ "$TEST_DEPS" ]; then

    go get -v $TEST_DEPS

  fi

--

I think the main "contentious" point about my suggestions above is

probably in point (2) since it introduces the need to tie the Travis

code to "github.com". This is because "go get" uses URLs to reference

packages and promotes the use of these in the packages themselves. I'm

not sure if there is a better way to handle this, but maybe someone else

on this list does.

Thanks very much for making Travis available and the hard work you

guys have done on it. You had already provided enough tools to make it

possible for me to have my own scripts to use Travis for my Go projects and

you can see it here:

https://github.com/daaku/go.travis/commits/master/install

I hope you guys keep Go support around and keep it as a supported

language!

Thanks,

-Naitik

[Peter Bourgon]

If you want to do this, the Best Practice method (currently) is to
fork their repo, and maintain your fork at the revision you need.

On Mon, Mar 25, 2013 at 3:47 PM, <alexba...@googlemail.com> wrote:
> One thing that does confuse me, is how to specify a specific version of
> someones code? say, their v1.0 or a git commit hash
>
> As at the moment, i could develop locally and then on build somewhere else,
> it would pull the latest version of their code which could break everything,
> if the api has changed..

[Alex Barlow]

Hmm,

I can see the thinking behind Go's current way of doing things. It also wasn't important to begin with, the language comes first.

But I would be interesting in a getting something a bit more along the lines of Aptitude/Rubygems. Versioning would be great as well as helping other developers/server get the correct decencies easily.

Perhaps we could float some ideas around and make something simple to start with? It could run on App Engine of course

--
Alex Barlow
alexba...@gmail.com
+447826621151

[atomly]

Yeah, I'm a big fan of how ivy, rubygems, etc. do this, where a package relies on specific versions of other packages, which can be checked out with a known command and then cached, etc..

:: atomly ::

[ ato...@atomly.com : www.atomly.com  : http://blog.atomly.com/ ...
[ atomiq records : new york city : +1.347.692.8661 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Job van der Zwan]

On Monday, 25 March 2013 23:10:53 UTC+1, atomly wrote:

Yeah, I'm a big fan of how ivy, rubygems, etc. do this, where a package relies on specific versions of other packages, which can be checked out with a known command and then cached, etc..

:: atomly ::

I've never used a system like that, but wouldn't that result in importing different versions of package files per .go file unless you're really carful about refactoring every time you change things? Or set off weird chain-reactions by changing a version number of an imported package file (because that updated packagefile also updates the version of its imports, and so forth). Again, no experience with it, just wondering.

[Alex Barlow]

Well, there are two trains of thought.

NPM (the node.js version manager) will use the version specified for each require.

For example, your software may want "twitter" >= 1.0.0 and "twitter_helper" => 1.0.0. "twitter_helper" wants "twitter" >= 0.0.9. NPM will do that, you will get the later "twitter" and "twitter_helper" will get the slightly older one, meaning both pieces of software will get the version they need and were written against/specified..

Bundler (for Ruby), will collect all the requirements for your application and all its requirements and then calculate the optimum version of each and use that.

NPM can only do as CommonJS (the require system in node) can import locally and the scope of imports are only to that file. Like C, Go and others. Ruby's requires are global.

Both work great, but are slightly different..

Changing a version in you specifier will indeed cause other repercussions, but that is intended, you want a new version..

--

Alex Barlow

alexba...@gmail.com

+447826621151

[atomly]

Yes, changing the version of a required package will have repercussions, but that is exactly the definition of a dependency graph.  I depend on XX1.0, which depends on YYY0.99.  XX adds foo suport in 1.1, which I need for a bug fix.  I release a new version of my code that depends on XX1.1, which in turn depends on YYY1.0, so both of those are now downloaded to build my package, which is exactly the desired behavior.

I think Ivy does an exceptional job with this: http://ant.apache.org/ivy/history/latest-milestone/index.html

:: atomly ::

[ ato...@atomly.com : www.atomly.com  : http://blog.atomly.com/ ...
[ atomiq records : new york city : +1.347.692.8661 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...

--