go build -i and rebuilding a library

[James Graves]

Hi All,

I'm seeing an issue with rebuilding a library when changes are made to the source code in a separate library.

Suppose there are two directories in my GOPATH:

export GOPATH=${HOME}/vendor:${HOME}/golang

There's a library in vendor/src/mylib/file1.go that is imported by golang/src/myapp/main.go.

Running 'go build -i' in golang/src/myapp will build mylib.a and store it in vendor/pkg/linux_amd64, and then build the main application.  So far so good.

Now, let's suppose there's a change to file1.go, and I again run 'go build -i' in golang/src/myapp.

The system will not re-build the library mylib.a.  Any changes to files in the vendor/ directory don't seem to trigger rebuilding the library binary.

Is this expected behavior?  Do I instead need to always run 'go install' in the vendor/src/mylib directory?

Thanks,

James Graves

[Konstantin Khomoutov]

Please read [1]. From that, it follows, that you should always use
`go install` in your golang/src/myapp directory, and have the libraries
it depends on be automatically re-built and re-installed, if needed.

To be honest, I'm not sure why "-i" of `go build` even exists.
I'd expect it to make `go build` behave like `go install` but it seems
to not be the case.

1. https://groups.google.com/d/topic/golang-nuts/57Cwx7i5g7U/discussion

[James Graves]

On Monday, December 29, 2014 12:46:13 PM UTC-6, Konstantin Khomoutov wrote:

Please read [1].  From that, it follows, that you should always use
`go install` in your golang/src/myapp directory, and have the libraries
it depends on be automatically re-built and re-installed, if needed.

Thanks for the reply.

I just tried this using 'go install' for myapp.  It also did not pick up any changes
to the mylib/file1.go and rebuild the mylib.a library.

It seems it is definitely necessary to go into the mylib directory, and run 'go install'
there first, and afterwards run 'go install' in the myapp directory to build the application.

To be honest, I'm not sure why "-i" of `go build` even exists.
I'd expect it to make `go build` behave like `go install` but it seems
to not be the case.

My expectation is that if 'go install' or 'go build' creates a library file once, it should
then re-build it when needed, and not ignore changes to the source files once the .a
file exists.

I'm inclined to submit a bug report for this.  Do you think I need to clarify the situation
further?

BR,

James

[James Bardin]

On Monday, December 29, 2014 2:59:42 PM UTC-5, James Graves wrote:

On Monday, December 29, 2014 12:46:13 PM UTC-6, Konstantin Khomoutov wrote:

Please read [1].  From that, it follows, that you should always use
`go install` in your golang/src/myapp directory, and have the libraries
it depends on be automatically re-built and re-installed, if needed.

Thanks for the reply.

I just tried this using 'go install' for myapp.  It also did not pick up any changes
to the mylib/file1.go and rebuild the mylib.a library.

Yes, `go install` is not recursive. It will install dependencies if they are not already there, but it doesn't check them. 

 

It seems it is definitely necessary to go into the mylib directory, and run 'go install'
there first, and afterwards run 'go install' in the myapp directory to build the application.

Just because one happens to be in a subdirectory of another, the go too doesn't assume any relationship. Each package is a separate entity, and the go tool doesn't assume any relationship between them. As an example, since multiple packages could be in the same VCS tree, it would otherwise be difficult to build two versions against one another when the whole tree is updated via a checkout.

You can use the ellipsis wildcard to install multiple packages from the same tree. This will call install on "github.com/username/package", as well as "github.com/username/package/mylib"

`go install github.com/username/package/...`

 

[DisposaBoy]

I can't guarantee that I didn't imagine this, but IIRC the go tool does not re-build outdated packages accross GOPATH directories. For the life of me, I can't seem to find any documentation or issues discussing it.

[James Graves]

On Monday, December 29, 2014 4:02:39 PM UTC-6, DisposaBoy wrote:

I can't guarantee that I didn't imagine this, but IIRC the go tool does not re-build outdated packages accross GOPATH directories. For the life of me, I can't seem to find any documentation or issues discussing it.

I didn't see any mention of it in the official documentation either.  This behavior caught us by surprise.

Thanks,

James

[Dmitri Shuralyov]

On Monday, December 29, 2014 2:02:39 PM UTC-8, DisposaBoy wrote:

I can't guarantee that I didn't imagine this, but IIRC the go tool does not re-build outdated packages accross GOPATH directories. For the life of me, I can't seem to find any documentation or issues discussing it.

That is correct.

I can link to an issue where this behavior is discussed in detail with links to source code, documentation, past issues where it was mentioned:

https://github.com/jimmyfrasche/goutil/issues/2

I think it's a problem that we should find a way to fix, because it makes it impossible for tools that try to execute generated Go code to work reliably - their only solution is to use -a flag and always rebuild all dependencies, but that's slow.

We've raised/described the issue here, but it didn't seem to catch attention back then:

https://groups.google.com/forum/#!topic/golang-dev/svxuGkGJTD8

[Patrick Bennett]

On Monday, December 29, 2014 5:24:45 PM UTC-5, James Graves wrote:

I didn't see any mention of it in the official documentation either.  This behavior caught us by surprise.

I think the fact that Google themselves don't use go build or go install to actually build Go code is probably part of it.  I've seen it mentioned by members of the Go team before.

It would be nice if they used the same tools everyone else uses.  

Patrick

[Benjamin Measures]

On Monday, 29 December 2014 21:51:06 UTC, James Bardin wrote:

Yes, `go install` is not recursive. It will install dependencies if they are not already there, but it doesn't check them. 

Not so. See "Getting started with the go command" <https://golang.org/doc/articles/go_command.html#tmp_3>:

Because the go command can analyze the dependency graph, "go install" also installs any packages that this package imports but that are out of date, recursively.

It'd be pretty rubbish if it didn't.

[Benjamin Measures]

On Monday, 29 December 2014 22:02:39 UTC, DisposaBoy wrote:

I can't guarantee that I didn't imagine this, but IIRC the go tool does not re-build outdated packages accross GOPATH directories. For the life of me, I can't seem to find any documentation or issues discussing it.

This was introduced to fix issue 3149 (recompilation of system-installed stdlib) <https://github.com/golang/go/issues/3149>.

It could probably be documented better (eg. 'go help gopath'). For now, it's in code <https://github.com/golang/go/commit/b03a5f66e8f8a6b36c9d67e82d2edc9b3d4076ba>:

// Have installed copy, probably built using current compilers,

// and built after its imported packages.  The only reason now

// that we'd have to rebuild it is if the sources were newer than

// the package.   If a package p is not in the same tree as any

// package named on the command-line, assume it is up-to-date

// no matter what the modification times on the source files indicate.

// This avoids rebuilding $GOROOT packages when people are

// working outside the Go root, and it effectively makes each tree

// listed in $GOPATH a separate compilation world.

This undocumented behaviour is probably a major contributing reason as to why it is generally recommended to use a single $GOPATH. Personally, I can see certain scenarios in which it could be taken advantage of, for example pseudo-vendoring.

[James Bardin]

On Fri, Jan 2, 2015 at 8:26 AM, Benjamin Measures <saint....@gmail.com> wrote:

Not so. See "Getting started with the go command" <https://golang.org/doc/articles/go_command.html#tmp_3>:

Because the go command can analyze the dependency graph, "go install" also installs any packages that this package imports but that are out of date, recursively.

It'd be pretty rubbish if it didn't.

sorry, yes you're right. I think I had the problem OP is facing at one point, and just adjusted my workflow to explicitly reinstall everything for good measure when I knew there were changes outside of my current package tree. 

The caveat of multiple GOPATH's still applies though. 

[Benjamin Measures]

On Sunday, 28 December 2014 22:45:54 UTC, James Graves wrote:

export GOPATH=${HOME}/vendor:${HOME}/golang

There's a library in vendor/src/mylib/file1.go that is imported by golang/src/myapp/main.go.

[...] Now, let's suppose there's a change to file1.go

Wait, what? How did this happen? Is this package vendored or forked?

* If vendored (and not using godeps), then try 'go get -u' instead for update workflow.

* Else, if forked, it should go in your project workspace.

Any changes to files in the vendor/ directory don't seem to trigger rebuilding the library binary.
Is this expected behavior? 

Yes. each tree listed in $GOPATH forms a separate compilation world. The go tools won't check other worlds for updates (except to check that packages are built/installed).

[Jakob Borg]

2015-01-02 15:26 GMT+01:00 Benjamin Measures <saint....@gmail.com>:

On Sunday, 28 December 2014 22:45:54 UTC, James Graves wrote:

export GOPATH=${HOME}/vendor:${HOME}/golang

There's a library in vendor/src/mylib/file1.go that is imported by golang/src/myapp/main.go.

[...] Now, let's suppose there's a change to file1.go

Wait, what? How did this happen? Is this package vendored or forked?

* If vendored (and not using godeps), then try 'go get -u' instead for update workflow.

When using godep is when I usually run into this, and it's seriously annoying. The "how did it happen" in that case is usually something along the lines of "git pull; ./build.sh" where the build script uses godep.

Since that uses a "split" $GOPATH, and those are mentioned on the actual line to "go install" or whatever, the dependencies don't get rebuilt. If there were API changes in them, weird hilariousness ensues, sometimes followed by bug reports.

//jb

[James Bardin]

On Mon, Jan 5, 2015 at 5:46 AM, Jakob Borg <ja...@nym.se> wrote:

When using godep is when I usually run into this, and it's seriously annoying. The "how did it happen" in that case is usually something along the lines of "git pull; ./build.sh" where the build script uses godep.

Since that uses a "split" $GOPATH, and those are mentioned on the actual line to "go install" or whatever, the dependencies don't get rebuilt. If there were API changes in them, weird hilariousness ensues, sometimes followed by bug reports.

Ah, this was one of the reasons I stopped using godep. It's very easy to end up with a vendored version of your own libraries when you have multiple packages in single repo.

I've found that github.com/robfig/glock works well with our workflow. It's a little less automatic, but I personally prefer it that way. 

 

[Helin Wang]

Just to want let people who reached this post (#1 search result) know (like me), I think this problem has been addressed (since 2015 I think), more documentation here: https://github.com/golang/go/blob/c20e54533ea49ca68640d9a59c9ed935b27da8e5/src/cmd/go/internal/load/pkg.go#L1183

Helin