"go build" performance question / feature request

[Phil S.]

Hi gophers,

now "go build" is a really neat tool but it seems like it recompiles
all source files everytime, making a clean temp directory that it
later deletes again, even if only one (and quite often fairly small)
source file was changed.

Now if I went the manual way with 6g and 6l, I would only have to 6g
the changed file and then 6l the output of that, and 6l whatever
depends on it up to the main package.

Which should be a great deal faster than brute-force recompiling and
relinking a bigger package's entire src tree.

So what I'm wondering -- is there a secret flag -- or any hope for a
future flag -- enabling the following behavior?

1. a work dir can be *specified* and this does not get cleaned or
deleted by go build

2. if for any candidate .go source file an equivalent .6 / .8 obj file
exists in work dir from a previous build, compare file modification
times to decide whether to gc it

3. perform linking only on newly created obj files or on those
depending on it

I presume go build already does parse imports to build a dependency
tree in order to issue gc and ld calls in the proper order.

Or would the above in fact do nothing to improve speeds?

Of course, the user would then have to assume the responsibility to
periodically clean their work dir or once per day perform a full clean
classic build. But for those fast change/compile/check dev cycles the
above would be extremely productivity-enhancing.

Of course, I know that the Go compiler by itself is comparatively
fast. Yet right now I'm having a go build time of 1s for a fairly
simple and not really too complex project linking in a couple of
external, pre-compiled packages. That 1s is almost beginning to "break
the flow" sometimes and this number will only get up as my project
gets more real-world over the coming months. Of course, anything below
say 8s is still acceptable in the grand scheme of things... but since
we don't have a hot-code interpreter/JITter to instantly test various
code ideas like in JS/Python/Lisp etc, getting at least "as instant as
possible while still producing a complete up-to-date build" is
definitely an important "programmer productivity" item.

[Corey Thomasson]

One problem that immediately springs to my mind is that go only cares about function definitions at the package level.

If my project contains foo.go and bar.go and foo calls functions from bar. If I only recompile foo how does the compiler know about the functions from bar? Even at best it needs to compile every edited package rather than changed files.

[Andrew Gerrand]

Just use 'go install' instead of 'go build'. That will only build what
is necessary.

Andrew

[Kyle Lemons]

On Wed, May 2, 2012 at 7:39 PM, Phil S. <philipp....@gmail.com> wrote:

Hi gophers,

now "go build" is a really neat tool but it seems like it recompiles
all source files everytime,

This is somewhat overstating the case; for a single package, yes, it compiles all source files whenever it recompiles that package.  That's not a problem, because the go compilers are blisteringly fast.  It does not recompile imported packages that are up-to-date, however, except when you are using local imports. (one of the many reasons not to use them).  If you notice that it's compiling a dependent package each time, you need to "go install" them, as by default "go build" won't try to clobber existing installed libraries.

 

making a clean temp directory that it
later deletes again, even if only one (and quite often fairly small)
source file was changed.

The temp directory is for build artifacts.  The number of files changed is irrelevant to what gets put there.  The "go build" command "shadows" any stale package archives by compiling them into this temp directory in favor of blowing away whatever may or may not be installed in $GOPATH/pkg without permission.

 

Now if I went the manual way with 6g and 6l, I would only have to 6g
the changed file and then 6l the output of that, and 6l whatever
depends on it up to the main package.

That is indeed what the go tool does, when you tell it to do so (via "go install").

 

Which should be a great deal faster than brute-force recompiling and
relinking a bigger package's entire src tree.

Again, that's not what it does.

 

I presume go build already does parse imports to build a dependency
tree in order to issue gc and ld calls in the proper order.

Yep, it does.  "go list -f {{.Deps}} ." to see :)

[Phil S.]

Oh ok, awesome. Should've thought of trying that! Thought go install
is just for remote packages but then, of course that's what go get is
for. Cheers!



On May 3, 12:38 pm, Kyle Lemons <kev...@google.com> wrote:

> On Wed, May 2, 2012 at 7:39 PM, Phil S. <philipp.schum...@gmail.com> wrote:
> > Hi gophers,
>
> > now "go build" is a really neat tool but it seems like it recompiles
> > all source files everytime,
>
> This is somewhat overstating the case; for a single package, yes, it
> compiles all source files whenever it recompiles that package.  That's not
> a problem, because the go compilers are blisteringly fast.  It does not

> recompile imported packages that are up-to-date, however, *except when you
> are using local imports*. (one of the many reasons not to use them).  If

[Peter Bourgon]

"go get" also works on local packages, as a shortcut for "satisfy all my import dependencies and then go install".