incremental build

[rneer...@gmail.com]

It seems that "go build -o foo foo.go" builds foo each time I issue the cmd
Is there to perform incremental build if my source or dependent packages have not changed ?

If this is a duplicate question, please point me to earlier posts.
I found some posts that were concerned with building packages, while I am concerned with building the final executable.

 thanks Neeraj

[Dave Cheney]

Use go install. Go build builds, then discards any intermediary results (except for final executables if applicable)

[rneer...@gmail.com]

Thanks for responding ,Dave.
Go install doesn't seem to take output -o, like go build.
I got the impression that it was for installing pkgs, not building final exe.
Is there a way to have incremental builds for final exe?

[Carlos Castillo]

If you run "go test -i" it will run go install on dependencies only, then "go build" commands will only build the final package until the dependencies get updated.

[rneer...@gmail.com]

Hi Carlos,

I am trying to solve a problem but I would also like to fix my understanding in the process.
Stating my understanding below with example. Please correct as needed.
Lets take a simple example of
      a) 1 package named pkg1, contained in file pkg1.go
      b) 1 final go program exe1.go that would be built into an exe and imports pkg1
   1. The build always rebuilds  (go build exe1.go)
   2. The install builds and installs incrementally if plg1.go has changed. Works on pkg1.go (go install pkg1)
   3. once packages are installed with go test -i , go build will not rebuild exe1 unless exe1.go or installed pkg is changed.
If that is correct, I don't see a use for go test. Why not use go install as intermediate step and then go build?
What is the difference between the 2 methods?

thanks
Neeraj

[Kevin Malachowski]

go install will cache any packages it compiled on your local file system. That way the next time you're compiling and you import the package it only has to read the library file (*.a files) to get all of the data regarding types, funcs, etc in that package. If you don't change your external libraries often then a lot of the work will be done once and reused often. For example: if the standard library weren't cached Go would compile much more slowly.

[rneer...@gmail.com]

 I can confirm the incremental build is working for me at package level but still no success at exe level.
Please advise if I am using this correctly and my expectations are right.

Here's my setup :
   pkg/
     linux_amd64/
         pkg.a
   src/
     pkg/
       pkg1.go
     prog/
       exe.go
       exe1.go

go install pkg   # builds pkg.a
go install pkg   # doesn't build pkg.a

go build -o src/prog/exe1 src/prog/exe.go
go build -o src/prog/exe1 src/prog/exe.go ## this is built everytime I issue the build
   I would like to build exe1 only if exe.go changed.

[Julie Ann Lawson]

On Saturday, October 25, 2014 4:04:43 PM UTC-2, rneer...@gmail.com wrote:

go build -o src/prog/exe1 src/prog/exe.go
go build -o src/prog/exe1 src/prog/exe.go ## this is built everytime I issue the build
   I would like to build exe1 only if exe.go changed.

I'm having trouble understanding what problem you are trying to solve here. If you run go build on a source file, it will build that source into a binary. That is what the go build command _does_.

Why do you need to run go build multiple times if you are not changing the source code? Are you cooking your own build/deploy system? if so, is building that source file taking too long for you? Go has been designed to be super fast to compile. This is not C++.

Maybe if you explain what is the actual problem you are trying to solve, we might be able to provide better solutions.

Jules.

[Dave Cheney]

Just always use go install, if your source is properly laid out, ie somecmd/main.go, then the binary will be installed as somecmd.

> --
> You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/57Cwx7i5g7U/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Lars Seipel]

On Sat, Oct 25, 2014 at 11:04:43AM -0700, rneer...@gmail.com wrote:
> go build -o src/prog/exe1 src/prog/exe.go
> go build -o src/prog/exe1 src/prog/exe.go ## this is built everytime I
> issue the build
> I would like to build exe1 only if exe.go changed.

Then don't use go build. Use go install. It will build the command (only
when necessary) and put the result to $GOPATH/bin (if your GOPATH has
more than one element, it takes the first one).

If you want the executable to go somewhere else than $GOPATH/bin, it's
possible by setting GOBIN in the environment.

[Carlos Castillo]

Go build can't know with any degree of certainty if the destination file is a previously compiled version of the same program, therefore it has to recompile. Otherwise the following sequence would result in no work done, and no working executable despite compilation "success":

vi foo.go

rm -f foo

touch foo

go build -o foo foo.go # foo.go is older than foo, should we compile?

Go install doesn't take a "-o" option, because the name of the output file, and where it will be located is determined automatically from the package path and the package's type (library vs. binary). With this restriction in place, it can be determined with much greater confidence that the output file is the product of the given input, and so testing the timestamps to determine if compilation can be avoided makes sense.

I also would recommend against using a command like "go build source.go". It is not specifying a package, and as a result is much less useful (as well as requiring more typing).

In general, even when using go build, each executable should be in its own package / folder, even if its code is in a single file. The install location of the executable is determined from the package path automatically that way (provided it's in the GOPATH), and that name will be the same as the executable installed by "go install" or "go get" (ie: the name of the containing folder).

For example, if prog1 is defined in $GOPATH/src/prog1/main.go, then you can compile it using:

•  "go build" if current directory is $GOPATH/src/prog1
•  "go build prog1"
•  or use "go install" versions instead to install the results (to $GOPATH/bin)

Important points:

•  don't build go executables that way (you shouldn't be using -o except in special cases)
•  your code arrangement is bad (each executable should be in it's own directory)
•  dependencies are only rebuilt if they are out of date (even when using go build)
•  "go build" will always rebuild the current package
•  "go build" will throw away any dependencies it (re-)compiles, so out of date dependencies will be recompiled every time

Also, in case you didn't know, the "-v" flag to go install, build, get, and test all print out a line for each package where work was done. It prints out the package path when a package is (re-)compiled.

--
Carlos Castillo

[rneer...@gmail.com]

Hi Dave, Lars, Carlos,

go install is the thing I was looking for. output goes to bin  - that's fine. I was putting it in current dir as I didn't know better.
and "go install -v prog" is added bonus that is really useful.

Is there any way to work around the 1 exe per dir restriction ?
I would prefer to dump all these .go programs into single dir and name the output same as filename without .go extension in bin dir.
I have  50/100 .go programs andIt'll be very cumbersome to change dir everytime to work on these utilities.

Hi Julie,

The build is fast with golang - takes 2 sec for the file I am currently working with.
But with 30 files, it adds up to 1 min.
I would like to change many files in the dir then issue build all to compile all that changed.
build install helps greatly in such situations.

[Dave Cheney]

On 26 Oct 2014, at 10:30, rneer...@gmail.com wrote:

Hi Dave, Lars, Carlos,

go install is the thing I was looking for. output goes to bin  - that's fine. I was putting it in current dir as I didn't know better.
and "go install -v prog" is added bonus that is really useful.

Is there any way to work around the 1 exe per dir restriction ?

No, it is by design. If you fight it, I'll only hurt more. 

I would prefer to dump all these .go programs into single dir and name the output same as filename without .go extension in bin dir.

That is not considered idiomatic, instead consider

$REPO/cmd/

                  Program1

                  Program2

                  ... Etc

I have  50/100 .go programs andIt'll be very cumbersome to change dir everytime to work on these utilities.

I think you're not taking enough advantage of the tools,

gp install github.com/$YOU/$REPO/...  (The ellipses are to be typed as presented, they are the go tools globig character)

This will rebuild any program that has changed, or is affected by a dependency and place it in $GOPATH/bin

Hi Julie,

The build is fast with golang - takes 2 sec for the file I am currently working with.
But with 30 files, it adds up to 1 min.
I would like to change many files in the dir then issue build all to compile all that changed.
build install helps greatly in such situations.

[rneer...@gmail.com]

Hi Dave,

Let me try to understand a bit more.
1.  That is not considered idiomatic, instead consider

      $REPO/cmd/

                  Program1

                  Program2

                  ... Etc
   a) What is program1, program2 here ? are these directories under which I put the exe.go and exe1.go ? or files in themselves Program1.go , Program2.go?
   b) I am not sure I understand the cmd dir. I am aware of src/pkg/bin dir so far.
            Where does that fit in the heirarchy of go dir structs ?
            Or are you proposing this out of scope due to the way I put my dir struct?

Would it be possible to for you to point out which dir/files in my current setup need to move to new dir/or be renamed.
   pkg/                           ## package dir created by go install
     linux_amd64/
         mypkg.a
   src/                           ## my source dir where I edit the files
     mypkg/                      ## my package source
       pkg1.go
     prog/                          ## my exe source that are not building with go install idiom
       exe.go
       exe1.go
   bin/                           ## exe dir creted by go install
      prog

2.go install github.com/$YOU/$REPO/...  (The ellipses are to be typed as presented, they are the go tools globig character)

   This will rebuild any program that has changed, or is affected by a dependency and place it in $GOPATH/bin

   If I understand this correctly, this will look for all go files under $REPO and build them. if it is a package, output goes to pkg dir and if it is exe, output goes to bin dir.
   That is a nice feature that I didn't know about.
   c) Will non-go files under the dir structure confuse the go install ?

3. Also didn't understand why mentioned it would hurt to fight the dir struct issue?

[Dave Cheney]

On 26 Oct 2014, at 13:56, rneer...@gmail.com wrote:

Hi Dave,

Let me try to understand a bit more.
1.  That is not considered idiomatic, instead consider

      $REPO/cmd/

                  Program1

                  Program2

                  ... Etc
   a) What is program1, program2 here ? are these directories under which I put the exe.go and exe1.go ? or files in themselves Program1.go , Program2.go?

Program1 and 2 are e names of packages, which contain files who's package declaration places them in main, therefor they are commands 

The go.tools repo gives a good example to follow.  https://code.google.com/p/go/source/browse?repo=tools

   b) I am not sure I understand the cmd dir. I am aware of src/pkg/bin dir so far.
            Where does that fit in the heirarchy of go dir structs ?
            Or are you proposing this out of scope due to the way I put my dir struct?

Would it be possible to for you to point out which dir/files in my current setup need to move to new dir/or be renamed.
   pkg/                           ## package dir created by go install
     linux_amd64/
         mypkg.a
   src/                           ## my source dir where I edit the files
     mypkg/                      ## my package source
       pkg1.go
     prog/                          ## my exe source that are not building with go install idiom
       exe.go
       exe1.go

Rename prog to cmd, create a subdirectory for each exeN.go, name that subdirectory for the name of the final binary you want. 

   bin/                           ## exe dir creted by go install
      prog

2.go install github.com/$YOU/$REPO/...  (The ellipses are to be typed as presented, they are the go tools globig character)
   This will rebuild any program that has changed, or is affected by a dependency and place it in $GOPATH/bin

   If I understand this correctly, this will look for all go files under $REPO and build them. if it is a package, output goes to pkg dir and if it is exe, output goes to bin dir.
   That is a nice feature that I didn't know about.
   c) Will non-go files under the dir structure confuse the go install ?

Depends, if they look like source code files, it'll try to compile them. As a workaround directories that start with an underscore or a period, or the word "testdata" are ignored.

3. Also didn't understand why mentioned it would hurt to fight the dir struct issue?

[rneer...@gmail.com]

Hi Dave,

I think we can close this thread as my oriiginal issue of incremental build has been resolved .
I'll ponder over the dir structure and may start a new post about it if needed.

thanks
Neeraj