system calls made by go install ./...

[Albert Strasheim]

Howdy

I'm running Go at tip. I have a $GOPATH with about 2000 .go files in $GOPATH/src.

$GOPATH also contains other stuff in directories at the same level as src: docs, some Java code, a pkg directory, etc. The total is about 4000 files.

Now I do:

$ cd $GOPATH && strace -c -q -e stat,lstat -f go install ./...

% time     seconds  usecs/call     calls    errors syscall

------ ----------- ----------- --------- --------- ----------------

 67.16    0.000906           0     39027       918 stat

 32.84    0.000443           0     37232           lstat

------ ----------- ----------- --------- --------- ----------------

100.00    0.001349                 76259       918 total

$ cd $GOPATH/src && strace -c -q -e stat,lstat -f go install ./...

% time     seconds  usecs/call     calls    errors syscall

------ ----------- ----------- --------- --------- ----------------

 69.54    0.000662           0     39027       918 stat

 30.46    0.000290           0     26999           lstat

------ ----------- ----------- --------- --------- ----------------

100.00    0.000952                 66026       918 total

With everything in the page cache:

$ for i in `seq 1 20`; do cd $GOPATH && time go install ./...; done 2>&1 | grep ^real | sort -n

real 0m0.453s

$ for i in `seq 1 20`; do cd $GOPATH/src && time go install ./...; done 2>&1 | grep ^real | sort -n

real 0m0.375s

Those extra lstats make the build about 20% slower. Other people that mix in lots of non-Go files into a directory in $GOPATH will probably have it worse.

Why does the go tool bother poking around in directories under $GOPATH that isn't src?

Thanks!

Cheers

Albert

[Andrew Gerrand]

On 5 April 2013 07:05, Albert Strasheim <ful...@gmail.com> wrote:

Why does the go tool bother poking around in directories under $GOPATH that isn't src?

To see if the binaries have already been built, and if they are fresher than the source code.

Andrew

[Albert Strasheim]

Hello

Thanks. It's not just going into bin though. It's also traversing my doc and java directories.

Is the go tool trying to find sources for commands that are in $GOPATH but don't live under $GOPATH/src?

Regards

Albert

[Andrew Gerrand]

On 5 April 2013 08:10, Albert Strasheim <ful...@gmail.com> wrote:

Thanks. It's not just going into bin though. It's also traversing my doc and java directories.

That is puzzling. I'm not sure why it would be doing that. It should only be looking inside the src, pkg, and bin directories of your workspace.

Andrew

[Albert Strasheim]

Hello

It's easy to test:

$ mkdir k

$ cd k

$ mkdir -p src/foo

$ echo "package main;func main(){}" > src/foo/foo.go

$ mkdir bar

$ mkdir baz

$ touch bar/bar

$ touch baz/bar

$ GOPATH=`pwd` strace -f -q -e lstat,stat -ostrace.log go install ./...

$ grep baz strace.log

20152 lstat("./baz", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0

20152 lstat(".//baz", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0

20152 lstat("baz", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0

20152 lstat("baz", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0

20152 lstat("baz/bar", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0

20152 lstat("baz/bar", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0

I can file an issue if you don't have time to poke at this in the pre-Go 1.1 madness.

Cheers

Albert

[Andrew Gerrand]

On 5 April 2013 08:42, Albert Strasheim <ful...@gmail.com> wrote:

I can file an issue if you don't have time to poke at this in the pre-Go 1.1 madness.

Please do so.

[Dave Cheney]

I think we hit something like this before in Juju, and had a fix
applied. At the time, the build time appeared to be relative to the
number of files in you $GOPATH/src, even if you were building a
specific package. At the time I had a lot of c source in my
$GOPATH/src and found moving it out caused a significant drop in build
time.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Gustavo Niemeyer]

The fix was never applied. I submitted something, had partial
approval, but then Russ suggested a different approach, which caught
my holidays and felt through the cracks.

I'll revive the patch.

--

gustavo @ http://niemeyer.net

[Albert Strasheim]

Hello

On Friday, April 5, 2013 5:09:53 AM UTC+2, Gustavo Niemeyer wrote:

The fix was never applied. I submitted something, had partial
approval, but then Russ suggested a different approach, which caught
my holidays and felt through the cracks.
I'll revive the patch.

Was this the patch?

https://codereview.appspot.com/6337043/

As I mentioned in my comment there, it doesn't seem fixed yet.

On Thu, Apr 4, 2013 at 7:55 PM, Dave Cheney <da...@cheney.net> wrote:
> I think we hit something like this before in Juju, and had a fix
> applied. At the time, the build time appeared to be relative to the
> number of files in you $GOPATH/src, even if you were building a
> specific package. At the time I had a lot of c source in my
> $GOPATH/src and found moving it out caused a significant drop in build
> time.
> On Fri, Apr 5, 2013 at 8:44 AM, Andrew Gerrand <a...@golang.org> wrote:
>> On 5 April 2013 08:42, Albert Strasheim <ful...@gmail.com> wrote:
>>> I can file an issue if you don't have time to poke at this in the pre-Go
>>> 1.1 madness.
>> Please do so.

I filed: https://code.google.com/p/go/issues/detail?id=5214

Cheers

Albert