Go 1.6.2 fails test when running on a local disk, succeeds when running on NFS

[Steven Kelso]

My normal build process is to create a temporary build area under /tmp, copy my source files there and run the build script and install from there.  Up until now this has worked 100% of the time.  However, I see the following behavior when compiling up go version 1.6.2 using 1.4 as a bootstrap.  My system information is as follows:

[kelso@stack-build-2 go]$ uname -rpo

2.6.32-504.12.2.el6.x86_64 x86_64 GNU/Linux

[kelso@stack-build-2 go]$ /lib64/libc.so.6 

GNU C Library stable release version 2.12, by Roland McGrath et al.

Copyright (C) 2010 Free Software Foundation, Inc.

This is free software; see the source for copying conditions.

There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A

PARTICULAR PURPOSE.

Compiled by GNU CC version 4.4.7 20120313 (Red Hat 4.4.7-16).

Compiled on a Linux 2.6.32 system on 2016-02-16.

Available extensions:

The C stubs add-on version 2.1.2.

crypt add-on version 2.1 by Michael Glad and others

GNU Libidn by Simon Josefsson

Native POSIX Threads Library by Ulrich Drepper et al

BIND-8.2.3-T5B

RT using linux kernel aio

libc ABIs: UNIQUE IFUNC

For bug reporting instructions, please see:

<http://www.gnu.org/software/libc/bugs.html>.

To get around potential issues with environment etc, I warp the complete build process in a script and run it with 'env -i':

#!/bin/bash

export CCACHE_DISABLE=1

export GOROOT_BOOTSTRAP="$(pwd)/go1.4";

export GOROOT="$(pwd)/go1.4"

export PATH="${GOROOT}/bin:$PATH"

export GOROOT_FINAL=/tmp/steve

export HOME=/tmp

printenv

cd go1.6.2/src

./all.bash

When I run the build/test script on an NFS mounted directory I see the following completing successfully:

[kelso@stack-build-2 go]$ df -h .

Filesystem            Size  Used Avail Use% Mounted on

filer2a:/vol/home_1/aitken

                       20G   13G  8.0G  61% /home/aitken

[kelso@stack-build-2 go]$ pwd

/home/aitken/wc/builds/go

[kelso@stack-build-2 go]$ env -i ./wrap.sh

 [ deleted ]

testing: warning: no tests to run

PASS

ok      _/home/steve/wc/builds/go/go1.6.2/test/bench/go1        2.830s

##### ../test

##### API check

Go version is "go1.6.2", ignoring -next /home/steve/wc/builds/go/go1.6.2/api/next.txt

ALL TESTS PASSED

---

Installed Go for linux/amd64 in /home/steve/wc/builds/go/go1.6.2

Installed commands in /home/steve/wc/builds/go/go1.6.2/bin

The binaries expect /home/steve/wc/builds/go/go1.6.2 to be copied or moved to /tmp/steve

When I run exactly the same on a local disk, I see the following errors:

[kelso@stack-build-2 build]$ df -h .

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/sys0-sys.tmp.var

                      7.8G  1.3G  6.1G  18% /var/tmp

[kelso@stack-build-2 build]$ pwd

/var/tmp/build

[kelso@stack-build-2 build]$ env -i ./wrap.sh

  [ deleted ]

testing: warning: no tests to run

PASS

ok      _/var/tmp/build/go1.6.2/test/bench/go1  6.475s

##### ../test

# go run run.go -- fixedbugs/bug142.go

exit status 1

runtime: failed to create new OS thread (have 5 already; errno=11)

fatal error: newosproc

runtime stack:

runtime.throw(0x4755a0, 0x9)

        /var/tmp/build/go1.6.2/src/runtime/panic.go:547 +0x90

runtime.newosproc(0xc820060000, 0xc820071fc0)

        /var/tmp/build/go1.6.2/src/runtime/os1_linux.go:149 +0x18c

runtime.newm(0x489578, 0xc820018a00)

        /var/tmp/build/go1.6.2/src/runtime/proc.go:1516 +0x135

runtime.startm(0xc820018a00, 0x100000001)

        /var/tmp/build/go1.6.2/src/runtime/proc.go:1586 +0x122

runtime.wakep()

        /var/tmp/build/go1.6.2/src/runtime/proc.go:1667 +0x59

runtime.resetspinning()

        /var/tmp/build/go1.6.2/src/runtime/proc.go:1994 +0xcf

runtime.schedule()

        /var/tmp/build/go1.6.2/src/runtime/proc.go:2082 +0x27b

runtime.mstart1()

        /var/tmp/build/go1.6.2/src/runtime/proc.go:1108 +0x118

runtime.mstart()

        /var/tmp/build/go1.6.2/src/runtime/proc.go:1068 +0x72

This is 100% reproducible whenever I run it on a local disk.  It always fails, and always succeeds when I run it on my NFS mounted directory.

I have tried under /tmp and  /var/tmp.  This machine happens to be a virtual machine, however the same behavior happens on a the underlying physical machine as well.

Yes, I could always use a NFS mounted directory for building this but it points to something else being at fault and and I'm sort of at a brick wall to think what it could be.  Does anyone have and advice?

Thanks,

S.

[Brad Fitzpatrick]

What's your ulimit?

"failed to create new OS thread (have 5 already" seems surprisingly low.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steven Kelso]

[ repost to group after I only replied personally]
After reading your message, my ulimit -u was 1024. I increased it to
4096 and now both the local build and the NFS build work.
So I have a workaround, but have no idea why increasing ulimit effects
a local disk build versus a build on an NFS partition. I can't work
out why would the number of processes be effected by the disk type on
which the binary is running?

[gshim...@gmail.com]

I hit this bug too on RHEL 6.6 which had a low default ulimit https://github.com/golang/go/issues/13968
While investigating this bug I found out that this all started after revisions d513ee7 and f034ee8 which make many tests to run in parallel.
Could it be that on NFS test executables just don't load fast enough to overflow user processes limit?

пятница, 15 июля 2016 г., 10:48:40 UTC+3 пользователь Steven Kelso написал: