GC crash, SIGSEGV: segmentation violation PC=0x4071dc

[Dave Cheney]

Hello,

One of our juju packages can reliably (100% at the moment) crash the
runtime during tests.

Interestingly the PC is always the same, although the stack trace does
vary from time to time.

As reproducing this involves installing mongodb and all our juju
dependencies, I'd like to help narrow this crash down before trying to
log an issue.

If you want to try anyway, details to reproduce the issue are at the
end of the message.

Cheers

Dave

SIGSEGV: segmentation violation
PC=0x4071dc

goroutine 0 [idle]:
scanblock(0x7f42b0080000, 0x12de700)
/home/dfc/go/src/pkg/runtime/mgc0.c:969 +0x8ec fp=0x7fffb7565df8
markroot(0xc20800c900, 0x1007f4200000001)
/home/dfc/go/src/pkg/runtime/mgc0.c:1307 +0xd9 fp=0x7fffb7565e78
runtime.parfordo(0xc20800c900)
/home/dfc/go/src/pkg/runtime/parfor.c:88 +0xa3 fp=0x7fffb7565ef0
gc(0xc2089b1460)
/home/dfc/go/src/pkg/runtime/mgc0.c:2339 +0x1aa fp=0x7fffb7566018
mgc(0xc20827cb40)
/home/dfc/go/src/pkg/runtime/mgc0.c:2285 +0x2e fp=0x7fffb7566028
runtime.mcall(0x42b297)
/home/dfc/go/src/pkg/runtime/asm_amd64.s:181 +0x4b fp=0x7fffb7566038

goroutine 12757 [garbage collection]:
runtime.gc(0x7f4200000000)
/home/dfc/go/src/pkg/runtime/mgc0.c:2265 +0x1a3 fp=0xc2089b1470
runtime.mallocgc(0x1500, 0xce31e0, 0x1)
/home/dfc/go/src/pkg/runtime/malloc.goc:204 +0x1d6 fp=0xc2089b14d8
runtime.new(0xce31e0, 0x0)
/home/dfc/go/src/pkg/runtime/malloc.goc:825 +0x3b fp=0xc2089b14f8
math/rand.NewSource(0x863e227d68306861, 0x0, 0x0)
/home/dfc/go/src/pkg/math/rand/rand.go:25 +0x38 fp=0xc2089b1530
math/big.nat.probablyPrime(0xc208584870, 0x4, 0x9, 0x14, 0xda9a21)
/home/dfc/go/src/pkg/math/big/nat.go:1431 +0x5a3 fp=0xc2089b16b0
math/big.(*Int).ProbablyPrime(0xc20864b0e0, 0x14, 0x7f42b1125000)
/home/dfc/go/src/pkg/math/big/int.go:721 +0x4a fp=0xc2089b16e8
crypto/rsa.(*PrivateKey).Validate(0xc208b385a0, 0x0, 0x0)
/home/dfc/go/src/pkg/crypto/rsa/rsa.go:91 +0xd9 fp=0xc2089b17b0
crypto/x509.ParsePKCS1PrivateKey(0xc208569040, 0x13e, 0x13e,
0xc208b385a0, 0x0, ...)
/home/dfc/go/src/pkg/crypto/x509/pkcs1.go:77 +0x4b5 fp=0xc2089b18b8
crypto/tls.parsePrivateKey(0xc208569040, 0x13e, 0x13e, 0x0, 0x0, ...)
/home/dfc/go/src/pkg/crypto/tls/tls.go:259 +0x81 fp=0xc2089b1968
crypto/tls.X509KeyPair(0xc2084a5acd, 0x0, 0x33, 0xc20865b7ed, 0x0, ...)
/home/dfc/go/src/pkg/crypto/tls/tls.go:213 +0x32c fp=0xc2089b1ae8
launchpad.net/juju-core/environs/config.verifyKeyPair(0xc2084a5800,
0x2cd, 0x300, 0xc20865b600, 0x1ed, ...)
/home/dfc/src/launchpad.net/juju-core/environs/config/authkeys.go:82
+0x96 fp=0xc2089b1b78
launchpad.net/juju-core/environs/config.Validate(0xc2082f2320, 0x0, 0x0, 0x0)
/home/dfc/src/launchpad.net/juju-core/environs/config/config.go:280
+0xfa1 fp=0xc2089b1e18
launchpad.net/juju-core/environs/config.New(0x7f42b004b200,
0xc208b3ddd0, 0x7f42b004b208, 0x0, 0x0)
/home/dfc/src/launchpad.net/juju-core/environs/config/config.go:126
+0x1fc fp=0xc2089b1f80
----- stack segment boundary -----

Full log: http://paste.ubuntu.com/7196494/

To reproduce:

sudo apt-get install mongodb-server
go get launchpad.net/juju-core/...
go test launchpad.net/juju-core/worker/uniter

[Aram Hăvărneanu]

Seems similar to https://code.google.com/p/go/issues/detail?id=7554.

--
Aram Hăvărneanu

[Russ Cox]

Except that it is a crash during garbage collection I don't see an obvious similarity.

Tip garbage collection is kind of buggy recently. It might be best to wait until next week.

I'd like to know why the crash handler is not printing the fault address (g->sigcode1).

[Dave Cheney]

Thanks Russ, I don't expect tip to be usable, but I wanted to get this
issue on the radar

% go version
go version devel +97ba656ccd5d Thu Apr 03 11:34:31 2014 +1100 linux/amd64

Is there anything I can do to get more information for you ?
GOTRACEBACK=crash maybe ?

[Russ Cox]

Please remove the 

if(!runtime·canpanic(gp))

goto Throw;

from your local copy of runtime/signal_amd64x.c and make the crash happen again. That will give us more information.

That call to canpanic is in the wrong place.

[Dave Cheney]

With the canpanic removed

unexpected fault address 0x0
fatal error: fault
[signal 0xb code=0x80 addr=0x0 pc=0x4071dc]

runtime stack:
runtime.throw(0x12db717)
/home/dfc/go/src/pkg/runtime/panic.c:520 +0x69 fp=0x7f0b28ff88c0
runtime.sigpanic()
/home/dfc/go/src/pkg/runtime/os_linux.c:237 +0x11b fp=0x7f0b28ff88d8
scanblock(0x7f0b30cda000, 0x7f0b28ff8c00)
/home/dfc/go/src/pkg/runtime/mgc0.c:969 +0x8ec fp=0x7f0b28ff8c00
markroot(0xc20800c900, 0x1007f0b00000075)
/home/dfc/go/src/pkg/runtime/mgc0.c:1307 +0xd9 fp=0x7f0b28ff8c80
runtime.parfordo(0xc20800c900)
/home/dfc/go/src/pkg/runtime/parfor.c:88 +0xa3 fp=0x7f0b28ff8cf8
gc(0xc2089ad738)
/home/dfc/go/src/pkg/runtime/mgc0.c:2339 +0x1aa fp=0x7f0b28ff8e20
mgc(0xc20837bc20)
/home/dfc/go/src/pkg/runtime/mgc0.c:2285 +0x2e fp=0x7f0b28ff8e30
runtime.mcall(0xa0f4dc)
/home/dfc/go/src/pkg/runtime/asm_amd64.s:181 +0x4b fp=0x7f0b28ff8e40

[Dave Cheney]

Thanks for submitting 83980043 overnight

The results this morning with that change applied are as expected

lucky(~/src/launchpad.net/juju-core) % go test ./worker/uniter
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x80 addr=0x0 pc=0x4071cc]

runtime stack:
runtime.throw(0x12daa45)
/home/dfc/go/src/pkg/runtime/panic.c:520 +0x69 fp=0x7faf167fb8c0
runtime.sigpanic()
/home/dfc/go/src/pkg/runtime/os_linux.c:222 +0x3d fp=0x7faf167fb8d8
scanblock(0x7faf2e439000, 0x7faf167fbc00)
/home/dfc/go/src/pkg/runtime/mgc0.c:969 +0x8ec fp=0x7faf167fbc00
markroot(0xc20800c900, 0x1007faf00000098)
/home/dfc/go/src/pkg/runtime/mgc0.c:1307 +0xd9 fp=0x7faf167fbc80
runtime.parfordo(0xc20800c900)
/home/dfc/go/src/pkg/runtime/parfor.c:88 +0xa3 fp=0x7faf167fbcf8
gc(0xc208ad3738)
/home/dfc/go/src/pkg/runtime/mgc0.c:2339 +0x1aa fp=0x7faf167fbe20
mgc(0xc2085f4120)
/home/dfc/go/src/pkg/runtime/mgc0.c:2285 +0x2e fp=0x7faf167fbe30
runtime.mcall(0xa1131c)
/home/dfc/go/src/pkg/runtime/asm_amd64.s:181 +0x4b fp=0x7faf167fbe40

[Russ Cox]

Thanks. I will see if I can reproduce it on a Linux box next week. 

[Dave Cheney]

I am very sorry that the test requires such a large harness to work.
Please let me know if there is anything I can do to assist.

[Russ Cox]

Great bug, thanks. I reproduced this using a fresh Ubuntu VM. I created issue 7725 and I am working on a fix.

[Dave Cheney]

Thanks Russ. I like that Juju can bring you such interesting bugs, but
I'm happier still that you could reproduce it :)