unexpected fault address?

[kortschak]

I get the following error that seems to be generated during the
function shown here <https://gist.github.com/1155727>. The struct
that's being written in the loop is simple an without any references
to other locations:

type FilterHit struct {
QFrom int
QTo int
DiagIndex int
}

The only thing that I can think is causing this problem is that I have
altered runtime/malloc.goc arena size to 16LL<<33 (we have 256GB on
board)

Any suggestions?

thanks
Dan

unexpected fault address 0xffffffc0961310a0
throw: fault

[signal 0xb code=0x1 addr=0xffffffc0961310a0 pc=0x40f8ef]

runtime.throw+0x40 /usr/local/src/go/src/pkg/runtime/runtime.c:111
runtime.throw(0x564bca, 0xffffffc0961310a0)
runtime.sigpanic+0xe7 /usr/local/src/go/src/pkg/runtime/linux/thread.c:
309
runtime.sigpanic()
MHeap_FreeLocked+0x7d /usr/local/src/go/src/pkg/runtime/mheap.c:287
MHeap_FreeLocked(0x581288, 0x7fd28a949568, 0xae, 0x8)
MHeap_AllocLocked+0x1e0 /usr/local/src/go/src/pkg/runtime/mheap.c:121
MHeap_AllocLocked(0x581288, 0x1, 0x100000007, 0x417952,
0x2585fa0, ...)
runtime.MHeap_Alloc+0x57 /usr/local/src/go/src/pkg/runtime/mheap.c:61
runtime.MHeap_Alloc(0x581288, 0x1, 0x7, 0x7fd2928aeba8,
0x2585fa0, ...)
MCentral_Grow+0x81 /usr/local/src/go/src/pkg/runtime/mcentral.c:176
MCentral_Grow(0x2585fa0, 0x22)
runtime.MCentral_AllocList+0x4a /usr/local/src/go/src/pkg/runtime/
mcentral.c:46
runtime.MCentral_AllocList(0x2585fa0, 0x20, 0x7fd2928aec30,
0x5f00000002, 0xfdee7ab1ff, ...)
runtime.MCache_Alloc+0x78 /usr/local/src/go/src/pkg/runtime/mcache.c:
23
runtime.MCache_Alloc(0x7fd2928ec000, 0xfa00000007, 0x60, 0x1,
0x7fd2928aed08, ...)
runtime.mallocgc+0xf0 /usr/local/src/go/src/pkg/runtime/malloc.c:36
runtime.mallocgc(0x60, 0x100000000, 0xfd00000001, 0x41,
0x40be7a, ...)
runtime.mal+0x3f /usr/local/src/go/src/pkg/runtime/malloc.c:301
runtime.mal(0x60, 0x200000002)
runtime.new+0x24 /usr/local/src/go/src/pkg/runtime/malloc.c:308
runtime.new(0xfd00000060, 0xfa00090d50, 0xfdb8c09460,
0xfa00090d50)
gob.(*Encoder).encodeInterface+0x2c8 /usr/local/src/go/src/pkg/gob/
encode.go:456
gob.(*Encoder).encodeInterface(0xfdb8c09420, 0xfdb8c09460,
0x4add30, 0xfdf93b38b0, 0x0, ...)
gob._func_017+0x120 /usr/local/src/go/src/pkg/gob/encode.go:608
gob._func_017(0xfa0a577940, 0x475fcf, 0xfa0a5783c0,
0xfdee7ab1b0, 0xfdf93b38b0, ...)
----- goroutine created by -----
bio/morass.(*Morass).Push+0x147 /home/kortschak/BioGo/src/bio/morass/
morass.go:134

goroutine 2 [4]:
runtime.gosched+0x5c /usr/local/src/go/src/pkg/runtime/proc.c:769
runtime.gosched()
runfinq+0x50 /usr/local/src/go/src/pkg/runtime/mgc0.c:700
runfinq()
runtime.goexit /usr/local/src/go/src/pkg/runtime/proc.c:246
runtime.goexit()
----- goroutine created by -----
runtime.gc /usr/local/src/go/src/pkg/runtime/mgc0.c:559

goroutine 1 [4]:
runtime.gosched+0x5c /usr/local/src/go/src/pkg/runtime/proc.c:769
runtime.gosched()
runtime.chanrecv+0x1c8 /usr/local/src/go/src/pkg/runtime/chan.c:368
runtime.chanrecv(0x486978, 0xfa00030f60, 0x7fd2928ac880, 0x0,
0x0, ...)
runtime.chanrecv1+0x38 /usr/local/src/go/src/pkg/runtime/chan.c:417
runtime.chanrecv1(0x486978, 0xfa00030f60, 0x10000000100000,
0x7fd2928ac800)
bio/morass.(*Morass).Push+0x164 /home/kortschak/BioGo/src/bio/morass/
morass.go:135
bio/morass.(*Morass).Push(0xfa00010ee0, 0xfa0004b840,
0xfdf93b3860, 0x0, 0x0, ...)
bio/align/pals/filter.(*Filter).addHit+0xbe /home/kortschak/BioGo/src/
bio/align/pals/filter/filter.go:232
bio/align/pals/filter.(*Filter).addHit(0xfa00001870,
0x238e7be017ab1dc, 0x238e874, 0x0, 0x0, ...)
bio/align/pals/filter.(*Filter).hitTube+0xc9 /home/kortschak/BioGo/src/
bio/align/pals/filter/filter.go:189
bio/align/pals/filter.(*Filter).hitTube(0xfa00001870,
0x23f6351017ab1dc, 0x0, 0x0, 0x17ab1dd2f563baa, ...)
bio/align/pals/filter.(*Filter).commonKmer+0x138 /home/kortschak/BioGo/
src/bio/align/pals/filter/filter.go:171
bio/align/pals/filter.(*Filter).commonKmer(0xfa00001870,
0x23f635100afef4f, 0x0, 0x0, 0x1415a34f928ac9d0, ...)
bio/align/pals/filter._func_001+0xa7 /home/kortschak/BioGo/src/bio/
align/pals/filter/filter.go:87
bio/align/pals/filter._func_001(0xfa00000f20, 0xfa00000f18,
0x43b572, 0xfa0000de40, 0x1e0928a3023f6351, ...)
----- goroutine created by -----
_rt0_amd64+0xc9 /usr/local/src/go/src/pkg/runtime/amd64/asm.s:65

[Russ Cox]

Looks like a memory corruption bug.
Can you post a diff showing the changes
you have made to malloc?

Is this reproducible? If so, can you post a
short program that reproduces it?

Thanks.
Russ

[Dan Kortschak]

Here is the full diff of my install against weekly (thread is altered to
tell me what type of fault is occurring):

diff -r 508513bbf607 src/pkg/runtime/linux/thread.c
--- a/src/pkg/runtime/linux/thread.c Thu Aug 18 16:07:28 2011 +1000
+++ b/src/pkg/runtime/linux/thread.c Tue Aug 23 06:25:46 2011 +0930
@@ -300,12 +300,12 @@
case SIGBUS:
if(g->sigcode0 == BUS_ADRERR && g->sigcode1 < 0x1000)
runtime·panicstring("invalid memory address or nil pointer dereference");
- runtime·printf("unexpected fault address %p\n", g->sigcode1);
+ runtime·printf("SIGBUS: unexpected fault address %p\n", g->sigcode1);
runtime·throw("fault");
case SIGSEGV:
if((g->sigcode0 == 0 || g->sigcode0 == SEGV_MAPERR || g->sigcode0 == SEGV_ACCERR) && g->sigcode1 < 0x1000)
runtime·panicstring("invalid memory address or nil pointer dereference");
- runtime·printf("unexpected fault address %p\n", g->sigcode1);
+ runtime·printf("SIGSEGV: unexpected fault address %p\n", g->sigcode1);
runtime·throw("fault");
case SIGFPE:
switch(g->sigcode0) {
diff -r 508513bbf607 src/pkg/runtime/malloc.goc
--- a/src/pkg/runtime/malloc.goc Thu Aug 18 16:07:28 2011 +1000
+++ b/src/pkg/runtime/malloc.goc Tue Aug 23 06:25:46 2011 +0930
@@ -277,7 +277,7 @@
// Actually we reserve 17 GB (because the bitmap ends up being 1 GB)
// but it hardly matters: fc is not valid UTF-8 either, and we have to
// allocate 15 GB before we get that far.
- arena_size = 16LL<<30;
+ arena_size = 16LL<<31;
bitmap_size = arena_size / (sizeof(void*)*8/4);
p = runtime·SysReserve((void*)(0x00f8ULL<<32), bitmap_size + arena_size);
if(p == nil)

>From my tests, any arena_size >= 16LL<<31 causes this problem 100% of
the time. Getting a minimal reproduction may take a little while - the
function/package that causes it to be generated is moderately
complicated, but I suspect the reproduction will just be a matter of
writing many structs through gob.Encode().

Dan

[Dan Kortschak]

This example gives a close match though not exactly the same as
previously seen, but is probably a good first approximation because of
the minimal complexity:

package main

type d struct {
A, B, C int64
}

func main() {
a := make([]d, 1<<30)
println(a[0].A)
}

This snippet gives:

SIGSEGV: unexpected fault address 0x343f0e0
throw: fault

[signal 0xb code=0x1 addr=0x343f0e0 pc=0x406417]

panic during panic

but not the stack trace I see with my other case (maybe gob recovers and
repanics? I haven't looked at that).

Dan

On Mon, 2011-08-22 at 12:26 -0400, Russ Cox wrote:

--
Omnes mundum facimus.

Dan Kortschak <dan.ko...@adelaide.edu.au>
F9B3 3810 C4DD E214 347C B8DA D879 B7A7 EECC 5A40

[Rob 'Commander' Pike]

You're asking for 24 GB of memory. Just to be sure: is that what you want to do? That is, is your complaint that this doesn't work for you, or that the runtime complains when it shouldn't, or that that the error is poorly diagnosed?

What environment are you in? On my amd64 Mac I see:

runtime: out of memory: cannot allocate 25769803776-byte block (1048576 in use)
throw: out of memory

panic during panic

Other than the last line, this is a reasonable explanation of a problem.

-rob

[Dan Kortschak]

Yes, I'm asking for 24GB to reproduce an error I see when I am running a
process that aligns two very large DNA sequences - the situation on the
production code is more complicated than the example, but it will also
be asking for large spaces (we work with genomes - the package that
causes the error is designed to sort arbitrarily large data sets). This
is why I have altered runtime/malloc.goc to increase the area_size
variable (a suggestion from Russ) - I haven't rechecked that > 16LL<<31
gives the same area, but I suspect it will.

The issue is that I am unable to use the resources available on our
system (256GB of system memory) and this impacts on our ability to use
Go as a tool for genome analysis - something that I'm pushing in our lab
because of the developer and peer review benefits of the language and
despite some of the existing limits (notably the 31 bit index size). I'm
hoping in the not too distant future to write a paper putting Go forward
as a bioinformatics programming language to fill a niche between perl
with BioPerl and C/C++ with libraries like SeqAn - i.e. for prototyping
novel analytical algorithms in a research environment.

thanks
Dan