Crashing at runtime.sigpanic() during runtime.mapassign() in highly concurrent map writing situation?

[ma...@crowdmob.com]

Hey there,

I'm running a go webserver on EC2 xlarges, and getting "fatal error: fault" crashes when I write to a map ~800 times a second across GOMAXPROCS=4

I have a map:

  MyMap map[string]int64

That I try to increment across a ~10k keys, many times a second:

  key := "a|b|c|d|e|f" // this might be a random collection of six words piped together

  MyMap[key] += int64(1)

After being run for less than 5 minutes, the app keeps crashing with:

fatal error: fault

[signal 0xb code=0x80 addr=0x0 pc=0x406448]

goroutine 3480 [running]:

[fp=0x7f622e18dde8] runtime.throw(0xacbd97)

/usr/local/go/src/pkg/runtime/panic.c:473 +0x67

[fp=0x7f622e18de00] runtime.sigpanic()

/usr/local/go/src/pkg/runtime/os_linux.c:239 +0xe7

[fp=0x7f622e18deb0] evacuate(0x64e580, 0xc2006a1800, 0xf)

/usr/local/go/src/pkg/runtime/hashmap.c:-334 +0x168

[fp=0x7f622e18ded8] grow_work(0x64e580, 0xc2006a1800, 0x76c)

/usr/local/go/src/pkg/runtime/hashmap.c:-219 +0x83

[fp=0x7f622e18df70] hash_insert(0x64e580, 0xc2006a1800, 0x7f622e18e148, 0x7f622e18e158)

/usr/local/go/src/pkg/runtime/hashmap.c:607 +0x11c

[fp=0x7f622e18df98] runtime.mapassign(0x64e580, 0xc2006a1800, 0x7f622e18e148, 0x7f622e18e158)

/usr/local/go/src/pkg/runtime/hashmap.c:1272 +0x97

----- stack segment boundary -----

[fp=0x7f622e18e138] runtime.mapassign1()

/usr/local/go/src/pkg/runtime/hashmap.c:1301 +0x7d

[fp=0x7f622e18e208] _/srv/www/realtimebidder/releases/20130808181954/models.(*InmemoryStat).IncrementContextTargets(0xc2000dc300, 0x734980, 0x5, 0xc2146b7b00, 0x11, ...)

/srv/www/realtimebidder/releases/20130808181954/models/inmemory_stat.go:30 +0x278

[fp=0x7f622e18ef98] _/srv/www/realtimebidder/releases/20130808181954/controllers.BidRequestsControllerDefinition.Create(0xc2000bda80, 0xc200714d00, 0xc2146338c0, 0xc212b45b60)

/srv/www/realtimebidder/releases/20130808181954/controllers/bid_requests_controller.go:109 +0xb7e

----- stack segment boundary -----

[fp=0x7f622e652d70] _/srv/www/realtimebidder/releases/20130808181954/controllers.BidRequestsControllerDefinition.Create·i(0x15, 0xc2000bda80, 0xc200714d00, 0xc2146338c0, 0xc212b45b60, ...)

/srv/www/realtimebidder/releases/20130808181954/controllers/bid_requests_controller.go:0 +0x4c

[fp=0x7f622e652db8] github.com/crowdmob/nutty.func·003(0xc200714d00, 0xc2146338c0, 0xc212b45b60)

/opt/aws/opsworks/releases/20130806085134_135/src/github.com/crowdmob/nutty/router.go:60 +0x4b8

[fp=0x7f622e652dd8] net/http.HandlerFunc.ServeHTTP(0xc2000cb580, 0xc200714d00, 0xc2146338c0, 0xc212b45b60)

/usr/local/go/src/pkg/net/http/server.go:1149 +0x3e

[fp=0x7f622e652e10] net/http.(*ServeMux).ServeHTTP(0xc20009b540, 0xc200714d00, 0xc2146338c0, 0xc212b45b60)

/usr/local/go/src/pkg/net/http/server.go:1416 +0x11d

[fp=0x7f622e652e50] net/http.serverHandler.ServeHTTP(0xc200718460, 0xc200714d00, 0xc2146338c0, 0xc212b45b60)

/usr/local/go/src/pkg/net/http/server.go:1517 +0x16c

[fp=0x7f622e652fb0] net/http.(*conn).serve(0xc2086a8000)

/usr/local/go/src/pkg/net/http/server.go:1096 +0x765

[fp=0x7f622e652fb8] runtime.goexit()

/usr/local/go/src/pkg/runtime/proc.c:1223

created by net/http.(*Server).Serve

/usr/local/go/src/pkg/net/http/server.go:1564 +0x266

I'm not sure where to look to start identifying the reason behind the crashes.  Does anyone have any ideas on where to look?

Thanks in advance!  Here's some environment info too:

$ go version

go version go1.1.1 linux/amd64

$ uname -a

Linux realtimebidder-nutty-kafka1 3.2.0-51-virtual #77-Ubuntu SMP Wed Jul 24 20:38:32 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Best,

Matt

 

[Brad Fitzpatrick]

You can't access a map from multiple goroutines without locks.

Use the race detector and find your bug.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Matthew Kane]

Are you adding new keys to a map without any synchronization? That's unsafe.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
matt kane
twitter: the_real_mkb / nynexrepublic
http://hydrogenproject.com

[Matthew Moore]

I am not locking.  Do I need to only lock around *adding* keys, or also writing the value?

Best,

Matt

--

Matthew Moore

Co-Founder & CTO, CrowdMob Inc.

Mobile: (650) 888-5962

Need to schedule a meeting?  Invite me via Google Calendar!  ma...@crowdmob.com

[Michael Jones]

Maps are not concurrency-safe. You've demonstrated that. ;-)

Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1 650-335-5765

[Matthew Kane]

You should lock every access. How do you know that the map isn't
growing as you read or write?

[ma...@crowdmob.com]

Locked solved my problem -- thanks guys.  Apologies for the n00bism ;-)