Concurrent Map Access Panic

52 views
Skip to first unread message

Dave Drake

unread,
Oct 16, 2017, 2:21:52 PM10/16/17
to gowebuitoolkit
Hi,

I'm seeing concurrent map iteration and map write panic. The top of the stack trace points to renderAttrs in style.go at line 810. The s.attrs map certainly doesn't appear to be concurrent protected at this level.

Any idea why this is happening and how to protect against it? I can't figure out exactly where else this is happening. 

Other gwu references at top of goroutine stacks:

sessCleaner - server.go:397 (time.Sleep)

access - session.go 229 (this is for acquiring a lock)

I've got timers running on pages that fire refreshes, perhaps this is related? But at the top level I don't see any other mentions to gwu where the s.attrs map is involved in the stack trace other than that first one? 

Thanks,
Dave



András Belicza

unread,
Oct 17, 2017, 6:55:22 AM10/17/17
to Dave Drake, gowebuitoolkit
Are you using the latest version? Latest v1.3.0 fixed a number of data race issues: https://github.com/icza/gowut/releases/tag/v1.3.0.

Locking is done at the end of Server.serveHTTP() method, using the session's mutex, so this shouldn't happen. Can you provide more details?

Dave Drake

unread,
Oct 18, 2017, 3:59:45 PM10/18/17
to András Belicza, gowebuitoolkit
We were not at the latest version, and I've updated now so we'll see if this happens again. Looking at the stack again I failed to mention a stack mention to gwu.(*serverImpl).Start(). Below is the stack involving gowut, starting from the top:

fatal error: concurrent map iteration and map write

goroutine 146 [running]:
runtime.throw(0xa8a43d, 0x26)
/usr/lib/go/src/runtime/panic.go:596 +0x95 fp=0xc420305738 sp=0xc420305718
runtime.mapiternext(0xc420305830)
/usr/lib/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc4203057e8 sp=0xc420305738
/src/github.com/icza/gowut/gwu/style.go:810 +0x180 fp=0xc4203058a0 sp=0xc4203057e8
/src/github.com/icza/gowut/gwu/style.go:791 +0xb5 fp=0xc4203058e8 sp=0xc4203058a0
/src/github.com/icza/gowut/gwu/comp.go:225 +0x10b fp=0xc420305998 sp=0xc4203058e8
/src/github.com/icza/gowut/gwu/label.go:46 +0x7d fp=0xc4203059e0 sp=0xc420305998
github.com/icza/gowut/gwu.(*serverImpl).renderComp(0xc420160dc0, 0xd79340, 0xc4202d2000, 0xd6be00, 0xc4245cab60, 0xc4246bb800)
/src/github.com/icza/gowut/gwu/server.go:733 +0x208 fp=0xc420305ad8 sp=0xc4203059e0
/src/github.com/icza/gowut/gwu/server.go:649 +0x535 fp=0xc420305c78 sp=0xc420305ad8
/src/github.com/icza/gowut/gwu/server_start.go:49 +0x48 fp=0xc420305ca8 sp=0xc420305c78
net/http.HandlerFunc.ServeHTTP(0xc420212010, 0xd6be00, 0xc4245cab60, 0xc4246bb800)
/usr/lib/go/src/net/http/server.go:1942 +0x44 fp=0xc420305cd0 sp=0xc420305ca8
net/http.(*ServeMux).ServeHTTP(0xdb1200, 0xd6be00, 0xc4245cab60, 0xc4246bb800)
/usr/lib/go/src/net/http/server.go:2238 +0x130 fp=0xc420305d10 sp=0xc420305cd0
net/http.serverHandler.ServeHTTP(0xc42028a0b0, 0xd6be00, 0xc4245cab60, 0xc4246bb800)
/usr/lib/go/src/net/http/server.go:2568 +0x92 fp=0xc420305d58 sp=0xc420305d10
net/http.(*conn).serve(0xc42047e000, 0xd6fc00, 0xc42048c040)
/usr/lib/go/src/net/http/server.go:1825 +0x612 fp=0xc420305fc8 sp=0xc420305d58
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420305fd0 sp=0xc420305fc8
created by net/http.(*Server).Serve
/usr/lib/go/src/net/http/server.go:2668 +0x2ce


goroutine 10 [IO wait, 9 minutes]:
net.runtime_pollWait(0x7f8674daaf00, 0x72, 0x0)
/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420230228, 0x72, 0x0, 0xc425040b20)
/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420230228, 0xffffffffffffffff, 0x0)
/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202301c0, 0x0, 0xd65980, 0xc425040b20)
/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42028e050, 0xc420395de0, 0x99d440, 0xffffffffffffffff)
/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42028e050, 0xc42021ecd8, 0xc42021ece0, 0xc42021ecd0)
/usr/lib/go/src/net/tcpsock.go:215 +0x49
net/http.tcpKeepAliveListener.Accept(0xc42028e050, 0xa9e5e8, 0xc420395d60, 0xd6fcc0, 0xc4202729f0)
/usr/lib/go/src/net/http/server.go:3044 +0x2f
net/http.(*Server).Serve(0xc42028a0b0, 0xd6ef80, 0xc42028e050, 0x0, 0x0)
/usr/lib/go/src/net/http/server.go:2643 +0x228
net/http.(*Server).ListenAndServe(0xc42028a0b0, 0xc42028a0b0, 0xc42021edf8)
/usr/lib/go/src/net/http/server.go:2585 +0xb0
net/http.ListenAndServe(0xc4201aea50, 0xe, 0x0, 0x0, 0x9, 0xc42024e080)
/usr/lib/go/src/net/http/server.go:2787 +0x7f



goroutine 2610846 [semacquire]:
sync.runtime_Semacquire(0xc4201ad768)
/usr/lib/go/src/runtime/sema.go:47 +0x34
sync.(*RWMutex).Lock(0xc4201ad760)
/usr/lib/go/src/sync/rwmutex.go:91 +0x6e
net/http.HandlerFunc.ServeHTTP(0xc420212010, 0xd6be00, 0xc42168cd20, 0xc42466c300)
/usr/lib/go/src/net/http/server.go:1942 +0x44
net/http.(*ServeMux).ServeHTTP(0xdb1200, 0xd6be00, 0xc42168cd20, 0xc42466c300)
/usr/lib/go/src/net/http/server.go:2238 +0x130
net/http.serverHandler.ServeHTTP(0xc42028a0b0, 0xd6be00, 0xc42168cd20, 0xc42466c300)
/usr/lib/go/src/net/http/server.go:2568 +0x92
net/http.(*conn).serve(0xc420395d60, 0xd6fc00, 0xc4256d2200)
/usr/lib/go/src/net/http/server.go:1825 +0x612
created by net/http.(*Server).Serve
/usr/lib/go/src/net/http/server.go:2668 +0x2ce


Thanks,
Dave

Dave Drake

unread,
Nov 28, 2017, 10:35:52 AM11/28/17
to András Belicza, gowebuitoolkit
I'm still seeing this with the latest version, and have seen it multiple times. Every time it's a concurrent map iteration and map write with the styleImpl.attrs map in the renderAttrs function at line 810. The top of the stack always looks like:

fatal error: concurrent map iteration and map write

goroutine 62 [running]:
runtime.throw(0xa964f1, 0x26)
/usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc427f1d6e8 sp=0xc427f1d6c8
runtime.mapiternext(0xc427f1d7e0)
/usr/local/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc427f1d798 sp=0xc427f1d6e8
    /workspace/src/github.com/icza/gowut/gwu/style.go:810 +0x180 fp=0xc427f1d850 sp=0xc427f1d798
/workspace/src/github.com/icza/gowut/gwu/style.go:791 +0xb5 fp=0xc427f1d898 sp=0xc427f1d850
/workspace/src/github.com/icza/gowut/gwu/comp.go:225 +0x10b fp=0xc427f1d948 sp=0xc427f1d898
/workspace/src/github.com/icza/gowut/gwu/table.go:372 +0x9a fp=0xc427f1d9d8 sp=0xc427f1d948
github.com/icza/gowut/gwu.(*serverImpl).renderComp(0xc42018a300, 0xd8a300, 0xc420452000, 0xd7cde0, 0xc421aec9a0, 0xc423971b00)
/workspace/src/github.com/icza/gowut/gwu/server.go:744 +0x208 fp=0xc427f1dad0 sp=0xc427f1d9d8
/workspace/src/github.com/icza/gowut/gwu/server.go:660 +0x53a fp=0xc427f1dc78 sp=0xc427f1dad0
/workspace/src/github.com/icza/gowut/gwu/server_start.go:49 +0x48 fp=0xc427f1dca8 sp=0xc427f1dc78
net/http.HandlerFunc.ServeHTTP(0xc4201e6010, 0xd7cde0, 0xc421aec9a0, 0xc423971b00)
/usr/local/go/src/net/http/server.go:1942 +0x44 fp=0xc427f1dcd0 sp=0xc427f1dca8
net/http.(*ServeMux).ServeHTTP(0xdc2240, 0xd7cde0, 0xc421aec9a0, 0xc423971b00)
/usr/local/go/src/net/http/server.go:2238 +0x130 fp=0xc427f1dd10 sp=0xc427f1dcd0
net/http.serverHandler.ServeHTTP(0xc420352000, 0xd7cde0, 0xc421aec9a0, 0xc423971b00)
/usr/local/go/src/net/http/server.go:2568 +0x92 fp=0xc427f1dd58 sp=0xc427f1dd10
net/http.(*conn).serve(0xc4200b2be0, 0xd80ba0, 0xc4202bc480)
/usr/local/go/src/net/http/server.go:1825 +0x612 fp=0xc427f1dfc8 sp=0xc427f1dd58
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc427f1dfd0 sp=0xc427f1dfc8
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2668 +0x2ce


Thanks,
Dave

András Belicza

unread,
Nov 29, 2017, 4:58:03 AM11/29/17
to Dave Drake, gowebuitoolkit
This is the relevant code where this happens:

rwMutex.RLock()
defer rwMutex.RUnlock()

// Render just a component
s.renderComp(win, w, r)

Where rwMutex is the session's mutex. As you can see it can only be reached while holding the session's mutex, only from a single goroutine.

If you still seeing the panics above, I'm guessing you're accessing / modifying the components (attributes) from your own gorotuines, which causes the data race. Can you confirm this?

Dave Drake

unread,
Nov 29, 2017, 12:30:56 PM11/29/17
to András Belicza, gowebuitoolkit
I am accessing/modifying components with methods like Text() and SetText(). Within each window, writes (e.g. SetText()) and markDirty() are mutex protected and require a Lock, reads (e.g. Text()) are not protected though. Also, I have a goroutine for each window, and there is no synchronization across windows, so they could be accessing/modifying components within their respective windows at the same time (window X is updating its component while window Y is updating its component). Furthermore, each window has a timer that is used to update the screen but that is protected with the window's mutex getting a Lock before calling markDirty() to ensure no other goroutine is writing to the components.

Is this architecture ok? Do I need to protect reads of components with the window's mutex as well? Or do I need to synchronize everything between windows and have one master mutex for all updates?

I did find one place in the code where the above isn't true, and it is possible that writes and markDirty() occur at the same time. 

Thanks,
Dave


András Belicza

unread,
Nov 30, 2017, 2:33:55 AM11/30/17
to Dave Drake, gowebuitoolkit
Every Go value that is accessed from multiple goroutine and at least one of those accesses is a write needs synchronization.
So if you have a goroutine that ever modifies a value, any other gorutine that even just reads it needs to be synchronized (both the write and the reads).

Reading / Modifying components in goroutines other than the handlers should be syncronized with the Session's mutex, however this is not currently not "exposed".

I will add a method to Session that will return the session's internal mutex.

Dave Drake

unread,
Nov 30, 2017, 4:44:33 PM11/30/17
to András Belicza, gowebuitoolkit
Ok, I will look to synchronize everything across all windows/goroutines.

Thanks for your help!
Dave
Reply all
Reply to author
Forward
0 new messages