Resource temporarily unavailable

782 views
Skip to first unread message

dlin

unread,
Jun 5, 2013, 8:52:58 PM6/5/13
to golan...@googlegroups.com
I wrote a proxy program to collect some tcp input into a buffer then transfer to another server.

The test environment is just in a local lan.  One client open/send/recv/close to this proxyserver one second once.

I put this proxy program overnight, and found it quit and show the following error log:

Is there any tip to debug such problem?

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f800773c1c9


goroutine 1 [chan receive]:
myproto.(*ProxyServer).Serve(0xc2000db000)
proxyserver.go:418 +0x5a9
main.main()
myview.go:46 +0x470

goroutine 2 [syscall]:

goroutine 4 [semacquire]:
sync.runtime_Semacquire(0xc2000002c8)
/usr/lib/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*WaitGroup).Wait(0xc2000b1600)
/usr/lib/go/src/pkg/sync/waitgroup.go:109 +0x141
myproto.(*ProxyServer).srvService(0xc2000db000)
proxyserver.go:449 +0x6c0
created by myproto.(*ProxyServer).Serve
proxyserver.go:362 +0xb9

goroutine 5 [IO wait]:
net.runtime_pollWait(0x7f8007ecef00, 0x72, 0x0)
/usr/lib/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2000ba080, 0xb, 0xc2000a3ab0)
/usr/lib/go/src/pkg/net/fd_poll_runtime.go:75 +0x4d
net.(*netFD).accept(0xc2000ba000, 0x73d8d0, 0x0, 0xc2000a3ab0, 0xb, ...)
/usr/lib/go/src/pkg/net/fd_unix.go:385 +0x36a
net.(*TCPListener).AcceptTCP(0xc200000240, 0x7f8007d89018, 0xc2000a2780, 0x453fe7)
/usr/lib/go/src/pkg/net/tcpsock_posix.go:229 +0x73
net.(*TCPListener).Accept(0xc200000240, 0x4, 0x4, 0x649d40, 0xc201f8b280, ...)
/usr/lib/go/src/pkg/net/tcpsock_posix.go:239 +0x33
myproto.func·005()
proxyserver.go:387 +0x21d
created by myproto.(*ProxyServer).Serve
proxyserver.go:410 +0x4d5

goroutine 6 [select]:
myproto.(*ProxyServer).sendCmdsOrHBToSrv(0xc2000db000)
proxyserver.go:483 +0x167c
myproto.func·008()
proxyserver.go:439 +0x51
created by myproto.(*ProxyServer).srvService
proxyserver.go:441 +0x5a7

goroutine 128413 [semacquire]:
sync.runtime_Semacquire(0xc202c2a6f0)
/usr/lib/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*WaitGroup).Wait(0xc202d9c440)
/usr/lib/go/src/pkg/sync/waitgroup.go:109 +0x141
myproto.(*ClientInfo).Handler(0xc202d9b770, 0xc2000dc000)
proxyserver.go:77 +0x258
created by myproto.func·005
proxyserver.go:407 +0x7ad

goroutine 128417 [semacquire]:
sync.runtime_Semacquire(0xc202c2a760)
/usr/lib/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*WaitGroup).Wait(0xc202d9c5c0)
/usr/lib/go/src/pkg/sync/waitgroup.go:109 +0x141
myproto.(*ClientInfo).Handler(0xc202d9bee0, 0xc2000dc000)
proxyserver.go:77 +0x258
created by myproto.func·005
proxyserver.go:407 +0x7ad

Jesse McNelis

unread,
Jun 5, 2013, 9:29:27 PM6/5/13
to dlin, golang-nuts
On Thu, Jun 6, 2013 at 10:52 AM, dlin <dli...@gmail.com> wrote:
> I wrote a proxy program to collect some tcp input into a buffer then
> transfer to another server.
>
> The test environment is just in a local lan. One client
> open/send/recv/close to this proxyserver one second once.
>
> I put this proxy program overnight, and found it quit and show the following
> error log:
>
> Is there any tip to debug such problem?
>
> runtime/cgo: pthread_create failed: Resource temporarily unavailable
> SIGABRT: abort
> PC=0x7f800773c1c9

You're running out of threads.
Which means either:
* The allowable threads per process on your machine in really low
* You set GOMAXPROCS really high
or more likely you've got a lot of threads blocking on syscalls or CGO calls.

I can't tell from your stacktrace, but I'm going to guess this is DNS
lookup related.
Most system DNS resolvers are blocking CGO calls that will tie up a
thread for the
time required to perform the DNS loop.

If you need to use the system DNS resolver(because you need access to
it's config, dhcp provided nameservers etc.) then your best option is
to
ratelimit the number of connections your proxy will accept to a number
within your thread limit.

Otherwise you can build the net pkg without CGO support and it will
fall back to using the pure
Go DNS resolver which should perform better.

For more details: https://code.google.com/p/go/issues/detail?id=5625#c13

dlin

unread,
Jun 5, 2013, 10:58:03 PM6/5/13
to golan...@googlegroups.com
I just use specific IP, so I guess this is not caused by DNS problem.
I guess it is caused by 'goroutine leak' problem.  Is there any trick to check 'gorutine leak' problem?

I saw the goroutine number from 1~6. And suddenly jump to 128417.  Does that matter?

That last segment in the log was:
rax     0x0
rbx     0xb
rcx     0xffffffffffffffff
rdx     0x6
rdi     0x5690
rsi     0x5692
rbp     0x7f7ff76fa900
rsp     0x7f7ff76fa6f8
r8      0x7f7ff76fb700
r9      0x7f800774f327
r10     0x8
r11     0x206
r12     0x7f7ff00008c0
r13     0x7fffe40a3620
r14     0x1000
r15     0x7f7ff76fb700
rip     0x7f800773c1c9
rflags  0x206
cs      0x33
fs      0x0
gs      0x0

James Bardin

unread,
Jun 6, 2013, 10:03:17 AM6/6/13
to golan...@googlegroups.com


On Wednesday, June 5, 2013 10:58:03 PM UTC-4, dlin wrote:
I just use specific IP, so I guess this is not caused by DNS problem.
I guess it is caused by 'goroutine leak' problem.  Is there any trick to check 'gorutine leak' problem?

I saw the goroutine number from 1~6. And suddenly jump to 128417.  Does that matter?


No, you only have 7 goroutines. 128417 is just how many you've had over the execution lifetime.

 

dlin

unread,
Jun 7, 2013, 5:34:20 AM6/7/13
to golan...@googlegroups.com
I use these code to print out some information.
return fmt.Sprint("NumCPU:", runtime.NumCPU(),
" NumCgoCall:", runtime.NumCgoCall(),
" NumGoRoutine:", runtime.NumGoroutine())

I found the goroutine number is not increasing, but the CgoCall() increasing even my client connection closed.

So, Is there any tip to print out which CgoCall caused this problem?
And how to prevent this?
Reply all
Reply to author
Forward
0 new messages