Resource temporarily unavailable

[dlin]

I wrote a proxy program to collect some tcp input into a buffer then transfer to another server.

The test environment is just in a local lan.  One client open/send/recv/close to this proxyserver one second once.

I put this proxy program overnight, and found it quit and show the following error log:

Is there any tip to debug such problem?

runtime/cgo: pthread_create failed: Resource temporarily unavailable

SIGABRT: abort

PC=0x7f800773c1c9

goroutine 1 [chan receive]:

myproto.(*ProxyServer).Serve(0xc2000db000)

proxyserver.go:418 +0x5a9

main.main()

myview.go:46 +0x470

goroutine 2 [syscall]:

goroutine 4 [semacquire]:

sync.runtime_Semacquire(0xc2000002c8)

/usr/lib/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e

sync.(*WaitGroup).Wait(0xc2000b1600)

/usr/lib/go/src/pkg/sync/waitgroup.go:109 +0x141

myproto.(*ProxyServer).srvService(0xc2000db000)

proxyserver.go:449 +0x6c0

created by myproto.(*ProxyServer).Serve

proxyserver.go:362 +0xb9

goroutine 5 [IO wait]:

net.runtime_pollWait(0x7f8007ecef00, 0x72, 0x0)

/usr/lib/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82

net.(*pollDesc).WaitRead(0xc2000ba080, 0xb, 0xc2000a3ab0)

/usr/lib/go/src/pkg/net/fd_poll_runtime.go:75 +0x4d

net.(*netFD).accept(0xc2000ba000, 0x73d8d0, 0x0, 0xc2000a3ab0, 0xb, ...)

/usr/lib/go/src/pkg/net/fd_unix.go:385 +0x36a

net.(*TCPListener).AcceptTCP(0xc200000240, 0x7f8007d89018, 0xc2000a2780, 0x453fe7)

/usr/lib/go/src/pkg/net/tcpsock_posix.go:229 +0x73

net.(*TCPListener).Accept(0xc200000240, 0x4, 0x4, 0x649d40, 0xc201f8b280, ...)

/usr/lib/go/src/pkg/net/tcpsock_posix.go:239 +0x33

myproto.func·005()

proxyserver.go:387 +0x21d

created by myproto.(*ProxyServer).Serve

proxyserver.go:410 +0x4d5

goroutine 6 [select]:

myproto.(*ProxyServer).sendCmdsOrHBToSrv(0xc2000db000)

proxyserver.go:483 +0x167c

myproto.func·008()

proxyserver.go:439 +0x51

created by myproto.(*ProxyServer).srvService

proxyserver.go:441 +0x5a7

goroutine 128413 [semacquire]:

sync.runtime_Semacquire(0xc202c2a6f0)

/usr/lib/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e

sync.(*WaitGroup).Wait(0xc202d9c440)

/usr/lib/go/src/pkg/sync/waitgroup.go:109 +0x141

myproto.(*ClientInfo).Handler(0xc202d9b770, 0xc2000dc000)

proxyserver.go:77 +0x258

created by myproto.func·005

proxyserver.go:407 +0x7ad

goroutine 128417 [semacquire]:

sync.runtime_Semacquire(0xc202c2a760)

/usr/lib/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e

sync.(*WaitGroup).Wait(0xc202d9c5c0)

/usr/lib/go/src/pkg/sync/waitgroup.go:109 +0x141

myproto.(*ClientInfo).Handler(0xc202d9bee0, 0xc2000dc000)

proxyserver.go:77 +0x258

created by myproto.func·005

proxyserver.go:407 +0x7ad

[Jesse McNelis]

On Thu, Jun 6, 2013 at 10:52 AM, dlin <dli...@gmail.com> wrote:
> I wrote a proxy program to collect some tcp input into a buffer then
> transfer to another server.
>
> The test environment is just in a local lan. One client
> open/send/recv/close to this proxyserver one second once.
>
> I put this proxy program overnight, and found it quit and show the following
> error log:
>
> Is there any tip to debug such problem?
>
> runtime/cgo: pthread_create failed: Resource temporarily unavailable
> SIGABRT: abort
> PC=0x7f800773c1c9

You're running out of threads.
Which means either:
* The allowable threads per process on your machine in really low
* You set GOMAXPROCS really high
or more likely you've got a lot of threads blocking on syscalls or CGO calls.

I can't tell from your stacktrace, but I'm going to guess this is DNS
lookup related.
Most system DNS resolvers are blocking CGO calls that will tie up a
thread for the
time required to perform the DNS loop.

If you need to use the system DNS resolver(because you need access to
it's config, dhcp provided nameservers etc.) then your best option is
to
ratelimit the number of connections your proxy will accept to a number
within your thread limit.

Otherwise you can build the net pkg without CGO support and it will
fall back to using the pure
Go DNS resolver which should perform better.

For more details: https://code.google.com/p/go/issues/detail?id=5625#c13

[dlin]

I just use specific IP, so I guess this is not caused by DNS problem.

I guess it is caused by 'goroutine leak' problem.  Is there any trick to check 'gorutine leak' problem?

I saw the goroutine number from 1~6. And suddenly jump to 128417.  Does that matter?

That last segment in the log was:

rax     0x0

rbx     0xb

rcx     0xffffffffffffffff

rdx     0x6

rdi     0x5690

rsi     0x5692

rbp     0x7f7ff76fa900

rsp     0x7f7ff76fa6f8

r8      0x7f7ff76fb700

r9      0x7f800774f327

r10     0x8

r11     0x206

r12     0x7f7ff00008c0

r13     0x7fffe40a3620

r14     0x1000

r15     0x7f7ff76fb700

rip     0x7f800773c1c9

rflags  0x206

cs      0x33

fs      0x0

gs      0x0

[James Bardin]

On Wednesday, June 5, 2013 10:58:03 PM UTC-4, dlin wrote:

I just use specific IP, so I guess this is not caused by DNS problem.

I guess it is caused by 'goroutine leak' problem.  Is there any trick to check 'gorutine leak' problem?

I saw the goroutine number from 1~6. And suddenly jump to 128417.  Does that matter?

No, you only have 7 goroutines. 128417 is just how many you've had over the execution lifetime.

 

[dlin]

I use these code to print out some information.

return fmt.Sprint("NumCPU:", runtime.NumCPU(),

" NumCgoCall:", runtime.NumCgoCall(),

" NumGoRoutine:", runtime.NumGoroutine())

I found the goroutine number is not increasing, but the CgoCall() increasing even my client connection closed.

So, Is there any tip to print out which CgoCall caused this problem?

And how to prevent this?