asynchttpclient with threadpoolexecutor issue

118 views

Skip to first unread message

harsha

unread,

Mar 3, 2011, 11:29:59 PM3/3/11

to asynchttpclient

Hi all,
I am using AsyncHttpClient using tomcat and jersey framework. I am
doing a spring injection for AsyncHttpClient with scope as singleton
for the application. Which means AsynchttpClient is created when
server is started
ThreadPoolExecutor tpe = new
ThreadPoolExecutor(50,100,2,TimeUnit.SECONDS,new
LinkedBlockingQueue());
Builder builder = new
AsyncHttpClientConfig.Builder();

builder.setExecutorService(tpe);

builder.setIdleConnectionTimeoutInMs(1000);

builder.setConnectionTimeoutInMs(http_timeout);
this.asyncHttpClient = new AsyncHttpClient(builder.build());

Whenever there is a request comes tomcat i make another request using
asyncHttpClient to another source to get some data.
I am not close on asyncHttpClient and subsequent uses the same object
and gets a future and none of these requests
calls close on the asyncHttpClient.
When i did a load test on server after a while the server stops
responding

This is what i found in jstack
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access
$800(LinuxDebuggerLocal.java:51)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal
$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal
$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
sun.jvm.hotspot.debugger.DebuggerException:
sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for
a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal
$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java

pstack
Thread 1245 (Thread 0x41ee5940 (LWP 31320)):
#0 0x0000003634a0ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002add27e0e897 in os::PlatformEvent::park ()
#2 0x00002add27de5ff5 in Monitor::IWait ()
#3 0x00002add27de67be in Monitor::wait ()
#4 0x00002add27b8b8b5 in GCTaskManager::get_task ()
#5 0x00002add27b8cb63 in GCTaskThread::run ()
#6 0x00002add27e0f16f in java_start ()
#7 0x0000003634a064a7 in start_thread () from /lib64/libpthread.so.0
#8 0x00000036342d3c2d in clone () from /lib64/libc.so.6

Is it because of too many requests are crowding asyncHttpClient and
not calling close once you done with request leaving threads made by
asynchttpclient. Any pointers will be greatly appreciated.
Thanks

jfarcand

unread,

Mar 4, 2011, 8:50:27 AM3/4/11

to asyncht...@googlegroups.com

Salut,

Can you attach the complete stack trace? I don't think the stack above
is the issue. Also does it makes a difference if you comment out

builder.setIdleConnectionTimeoutInMs(1000);

Thanks!

-- Jeanfrancois

> Thanks
>

harsha

unread,

Mar 4, 2011, 1:09:51 PM3/4/11

to asynchttpclient

Hi,
Thanks for the reply. Will do that change and try it out.
Here is the part of stack trace and rest looks similar too. I am not
able post entire
stack trace here .

Attaching to process ID 31319, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 17.1-b03

sun.jvm.hotspot.debugger.DebuggerException:
sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for
a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal

$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:
466)
at
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:
65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:
92)
at
sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:
256)
at
sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:
218)
at sun.jvm.hotspot.tools.PStack.initJFrameCache(PStack.java:208)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:66)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)

Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs
failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native
Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access
$800(LinuxDebuggerLocal.java:51)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal
$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal
$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
sun.jvm.hotspot.debugger.DebuggerException:
sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for
a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal

harsha

unread,

Mar 7, 2011, 1:22:10 AM3/7/11

to asynchttpclient

Hi,
I tried commenting out this
builder.setIdleConnectionTimeoutInMs(1000);
It didn't help. The reason why I have that is there were idle
connection dangling and with only few requests
cpu usage used to shot up and after a while server was unable to
handle any connections this got resolved
by the above the setting.
Right now the application is served by tomcat server fronted by apache
using ajp. AsyncHttpClient is spring injected with
singleton scope. After doing a load test apache reaches its max
clients and after a while tomcat doesn't repsond. I find the above
stack trace
in server crash. I got rid of thread pool and tested it again it
didn't help.
Thanks for any help.

-Harsha

> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterS et0(Native
> Method)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access
> $800(LinuxDebuggerLocal.java:51)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal
> $1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:460)
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal
> $LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:127)
> sun.jvm.hotspot.debugger.DebuggerException:
> sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for
> a lwp
> at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal

> $LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
> at
> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterS et(LinuxDebuggerLocal.java:

> 466)
> at
> sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:
> 65)
> at
> sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrent FrameGuess(LinuxAMD64JavaThreadPDAccess.java:
> 92)
> at
> sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:
> 256)
> at
> sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:
> 218)
> at sun.jvm.hotspot.tools.PStack.initJFrameCache(PStack.java:208)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:66)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
> at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
> at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
> at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
> at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 39)

> at...
>
> read more »

dominict

unread,

Mar 7, 2011, 7:47:53 AM3/7/11

to asynchttpclient

Hi harsha
The above stack trace is from jstack not being able to connect to the
tomcat jvm to provide a stack dump (not the threads from inside
tomcat). This could be for numerous reasons:

- Incorrect user trying to connect to the tomcat jvm process
- Different jstack from a different jvm trying to connect to the
tomcat running a different jvm version.

Easiest way to get a stack trace for the tomcat in the catalina.out is
to send a signal to the tomcat process via a [kill -3 <pid>] of the
tomcat process.

cheers
/dom