Linux java.lang.OutOfMemoryError: unable to create new native thread
Arnaud Berger
We also encountered this problem before.
You will probably have to play with the "ulimit" command.
Regards,
Arnaud
"Mit Majithia" <mmaj...@i411.com> a écrit dans le message news:
f25470d2.04021...@posting.google.com...
> Hi,
> We are running a web application which is a stateless system and
> gets a lot of hits a day per server.
> We are running the following configuration :
> Red Hat Linux 7.3
> Linux Kernel 2.4.20-19.7smp
> Sun JVM JDK 1.4.1_02
> The machine has 2GB of RAM and we have set the -XMx=1500M.
>
> Pretty much everyday after about 200,000 - 250,000 hits, the
> server get the error shown below. We have looked at the memory usage
> and it looks ok.The application does load up some data and use a
> substantial amount of memory(about 500M). But we looking through both
> top and Runtime.getRuntime().totalMemory() we see that it has still
> not close to the 1500M limit that we have set. Also we use the
> following code on a jsp page to montior the threads and they don't
> seem to be spiralling out of control
>
> Thread[] threads = new Thread[1000];
> ThreadGroup tg = Thread.currentThread().getThreadGroup();
>
> while (tg.getParent() != null)
> {
> tg = tg.getParent();
> }
>
> tg.enumerate( threads );
> for (int i = 0; i < threads.length; i++)
> {
> if (threads[i] != null)
> {
> out.print(i + ": ");
> out.print( "pri=" + threads[i].getPriority() + " ");
> out.print( "ac=" + threads[i].activeCount() + " ");
> out.println(threads[i]);
> }
> }
>
> Does anyone have any idea what the problem could be ? What the
> possible issues could be ? We have looked at Memory and threads and
> they don't seem to be the issue. Is here any documentation about the
> cases that would through up this error ? Any help in this matter would
> be greatly appreciated.
>
> Thanks,
> Mit
>
>
>
>
>
>
>
>
> javax.servlet.ServletException: Servlet execution threw an exception
> at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown
> Source)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(Unknown
> Source)
> at org.apache.catalina.core.StandardWrapperValve.invoke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
> at org.apache.catalina.core.StandardContextValve.invoke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
> at org.apache.catalina.core.StandardContext.invoke(Unknown Source)
> at org.apache.catalina.core.StandardHostValve.invoke(Unknown Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.valves.ErrorDispatcherValve.invoke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.valves.ErrorReportValve.invoke(Unknown Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
> at org.apache.catalina.core.StandardEngineValve.invoke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
> at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
> at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:594)
> at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConne
ction(Http11Protocol.java:392)
> at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
> at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:619)
> at java.lang.Thread.run(Thread.java:536)
> ----- Root Cause -----
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start(Native Method)
> at
com.i411.search.controller.LowLevelSearch.performSocketSearch(LowLevelSearch
.java:428)
> at
com.i411.search.controller.LowLevelSearch.performNetworkSearch(LowLevelSearc
h.java:348)
> at
com.i411.search.controller.LowLevelSearch.ExecuteSearch(LowLevelSearch.java:
145)
> at
com.i411.search.controller.SearchController.catItemsSearch(SearchController.
java:407)
> at
com.i411.search.controller.SearchController.getSearchResults(SearchControlle
r.java:84)
> at com.i411.dex.searchflow.SearchExecutor.run(SearchExecutor.java:45)
> at
com.i411.dex.searchflow.BusinessNameSearchExecutor.run(BusinessNameSearchExe
cutor.java:82)
> at
com.i411.dex.searchflow.DexSearchFlow.performSearchFlow(DexSearchFlow.java:3
35)
> at
com.i411.web.common.WebController.executeSearchWithSearchFlow(WebController.
java:137)
> at
com.i411.web.common.WebController.executeSearchWithSearchFlow(WebController.
java:182)
> at
com.i411.dex.xml.XMLSearchServlet.PerformQuerySearch(XMLSearchServlet.java:2
60)
> at com.i411.dex.xml.XMLSearchServlet.doGet(XMLSearchServlet.java:178)
> at com.i411.dex.xml.XMLSearchServlet.doPost(XMLSearchServlet.java:366)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java)
> at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown
> Source)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(Unknown
> Source)
> at org.apache.catalina.core.StandardWrapperValve.invoke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
> at org.apache.catalina.core.StandardContextValve.invoke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
> at org.apache.catalina.core.StandardContext.invoke(Unknown Source)
> at org.apache.catalina.core.StandardHostValve.invoke(Unknown Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.valves.ErrorDispatcherValve.invoke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.valves.ErrorReportValve.invoke(Unknown Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
> at org.apache.catalina.core.StandardEngineValve.invoke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(Unknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unknown Source)
> at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
> at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:594)
> at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConne
ction(Http11Protocol.java:392)
> at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
> at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:619)
> at java.lang.Thread.run(Thread.java:536)
Jayaram
the soft and hard limits (kernel settings) on the number of threads
allowed per process on your system. These limits may have been set to
a low value on your system.
Reagrds,
Jay
Matthias Ernst
> Hi,
> We are running a web application which is a stateless system and
> gets a lot of hits a day per server.
> We are running the following configuration :
> Red Hat Linux 7.3
> Linux Kernel 2.4.20-19.7smp
> Sun JVM JDK 1.4.1_02
> The machine has 2GB of RAM and we have set the -XMx=1500M.
The higher you set -mx, the less threads you can create (why does a
stateless system need 1.5G anyway ...)
I think a very naive formula is max threads = (2G - mx) / stacksize.
You may try to work with a smaller heap size or smaller stack sizes.
Matthias
--
Matthias Ernst
CoreMedia - The Content Technology Experts
"Toll takes its time."
Chris Bailey
"Mit Majithia" <mmaj...@i411.com> wrote in message
news:f25470d2.04021...@posting.google.com...
> Update:
> We have been playing around with the settings.One of the
> settings that we changed was to reduce the memory allocated to the JVM
> (-Xmx1500m) to be lesser to -Xmx1000m. This seems to have alleviated
> the situation a little for the time being. It does not go down and
> does not give the error as before. Also we do have to restart the
> server on a daily basis to update the data.But with the new settings
> it makes it easily through the entire day without giving an error. Can
> anyone explain this ? Due to the growing traffic and data we will
> probably need the expanded memory sometime in the future. We feel that
> at the moment we have just pushed the problem to surface at some point
> in the future.
>
> Thanks,
> Mit
The Java process has two memory areas: the Java heap, and the "native heap",
which combine total the memory usage of the process.
The Java heap is controlled via the -Xms and -Xmx setting, and the space
available to the native heap is that which isn't used by the Java heap.
The act of reducing the maximum Java heap size has made the "native heap"
bigger, and this is the area that was memory constrained.
We know this because the OutOfMemoryError was generated the message informed
you that the JVM was unable to allocate a new native stack, this is
allocated onto the native heap (there is also a Java thread object which is
created and allocated onto the Java heap).
It is entirely possible that the amount of "native heap" available to the
JVM was insufficient to allocate the underlying resources to run the Java
process under the load that was being driven through it. The native heap is
now 500MB bigger, and unless there is a memory leak or the load is
significantly increased, this change should prevent any OutOfMemoryErrors
based on the native heap.
Moazam Raja
What are the full JVM options you are running with? Do you have -Xms
specified also, and are you using any custom PermGen settings? You may
want to try setting -Xms to the same value as -Xmx.
Try using JVMSTAT on the process and check to see what is actually
happening in the VM (as per young generation, old generation, etc.).
JVMSTAT will help you visualize what is happening with memory inside the
VM. This will help you figure out what you need to tune.
To get jvmstat, http://developers.sun.com/dev/coolstuff/jvmstat/ .
Otherwise, if you just want to manually look at the garbage collection
information you can add flags such as '-verbose:gc' and
'XX:+PrintGCDetails'. You can also output the GC information to a file
using '-Xloggc:<file>'.
-Moazam
Mit Majithia wrote:
> Hi,
> We are running a web application which is a stateless system and
> gets a lot of hits a day per server.
> We are running the following configuration :
> Red Hat Linux 7.3
> Linux Kernel 2.4.20-19.7smp
> Sun JVM JDK 1.4.1_02
> The machine has 2GB of RAM and we have set the -XMx=1500M.
>
> ----- Root Cause -----
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start(Native Method)
> at com.i411.search.controller.LowLevelSearch.performSocketSearch(LowLevelSearch.java:428)
> at com.i411.search.controller.LowLevelSearch.performNetworkSearch(LowLevelSearch.java:348)
..............