I have installed and configured WebSphere 6.0.2.21 on AIX 5.3 ML8 - 2 CPU.
I have set the initial heap size as 1024M and Max heap size as 2048M. After every 2-3 days I keep getting this error and the server stops responding. I have to restart the WAS instance again and a huge core file and a javacore file are created. There are a couple of J2EE apps deployed on the server and there are currently no users accessing this server. Also, there are no other processes or programs that are hitting this server. Any help is appreciated. Here is the error from the SystemErr.log. I am also attaching the javacore file that gets generated everytime this error occurs.
[1/7/09 18:24:39:171 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.hamanager.runtime.CoordinatorComponentImpl.initialize(CoordinatorComponentImpl.java:193)
[1/7/09 18:24:39:172 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.runtime.component.ContainerImpl.initializeComponent(ContainerImpl.java:1160)
[1/7/09 18:24:39:172 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.runtime.component.ContainerImpl.initializeComponents(ContainerImpl.java:979)
[1/7/09 18:24:39:172 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.runtime.component.ServerImpl.initialize(ServerImpl.java:284)
[1/7/09 18:24:39:172 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:173)
[1/7/09 18:24:39:172 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:133)
[1/7/09 18:24:39:173 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:387)
[1/7/09 18:24:39:173 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.runtime.WsServer.main(WsServer.java:53)
[1/7/09 18:24:39:173 GMT+05:30] 0000000a SystemErr R at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[1/7/09 18:24:39:173 GMT+05:30] 0000000a SystemErr R at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
[1/7/09 18:24:39:173 GMT+05:30] 0000000a SystemErr R at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)
[1/7/09 18:24:39:174 GMT+05:30] 0000000a SystemErr R at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
[1/7/09 18:24:39:174 GMT+05:30] 0000000a SystemErr R at java.lang.reflect.Method.invoke(Method.java:391)
[1/7/09 18:24:39:174 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.bootstrap.WSLauncher.run(WSLauncher.java:219)
[1/7/09 18:24:39:174 GMT+05:30] 0000000a SystemErr R at java.lang.Thread.run(Thread.java:571)
[1/7/09 18:24:39:174 GMT+05:30] 0000000a SystemErr R Caused by: java.lang.OutOfMemoryError: JVMCI015:OutOfMemoryError, cannot create anymore threads due to memory or resource constraints
[1/7/09 18:24:39:175 GMT+05:30] 0000000a SystemErr R at java.lang.Thread.start(Native Method)
[1/7/09 18:24:39:175 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.hamanager.impl.UserCallbacks.initialize(UserCallbacks.java:159)
[1/7/09 18:24:39:175 GMT+05:30] 0000000a SystemErr R at com.ibm.ws.hamanager.runtime.CoordinatorComponentImpl.initialize(CoordinatorComponentImpl.java:182)
[1/7/09 18:24:39:175 GMT+05:30] 0000000a SystemErr R ... 14 more
[1/7/09 18:25:54:119 GMT+05:30] 0000000a SystemErr R com.ibm.ws.exception.ConfigurationError: com.ibm.ws.exception.ConfigurationError: unable to initialize the CoordinatorComponentImpl
at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:180)
at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:133)
at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:387)
at com.ibm.ws.runtime.WsServer.main(WsServer.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:58)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:60)
at java.lang.reflect.Method.invoke(Method.java:391)
at com.ibm.ws.bootstrap.WSLauncher.run(WSLauncher.java:219)
at java.lang.Thread.run(Thread.java:571)
Caused by: com.ibm.ws.exception.ConfigurationError: unable to initialize the CoordinatorComponentImpl
at com.ibm.ws.hamanager.runtime.CoordinatorComponentImpl.initialize(CoordinatorComponentImpl.java:193)
at com.ibm.ws.runtime.component.ContainerImpl.initializeComponent(ContainerImpl.java:1160)
at com.ibm.ws.runtime.component.ContainerImpl.initializeComponents(ContainerImpl.java:979)
at com.ibm.ws.runtime.component.ServerImpl.initialize(ServerImpl.java:284)
at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:173)
... 10 more
Caused by: java.lang.OutOfMemoryError: JVMCI015:OutOfMemoryError, cannot create anymore threads due to memory or resource constraints
at java.lang.Thread.start(Native Method)
at com.ibm.ws.hamanager.impl.UserCallbacks.initialize(UserCallbacks.java:159)
at com.ibm.ws.hamanager.runtime.CoordinatorComponentImpl.initialize(CoordinatorComponentImpl.java:182)
... 14 more
-Thanks
Srikant
Looking at the thread dump, wow, you have thousands of thread created. The Application runs out of "Native" Memory and dumps core. This is not your HEAP. This is native memory which is only limited by the OS.
There is good possibility that somewhere in the app you are running into a infinite loop, or you are opening something but never closes it.
Check out the code. Also once you have a good dump, post it here - I'll take a look at it. You don't need to wait for the app to crash. YOu can execute "kill -3 " to genereate the core dump (do this after the app server runs for about a day or two)
- Karun
See *Webcast replay: Introduction to the Memory Usage of a Java Process* at http://www.ibm.com/support/docview.wss?rs=1045&uid=swg27013757.
For example, the *IBM WebSphere Portal Version 6.0 Tuning Guide* (which uses WAS 6.0.2.x) at http://www.ibm.com/support/docview.wss?rs=688&uid=swg27008511 says they used 1792 MB for the highest Java maximum heap size on AIX 5.
Thank you,
David Yang
WebSphere Portal support
The postings on this site are my own and do not necessarily represent the positions, strategies, or opinions of IBM.
Each thread needs its own piece of native memory (which is different from
the Java heap controlled by the maximum heap size setting).
See also the IBM JVM Diagnostic Guide.
http://www.ibm.com/developerworks/java/jdk/diagnosis/
See also an old discussion on the subject:
http://groups.google.com/group/comp.lang.java.machine/browse_thread/thread/bd8c8229ce19ffd7.