Portal 6.1 Crashed with "java/lang/OutOfMemoryError" Error

[lee...@hotmail.com]

Hi,

Had a problem where the websphere portal server crashed. We are running WPS 6.1.0.1 with WAS 6.1.0.19. It is connected to DB2 9.1 database.

It actually crashed in an irregular state. One of the scenarios is when we are deploying a portlet application.

In the native.stderr.log, the following is captured:

UTE430: can't allocate buffer
UTE437: Unable to load formatStrings for j9mm

with jvm dump indication:
JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait.

The maximum heap size available to the JVM in Portal instance is 768MB.
Attached are the archived portal ffdc exception logs and the native.stderr.log.

Most appreciate of any feedbacks.

Thanks in advance,
CW

[David Yang]

Hello,
Many customers use a Java maximum heap size that is higher than 768 MB. You increase it to see if that helps.

Otherwise if the problem continues, please follow technote *MustGather: OutOfMemoryError on WebSphere Portal 6.1* at http://www.ibm.com/support/docview.wss?rs=688&uid=swg21316563 and then create a PMR to investigate the problem. Remember to enable verbosegc.

Thank you,
David Yang
WebSphere Portal support

The postings on this site are my own and do not necessarily represent the positions, strategies, or opinions of IBM.

[lee...@hotmail.com]

Hi,

Noted with thanks. But what is recommended optimum Java heapsize? Our portal server is running in Windows 2003 with 10GB physical memory. Currently serving more than 1000 users with possible 300 concurrent users.

Additionally, will I be able to view the JVM memory usage stats for the portal instance in the Tivoli Performance Viewer in WAS console? Currently, only the WAS instance is available for monitoring. I have already enabled Performance Monitoring Infrastructure (PMI) with basic monitored stats set for the portal instance.

Appreciate and thanks again,
CW

[Greg]

Your problem happened during deployment, which leads me to believe
you're hitting an interesting issue with WAS/PS that we also
experienced. What happens is that the EAR that is deployed, when
serialized, ends up being a large enough size that the heap does not
have a contiguous block of free memory to place it in because the JVM
has been up and running for a while and has bits of memory used all
over the place. The resolution, sadly, is to simply restart prior to
deployment and or move to a bigger heap size. I wrote a bit about it
here: http://www.gregheidorn.com/wordpress/2008/08/25/outofmemory-errors-during-deployment/

FYI, we use 2GB min/max for portal per JVM, using large pages (-xlp).
Not sure about Tivoli, I tend to prefer native_stderr.log for
analysis...call me old fashioned. If this problem only happens during
deployment, then I don't think your application needs a bigger heap,
just a restart prior to deployments if the system has been up for a
while.

Hope this helps,
Greg

[lee...@hotmail.com]

Hi,

Noted with thanks. I have actually increased the heapsize as per reference to the tuning guide. The current max heapsize is 1408MB. But still hit the java.lang.OutOfMemory error regularly in different state. One of the notable scenario I have monitored is the portal still crashed even though the load request is minimal. Not too sure on the internal portal process though.

Below are a few of the log excerpts captured in the System.out:

--[4/14/09 9:57:41:247 SGT]-- 000000bd InitThemePoli E com.ibm.wps.theme.policy.tags.InitThemePolicyTag doStartTag EJPNG0006E: There was an error in trying to retrieve information in the InitThemePolicy Tag.
java.lang.OutOfMemoryError:
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:212)
at com.ibm.icm.jcr.FileMultiReadBinaryValue$LazyInitInputStream.read(FileMultiReadBinaryValue.java:149)
at com.ibm.wps.policy.commands.StorageApi.debugBlob(StorageApi.java:2179)
at com.ibm.wps.policy.commands.StorageApi.getPvsProperties(StorageApi.java:1258)
at com.ibm.wps.policy.services.PolicyService.getPolicy(PolicyService.java:191)
at com.ibm.wps.policy.services.PolicyCacheManager.getPoliciesFromRespository(PolicyCacheManager.java:409)
at com.ibm.wps.policy.services.PolicyCacheManager.traverseDownTree(PolicyCacheManager.java:698)
at com.ibm.wps.policy.services.PolicyCacheManager.traverseTree(PolicyCacheManager.java:573)
at com.ibm.wps.policy.services.PolicyCacheManager.getPolicies(PolicyCacheManager.java:188)
at com.ibm.wps.policy.services.PolicyManagerServerSideImpl.getPVSByPath(PolicyManagerServerSideImpl.java:352)
at com.ibm.wps.theme.policy.ThemePolicyManager.getThemePolicyValueSet(ThemePolicyManager.java:227)
at com.ibm.wps.theme.policy.tags.InitThemePolicyTag.doStartTag(InitThemePolicyTag.java:149)

and when uploading .war for portlet application:

--[4/13/09 18:43:23:184 SGT]-- 00000033 ServletWrappe E SRVE0068E: Uncaught exception thrown in one of the service methods of the servlet: PortletManager. Exception thrown : java.lang.OutOfMemoryError:
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:212)
at com.ibm.wps.portlets.struts.upload.WpFileItem.get(WpFileItem.java:265)
at com.ibm.wps.portlets.struts.upload.WpsCommonsMultipartRequestHandler$CommonsFormFile.getFileData(WpsCommonsMultipartRequestHandler.java:415)
at com.ibm.wps.portlets.portletmanager.utils.WebModuleInstallUpdateUtil.getWebModuleInstallUpdateDataFromFormFile(WebModuleInstallUpdateUtil.java:238)
at com.ibm.wps.portlets.portletmanager.actions.UploadWarFileAction.putDisplayableStringsIntoActionForm(UploadWarFileAction.java:340)
at com.ibm.wps.portlets.portletmanager.actions.UploadWarFileAction.doExecute(UploadWarFileAction.java:225)
at com.ibm.wps.portlets.adminstruts.actions.BaseAction.execute(BaseAction.java:64)
at com.ibm.wps.portlets.struts.WpsRequestProcessor.processActionPerform(WpsRequestProcessor.java:338)

Would appreciate of any views. Or should I raise a PMR on this issue?

Thanks again,
CW

[David Yang]

Hello,
Sorry but creating a PMR would be the best way to investigate this problem.

[lee...@hotmail.com]

Hi,

No worries. Thanks. I have already raised a PMR on it.

Thanks,
CW