CFv1 DEA came across JVM crashed

[Harry Zhang]

We encountered something wrong when testing apps running on CFv1, it seems JVM in CF has some conflicts or bugs with OS. The fatal error occurs whenever we run parallel tests with even very small concurrency:

INFO [http-21643-1] (AnnouncementServiceImpl.java:97) 19:50:17,410 -- Request from client contains [lastRefreshTime,waterInfo,gasInfo,parkingInfo,constructionInfo]:[null,true,true,true,true].

 INFO [http-21643-3] (AnnouncementServiceImpl.java:97) 19:50:17,421 -- Request from client contains [lastRefreshTime,waterInfo,gasInfo,parkingInfo,constructionInfo]:[null,true,true,true,true].

 INFO [http-21643-2] (AnnouncementServiceImpl.java:97) 19:50:17,424 -- Request from client contains [lastRefreshTime,waterInfo,gasInfo,parkingInfo,constructionInfo]:[null,true,true,true,true].

 INFO [http-21643-4] (AnnouncementServiceImpl.java:97) 19:50:17,424 -- Request from client contains [lastRefreshTime,waterInfo,gasInfo,parkingInfo,constructionInfo]:[null,true,true,true,true].

 INFO [http-21643-5] (AnnouncementServiceImpl.java:97) 19:50:17,430 -- Request from client contains [lastRefreshTime,waterInfo,gasInfo,parkingInfo,constructionInfo]:[null,true,true,true,true].

#

# A fatal error has been detected by the Java Runtime Environment:

#

# java.lang.OutOfMemoryError: pthread_getattr_np

#

#  Internal Error (os_linux_x86.cpp:681), pid=21329, tid=139800558319360

#  Error: pthread_getattr_np

#

# JRE version: 6.0_24-b07

# Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode linux-amd64 compressed oops)

# An error report file with more information is saved as:

# /var/vcap/data/dea/apps/announcement-1-9da46fd0e25b4f108b3ff2056a60a29f/tomcat/hs_err_pid21329.log

#

# If you would like to submit a bug report, please visit:

#   http://java.sun.com/webapps/bugreport/crash.jsp

Our CF is deployed by nise_BOSH and the only difference we made is that our stemcell is based on Ubuntu10.04.4 with packages updated to version below:

libncurses5-dev_5.7+20100626-0ubuntu1_amd64.deb

libncurses5_5.7+20100626-0ubuntu1_amd64.deb

libreadline5-dev_5.2-7build1_amd64.deb

libreadline5_5.2-7build1_amd64.deb

ncurses-bin_5.7+20100626-0ubuntu1_amd64.deb

Does this matter? Or some part in DEA agent.rb has conflicts with JVM somehow, especially on Non-10.04 OS?

[Harry Zhang]

Fixed! It seems the JVM in CFv1 has conflicts with my IaaS when using ulimit. Stop the secure mode and enforce ulimit solves the problem.