Java OOM debugging

[Head-Rapson, David]

Hi,

I’m after some guidance on how to get profile Java apps in CF, in order to get to the bottom of memory issues.

We have an app that’s crashing every few hours with OOM error, most likely it’s a memory leak.

I’d like to profile the JVM and work out what’s eating memory, however tools like yourkit require connectivity INTO the JVM server (i.e. the warden container), either via host / port or via SSH.

Since warden containers cannot be connected to on ports other than for HTTP and cannot be SSHd to, neither of these works for me.

 

I tried installed a standalone JDK onto the warden container, however as soon as I ran ‘jmap’ to invoke the dump, warden cleaned up the container – most likely for memory over-consumption.

 

I had previously found a hack in the Weblogic buildpack (https://github.com/pivotal-cf/weblogic-buildpack/blob/master/docs/container-wls-monitoring.md) for modifying the start script which, when used with –XX:HeapDumpOnOutOfMemoryError, should copy any heapdump files to a file share somewhere.  I have my own custom buildpack so I could use something similar.

Has anyone got a better solution than this?

 

We would love to use newrelic / app dynamics for this however we’re not allowed. And I’m not 100% certain they could help with this either.

 

Dave

 

The information transmitted is intended for the person or entity to which it is addressed and may contain confidential, privileged or copyrighted material. If you receive this in error, please contact the sender and delete the material from any computer.  Fidelity only gives information on products and services and does not give investment advice to retail clients based on individual circumstances. Any comments or statements made are not necessarily those of Fidelity. All e-mails may be monitored. FIL Investments International (Reg. No.1448245), FIL Investment Services (UK) Limited (Reg. No. 2016555), FIL Pensions Management  (Reg. No. 2015142) and Financial Administration Services Limited (Reg. No. 1629709) are authorised and regulated in the UK by the Financial Conduct Authority. FIL Life Insurance Limited (Reg No. 3406905) is authorised in the UK by the Prudential Regulation Authority and regulated in the UK by the Financial Conduct Authority and the Prudential Regulation Authority. Registered offices at Oakhill House, 130 Tonbridge Road, Hildenborough, Tonbridge, Kent TN11 9DZ.

[Lingesh Mouleeshwaran]

Hi Dave ,

1) We have tried plumbr , dynatrace agents , could be these all you can get an trial license and try to push the jars as part of buildpack. then try to set agent path on CATALINA_OPTS to load the respective agent and lets agents can post stats via HTTP to the server.

2) tmate.io , cf-ssh and pwnshell (http://i8jesus.com/?p=191) could be helpful for monitor and ssh purpose of wardens.

Regards

Lingesh M

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/DFFA4ADB9F3BC34194429921AB329336408CAB04%40UKFIL7006WIN.intl.intlroot.fid-intl.com.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.