Memory leak taking down server

[Mike817]

I setup a new server a few days ago that's running Railo 4.1.3.004 final on a Centos 6 server (Linux (2.6.32-042stab079.6) 64bit) using apache and tomcat (Apache Tomcat/7.0.37). It's getting almost no traffic, but the heap space is filling up daily to the point where it runs out and I'm getting bad gateway errors when I try to access the site. The only thing running on the machine is a single site with a couple of scheduled tasks.

When i investigated the logs, i noticed several entries like this, "Mar 17, 2014 1:32:34 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [Thread-11] but has failed to stop it. This is very likely to create a memory leak."

Is there any easy way to track down what is causing this so I can shut it down?

[Michael Offner]

Best do a heap dump with help of this tag

<cfadmin type="server" password="server-password" action="heapDump" destination="/.../...prof">

(Destination has do be a file path with a file havng the extension .prof)

there are different tools that give you a graphical overview of this dump, including eclipse, you can also send me a link to the dump and I can look into it.

Micha

--
Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki
---
You received this message because you are subscribed to the Google Groups "Railo" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/railo/1fdbec10-7b02-48a6-bb34-7cdb6a2b4546%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

/micha

Michael Offner CTO Railo Technologies GmbH

[Mike817]

Thanks Michael.

I just restarted the server this morning, so the memory levels are at normal levels. That doesn't seem to last for too long though so I'll check back in a few hours and get that dump.

[Mike817]

I ran a dump when the memory usage was around 300mb. Then i ran the prof through memory analyzer and the only suspect thing it found was:

One instance of "railo.runtime.query.CacheQueryCache" loaded by "railo.loader.classloader.RailoClassLoader @ 0xc2003e50" occupies 180,706,848 (74.37%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class loader>".

I'm not sure if this is normal or if I need to wait longer in hopes of catching the real culprit.

On Tuesday, 18 March 2014 09:09:39 UTC-4, Michael Offner wrote:

[Igal @ getRailo.org]

most likely that's your issue but I would still wait longer and trigger another heap dump when memory consumption is higher.

do you do a lot of query caching?  or cache very large queries?

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/740e5d85-8c5d-48d7-a7f3-e1a8c5817f46%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 
Igal Sapir
Railo Core Developer
http://getRailo.org/

[Brad Wood]

That's awesome that you can do a heap dump with the cfadmin tag.  I never knew that.  We really need to get that tag documented-- most everything I know about it are just snippets I've seen on this list or in code.

[Igal @ getRailo.org]

https://issues.jboss.org/browse/RAILO-2162

the cfadmin tag is undocumented on purpose

--

Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki
---
You received this message because you are subscribed to the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/cada6751-a0aa-4c16-ae1d-5ecaa1815c22%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 
Igal Sapir
Railo Core Developer
http://getRailo.org/

[Brad Wood]

I know.  You guys say that every time I bring up the lack of documentation :)  I'll be honest though-- I don't understand why.  It's part of the platform and I see it recommended for people to use (as in Micha's post in this thread) and I've seen the Railo team use it in their code.  The only reason not to document it is if you don't want people to use it (though, I'd still document it anyway, if for nothing else internal use).  However, why tell someone to use something if they're not supposed to use it?

I've also heard the excuse that someday an admin API will be created, but that hasn't happened yet (RAILO-2238).  Frankly, in my opinion, if <cfadmin> is what we have now, it works, and you're going to tell people on the lists to use it and you're going to use it in Extensions, just rip of the band-aid and write some docs for the puppy!  For goodness sake, it's better than nothing.

P.S. I would gladly help write docs, but I don't know what all it does :)

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

You received this message because you are subscribed to a topic in the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/5328FB6B.1010402%40getrailo.org.

[Mike817]

Unsurprisingly, the heap has stayed under control. Is there a way to run the dump from a command line? Once I run out of heap space, I won't be able to run the page with the cfadmin tag.

[Brad Wood]

You can use the JVM arg: 

- XX:+HeapDumpOnOutOfMemoryError

Or if you have the JDK tools installed, just use jmap.

http://docs.oracle.com/javase/6/docs/technotes/tools/share/jmap.html

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

On Tue, Mar 18, 2014 at 9:29 PM, Mike817 <mik...@gmail.com> wrote:

Unsurprisingly, the heap has stayed under control. Is there a way to run the dump from a command line? Once I run out of heap space, I won't be able to run the page with the cfadmin tag.

--

Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki
---

You received this message because you are subscribed to a topic in the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/fbb926c7-dfea-41cf-991d-c52f4398f2ec%40googlegroups.com.

[Mike817]

Perfect. Setting that up now. Does it get dumped to a specific location?

[Brad Wood]

Both the JVM args and jmap (use format=b) let you configure that. If memory serves, the default location was runtime/bin last time I did it.  Try it and see :)  If you Google "java heap dump" you can find info out there on it.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/6284dd5f-8c20-4273-9728-91857916ca43%40googlegroups.com.

[Igal @ getRailo.org]

it's not because we don't want people to use it and not because of the docs.  it's because the API is likely to change at some point in the future so it wasn't made public.

better than writing docs you can help by adding code to the Administrator tag:
https://github.com/getrailo/railo/blob/develop/railo-java/railo-core/src/resource/component/org/railo/cfml/Administrator.cfc

but the API must be coordinated with Micha et al, of course.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/CALbQ1om6TircJ194_1L8yvpfi5R2_pYH%2Be4nCh0XDSkqXHaEOw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Offner]

that is exactly the point, cfadmin can change at any time without warnings (no deprecated phase), it was never the idea to make it public, i will also set it to "hidden", so it will be no longer visible in the doc.

the idea is to use the Administrator.cfc, where we also have testcases for every action to make sure the interface is not affected by changes in cfadmin.

atm this cfc does not cover all cfadmin actions, but you are welcome to extend ;-)

Micha

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/532915D0.7040702%40getrailo.org.

For more options, visit https://groups.google.com/d/optout.

--

[Mike817]

Hi Brad.

My server is current giving me the bad gateway error, but I can't find a dump anywhere on the server. Is there any other way of manually doing it?

[Mike817]

I finally got the full memory dump. Out of a total heap of 740.4 MB, 697.5 MB was taken up by one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]". I'm looking through the code to see what query might be causing this, but I don't think any of them is pulling that much data.

This issue does not occur on the dev server. The two machines have similar specs, the only difference being the problem server is running 64bit centos and the dev is still 32bit. Is there any chance the architecture difference could be causing the increased memory usage on the problem machine?

[Brad Wood]

So assuming that ConcurrentHashMap is part of a CacheQueryCache object again, you can probably climb the reference tree and inspect the object to look for clues about what query it is.  Hopefully Railo stores human-readable information about the query and its parameters and not just the hash so you can identify it.  Micha can probably confirm.

I've use the Eclipse Memory Analyzer (MAT) tool with success in the past to introspect these things.  It's a bit messy, but it's the best tool I know of for post-mortem heap analysis.  

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

You received this message because you are subscribed to a topic in the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/05e44a54-88d8-4b94-9bb0-32189fa48de6%40googlegroups.com.

[Jared Rypka-Hauer]

So Mike,

My question is this: Have you tried any of the JVM instrumentation tools that are out there? Here are some instructions on getting the instrumentation set up and running on the Railo server:

https://tomcat.apache.org/tomcat-7.0-doc/monitoring.html

Then you can use something like http://visualvm.java.net (which you could have running locally on your machine watching the JMX server on the remote machine) to monitor the JVM in real-time, trigger garbage collection, take thread, core and memory dumps, analyze them, etc.

This may be your best bet for figuring out what's actually going on within the JVM your Tomcat instance is running on.

Or, and I don't know if this is possible or not, do either SeeFusion or FusionReactor run on Railo? I have my doubts, but I figured it was an option for investigation.

Laterz!

Jared

On Mon, Mar 24, 2014 at 9:55 AM, Mike817 <mik...@gmail.com> wrote:

I finally got the full memory dump. Out of a total heap of 740.4 MB, 697.5 MB was taken up by one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]". I'm looking through the code to see what query might be causing this, but I don't think any of them is pulling that much data.

This issue does not occur on the dev server. The two machines have similar specs, the only difference being the problem server is running 64bit centos and the dev is still 32bit. Is there any chance the architecture difference could be causing the increased memory usage on the problem machine?

...

[Igal @ getRailo.org]

+1 for VisualVM
+1 for Memory Analyzer Tool

quick questions:

1) in the Railo Modern Debug Template you get an estimate of the scope sizes at the bottom.  is any of them (Server/Application/Session) much larger than you expect?

2) are you running your application with only 740MB of RAM?

--

Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki
---
You received this message because you are subscribed to the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/CAPsSNqO4yiPT-w9S7Fj-0X0awJ9V9hpf0%3DNSPCUcwuZGDFb1Vw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Jared Rypka-Hauer]

I incorrectly stated that the instructions were for getting JMX running on the "Railo server".

Should have said Tomcat... not everyone is running on Tomcat and you may be using Railo Express... just sayin.

A few years ago I took the Sun Java performance monitoring and analysis class in San Francisco... it was an awesome class and must have stuck with me because I dredged this all up from memory. :)

It WILL give you the info you need to figure this out... or at least let you collect it to give someone else the info they need to figure it out for you.

:)

Sent from my iPad

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/53305D66.4060405%40getrailo.org.

[Igal @ getRailo.org]

yes, but JMX works on the JVM level, so it will work with any Java application (or any Servlet container in this case). 

the "standard" way to enable JMX is with system properties that can be added to the command line.  for example to enable for port 9500, assuming the network is secured and no authentication is required:

    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.port=9500
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false

the most common issues you may run into with JMX is when you are monitoring a Service that is run by a different user from the one that runs that JMX monitoring application, but there are usually ways around that.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/B813FD9C-37E6-491B-9AED-3D965FF61C81%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Mike817]

1.  It definitely seems to happen overnight when I'm running a couple data scripts. I'm going to have to look into that a little more.

2. I thought I was setup to run it with 1GB. I have 2 total and I need to save a little for apache and the mail server, so I figured a gig was the best number. I'm still learning about tweaking the server settings so I probably did it wrong. My setenv is: JAVA_OPTS="-Xms512m -Xmx1024m -XX:MaxPermSize=128m -javaagent:/opt/railo/lib/railo-inst.jar -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/javadumps/"

[Mike817]

I haven't yet. My previous experiences with railo have been for much smaller sites so the tweaking hasn't really been necessary.   I definitely give those docs a read later today though.

[Igal @ getRailo.org]

even 1GB is too low IMO.  memory nowadays is cheap and adding some more RAM should be much cheaper than all the time you've spent so far debugging this issue.  I strongly recommend adding more memory.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/48c9891a-4f4e-4b7e-bc4e-5e38f4946bdc%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jared Rypka-Hauer]

+1 more memory

On the other hand you may find the issue continues to occur. Throwing hardware at poor code will only get you so far... finding out what the issue is and fixing it is probably a good idea either way.

J

On Mon, Mar 24, 2014 at 12:02 PM, Igal @ getRailo.org <ig...@getrailo.org> wrote:

even 1GB is too low IMO.  memory nowadays is cheap and adding some more RAM should be much cheaper than all the time you've spent so far debugging this issue.  I strongly recommend adding more memory.

On 3/24/2014 9:59 AM, Mike817 wrote:

1.  It definitely seems to happen overnight when I'm running a couple data scripts. I'm going to have to look into that a little more.

2. I thought I was setup to run it with 1GB. I have 2 total and I need to save a little for apache and the mail server, so I figured a gig was the best number. I'm still learning about tweaking the server settings so I probably did it wrong. My setenv is: JAVA_OPTS="-Xms512m -Xmx1024m -XX:MaxPermSize=128m -javaagent:/opt/railo/lib/railo-inst.jar -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/javadumps/"

...

[Igal @ getRailo.org]

Throwing hardware at poor code will only get you so far...

+2

but it is also possible that in this case the code is reasonable and more memory will solve the issue, because even though everything is relative -- 1GB is really not that much as an absolute value.  in any event , it is impossible to tell with the information that we (the community) have at this point.

--

Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki
---
You received this message because you are subscribed to the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/CAPsSNqNm_hULFe_YSfCYpJz4V_Fe%3DOLQG%2BMdxFdfVm52cp0dFw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Igal @ getRailo.org]

1.  It definitely seems to happen overnight when I'm running a couple data scripts. I'm going to have to look into that a little more.

@Mike: what happens if you run these scripts while you're monitoring the server?  setup VisualVM (or the Railo Memory Monitor Extension which will be easier to setup -- see http://blog.getrailo.com/post.cfm/railo-4-0-previews-memory-monitor-extension) and run the nightly scripts...

[Phil Sweeney]

A long time ago I had issues with query cache using up RAM, so I switched to ehCache for the query cache and it went away.  Never went back to work out why normal cache was a problem though.

You can try it easily.. it's a few clicks in the admin page.

Phil