Advice on memory management in server processes - particularly running out of memory

[Alan Kent]

I have a long running server process (in Java of course!). Currently I
have a number of different subsystems all in the same server, with lots
of thread pools etc. However, sometimes an operation comes along where
it runs out of memory. It is hard to predict the memory usage given the
inputs, particularly when using third party libraries. Multiple
requests may also arrive at the same time, also making it hard to manage.

I can increase the memory the JVM can use, but this just makes it happen
less often - it does not mean its not possible to still happen. It may
even be that some operations can never be done due to memory limits -
but its hard to know this until you get into doing the operation and
actually running out of memory.

The problem I have is that currently I have lots of things going on in
the one server. If one thread uses up too much memory, OutOfMemory
exceptions come flying from all over the place (not just the thread
doing the expensive operation). I am starting to think to make things
robust I really need to have multiple JVMs in separate processes so that
if one has a problem, it won't affect other parts of the system.

Are there any good resources or sources of information around talking
about resource management in server processes? For robustness, is it
better to segregate functionality into separate processes and introduce
communication between processes? Or are there ways to control the
resource usage (typically memory usage) of say a thread, including when
calling into third party libraries that you didn't write (and so cannot
control how much memory they use).

Thanks for any opinions/advice/references!
Alan.

[Kirk Pepperdine]

Hi Alan,

First you seem to be deciding on a course of action without really understanding what the underlying problem really is. The first question I would ask is, do you have a resource consumption problem or a memory leak? If you have a memory leak then all of your efforts to segregate your application would be for not as you'd still run out of memory and the segregation may or may not address dependencies that may still cause things to fail. These may take longer to fail and that failure will most likely be more complex to understand. The flip side is that a vertical segregation may accidentally help you identify the responsible layer or component in your application.

Some recommendations that I'd make before starting out on this adventure.
1) Do try and use the 1.6 if you can. If not, try to use a release of the 1.4.2 or 1.5.0 that supports dumping of heap on out of memory exceptions. The flag you need to use for the 142 adn 150 looks something like -XX:+dumpheaponoutofmemoryerror.

2) reduce the overall size of your heap. You won't be able to load a 1.5gig heap dump ino any tool that you'd want to use to analyze it. Also you want fast failure and less memory is better in this case. Don't make it too small as you don't want a false failure. Use jhat of NetBeans profiler to take a look at what is in the heap dump.

3) try to isolate the request that is responsible for the increased memory utilization. Look at logs or what ever you have. Use -verbose:gc (-Xloggc:log.txt) to help you corrolate timings with your logs. You may want to look at HPJMeter (free download from HP) to help you analyze what is going on in memory. You can use the jconsole to follow along but the gc logs are easier for out of band analysis especially when looked at with HPJMeter.

4) If you happen to find an offending request, you can get it into a test environment where you can really study it using a profiler. Recommend either yourkit or NetBeans profiler with interest in either sites or generations information.

This not to say that I wouldn't recommend some sort of clustering or reasons of robustness. I just think that if you sell this as a solution to the current difficulties and it fails, it will be more difficult to recommend other remedies in the future.

--
Kind regards,
Kirk Pepperdine

http://www.kodewerk.com
http://www.javaperformancetuning.com
http//www.cretesoft.com

[Alan Kent]

Kirk Pepperdine wrote:
> Some recommendations that I'd make before starting out on this adventure.

Thanks for the tips, but I know exactly what the problem is and its not
a memory leak. I was trying to simplify here to keep the post short
(and not put people to sleep with gory details). At the risk of over
simplifying, let us say we get XML documents holding requests and we
need to convert to a DOM tree. The documents may be large (tens of
megabytes). The amount of memory required for the DOM tree is however
difficult to predict before you start the parse, as a document may have
very high density of tags, or it may have a low density of tags. There
may be more attributes, or may be less. Because we use a third party
XML parser (for example) its hard to predict given the input XML
document how much memory the parse tree is going to take up. In the
real application (which is not just XML) we have observed memory foot
prints from 5 times to 100 times the input document size - with no
guarantee it won't go more than 100 times. One 10mb document may
therefore need 50mb or 1gb of memory.

Further, there may concurrent requests going on in parallel. Normally
you would not get too big requests at the same time, but its certainly
possible. And there may be more than two.

We attempt to throttle the number of concurrent requests using thread
pools. We attempt to predict resource usage and hold back some
operations if we think we might run out of memory - but because its all
estimates, sometimes it still runs out because of estimation errors. We
currently have say 5 different sorts of things going on in the same
server at present (sharing some internal data structures), but the end
result is if one thing goes wrong, everything goes wrong. Ideally I
would like to limit the heap that can be used by a thread so I can give
a thread a fixed space to play in, so if that thread runs out of memory
it does not mean other threads will automatically run out of memory.

If I was writing all the code myself in something like C++, I would do
the memory management myself - allocating memory from fixed sized
pools. With Java, that control over memory management does not seem to
be available in the same way. Of course its made worse by the fact that
I want to use open source code which just uses 'new' as desired, so I
cannot (easily) get it to request memory from a pool even if supported
by Java.

So its not memory leaks I am concerned with - it is good resource
management practices in a JVM for highly threaded server applications
that I am after.

Thanks again!
Alan

[Mike Jones]

Is there any possibility of using SAX instead of DOM? It should use a
lot less memory.

[Jess Holle]

It would be best to avoid DOM as it is a resource pig.  Using SAX or StAX would be better.  XSLT is instrinsically based on DOM-like approaches (except for some single-pass cases which no XSLT engine that I'm aware of is smart enough to recognize) -- but most XSLT engines at least use a special reduced-size DOM internally.

All that said, if you can't get rid of DOM this is a tough problem -- one which I'd love answers for as well, albeit not due to DOM.

--
Jess Holle

[Kirk Pepperdine]

Ahh.. well this is why I start all of my talks with the performance advice you are about to get will be wrong! The devil is in the detailss ;-}

Ok, you may want to consider an XML appliance for this type of parsing.. that said, do you need the entire DOM all at one? If not SAX or StAX are both good alternatives (as has been mentioned here). The argument for DOM has always been simplicity but I believe this argument is based mostly on FUD as I've never found SAX to be so difficult to use. But that is another topic.

So on splitting, you may want to wind up another JVM just for the parsing (or keep a couple wound up in a pool). You can throw it away or reuse it when you are finished. I'd consider treating the VM pools as a "data grid" and remote in the DOM requests. This may not work for you but I think you can see that it goes along with your earlier thought that splitting things up will help.

[Alexey Zinger]

This situation reminds me of a problem I had way back when using Macromedia
Generator. The gist was that we had to make a Java call to the Macromedia
Generator library in our app server, but we realized that their stuff was
really designed to be run as a standalone tool. If kept alive in a JVM (in
order to save time on reloading it constantly), occasionally it would stop
responding. The solution we came to was to design a Generator abstraction API
and keep several JVM's alive, each of which linked to the Generator libraries.
These instances exposed themselves through RMI and were spawned by a central
manager instance, which I think we also called via RMI. The Generator manager
was responsible for not only distributing the calls round-robin style across
the spawned JVM's (the process was time consuming, as well as unstable), it
also throttled the jobs so as not to give any one worker JVM more than a
certain number of jobs at once, as well as kept track of the state of
responsiveness of each worker JVM. So if a job was not heard from in some
time, it would attempt to tell that worker JVM to cancel it without expecting a
response and give that job to another JVM. At the same time, it would try to
shut down the questionable JVM, at first through an RMI call, and if that
failed, it would kill the process, eventually replacing it with a new worker
instance. After we got all the aforementioned pieces in place, we had no
problem with it at all, while we were waiting on a more stable version of
Generator to come out.

Alexey
2001 Honda CBR600F4i (CCS)
1992 Kawasaki EX500
http://azinger.blogspot.com
http://bsheet.sourceforge.net
http://wcollage.sourceforge.net

____________________________________________________________________________________
Looking for last minute shopping deals?
Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping

[Viktor Klang]

think about taking advantage of Terracotta in this case.

 

-V

 

        /lift/ committer (www.liftweb.net)
      SGS member (Scala Group Sweden)
  SEJUG member (Swedish Java User Group)
\_____________________________________/

[Kirk Pepperdine]

• I like Terracotta very much. That said, I think DSO works best when you are trying to keep things the same where as this is a case where you want to keep things different. IOWs, I'm not sure that it is a fit in this case.

[patrickw]

It's been some years since I did this, but the dom4j framework has
specific techniques for handling large XML documents - you really
don't want to load the whole document into memory and then try and
process it!! I think that the distinction between sax and dom is not
relevant here - what you need is a framework that will allow you to
process the XML bit by bit, how it does that shouldn't be your
concern.

The dom4j FAQ says:
"dom4j provides an event based model for processing XML documents.
Using this event based model allows developers to prune the XML tree
when parts of the document have been successfully processed avoiding
having to keep the entire document in memory."

dom4j is here - http://www.dom4j.org.

Patrick

[Alan Kent]

patrickw wrote:
> It's been some years since I did this, but the dom4j framework has
> specific techniques for handling large XML documents - you really
> don't want to load the whole document into memory and then try and
> process it!!

Sorry - I regret mentioning DOM. I am not using DOM. I just used it as
an example to make the point that I have a situation where it is hard to
predict the memory usage of code given its inputs.

The question is how to best to manage a process servicing lots of
requests (and different types of requests) where its hard to predict the
memory usage of the requests. Is there anything better than "add more
swap and cross your fingers"? Are there any good architectural patterns
to follow? We are using things like thread pools to limit levels of
concurrency, but we almost want to dynamically adapt based on current
memory usage - *if* we could work out the memory usage of a request.
For example, pause a thread that seems to be using up lots of memory,
let some of the smaller requests finish, don't start any new requests,
then let the big one through and start allowing smaller requests back in.

At present I don't know how to predict memory usage given inputs. So
maybe the question is: is there a way to (efficiently) measure memory
usage of a request. More precisely, while a thread is running, can
another thread peek in a look and work out over time the memory usage of
another thread processing a request - then a scheduling / controlling
thread has information it needs to co-ordinate things. Or similar -
just tossing up an example of what I mean.

Thanks!
Alan

[Jess Holle]

At least with Sun's JVMs you can't really get any notion of memory usage on a per-thread basis.

You could look at overall memory usage when you're about to start a request and decide whether to have it wait, redirect to another JVM, or some such.

--
Jess Holle

[Alexey Zinger]

--- Alan Kent <ALAN....@saic.com> wrote:

> At present I don't know how to predict memory usage given inputs. So
> maybe the question is: is there a way to (efficiently) measure memory
> usage of a request. More precisely, while a thread is running, can
> another thread peek in a look and work out over time the memory usage of
> another thread processing a request - then a scheduling / controlling
> thread has information it needs to co-ordinate things. Or similar -
> just tossing up an example of what I mean.

In another response, I had given an example of managing jobs that have a
potential to bring down enclosing JVM's using an RMI broker process that starts
and stops these worker JVM's and gives them jobs. Obviously, if applied to
your problem, it would give you some ability to control allowed heap size for
each spawned worker JVM. The state of a particular JVM can be monitored
asynchronously using Runtime API to poll maximum and allocated memory numbers.
I'm not sure what you would do if you found out you were approaching the limit
of memory allocation, but you could conceivably run such a monitor thread.
Should you run out of memory, just have the broker spawn a new worker JVM with
more maximum memory allowed.

You could also attempt to make use of some predictive logic for the size of
needed memory, given the size of your XML data set. As you said, it's not a
linear correlation, but you could still try to come up with rough estimates,
provided you've taken care of OutOfMemoryError situations as described above.
So it would largely be a performance strategy.

[Viktor Klang]

Wouldn't it work with some SoftReference/WeakReference/PhantomReference-cache and then couple it with transactional behaviour.

 

So that if the JVM peaks the memory, it will randomly cannibalize some of the transactions, and then you'll have to re-try them later.
 

Might give you some overhead in retries, but you'll never crash the JVM cause of memory useage.

 

-V
 

        /lift/ committer (www.liftweb.net)

[Alexey Zinger]

If I'm not mistaken, Alan doesn't have that option, since he can never be sure
if trying to load the job into a 3rd party library he's using will result in
OutOfMemoryError, which likely would put his or other libraries in a bad state.

> --
> _____________________________________
> / \

[Viktor Klang]

On 1/25/08, Alexey Zinger <inlin...@yahoo.com> wrote:

If I'm not mistaken, Alan doesn't have that option, since he can never be sure
if trying to load the job into a 3rd party library he's using will result in
OutOfMemoryError, which likely would put his or other libraries in a bad state.

 

Yes, that is true if the 3rd party libraries do hard-references which would render the GC unable to collect the memory.

However, I strongly suspect that this kind of approach would solve his problems.

 

Cheers,

-V