JMS File Store Growing Causes Problem

Andrew Spyker

unread,

Aug 14, 2001, 9:42:26 AM8/14/01

to

All,

I found the following:
http://groups.google.com/groups?hl=en&safe=off&th=9c945704be74defd,2&seekm=3
b13f883%241%40newsgroups.bea.com#p

My problem is similar. I setup a JMS Queue for distributed logging/tracing.
I have a singleton interface in the client code to write to the Queue from
any place in the cluster. I also have a startup thread that reads the Queue
and puts the messages into the database via JDBC. Basically, I wanted a
guaranteed delivery fire and forget log/trace engine. I think JMS is
perfect for this.

The problem is the file store I defined for guaranteed delivery grows
without bound. We don't end up with a problem running out of memory while
the application is running (the queue seems to be processed as fast as
messages are added). We do however run into a problem as it seems like the
file store holds all messages until the end of time. Our real problem is
that given a long running server, or a server under considerable load, this
file grows to larger than the heap size of the server. If the server is to
crash, starting the server yields the stack trace below. It seems like the
server is trying to verify the messages and it is making sure none of them
need to be sent. The problem seems to be that the server tries to read them
all into memory while processing, thus the OutOfMemoryException.

Am I understanding this correctly? I can understand not trimming the file
and reusing "empty" space the next time the server starts, but reading the
whole file into memory seems to be braindead. I can't see how this will
ever work in a server that has significantly high load and then crashes and
must be restarted. Please tell me what I am missing!

Our setup is Solaris with WLS 6.0 SP1.

Thanks ...

---
Andrew Spyker (asp...@nc.rr.com)
Software Engineer
http://spyker.dyndns.org

Tom Barnes

unread,

Aug 14, 2001, 10:09:54 AM8/14/01

to Andrew Spyker

If your file store is growing, then your receivers are not keeping up, yet you
write below that they are. This suggests that your application may be
failing to ackowledge/commit received messages, a common mistake.
Or it may be that your application is not processing incoming messages
nearly as fast as you think it is. A file store should not grow if you are receiving
messages as fast as they are sent, it only grows proportional to the size of the
maximum number of current messages.

Check out your statistics for current and pending on the queue - if
current and/or pending do not stabilize but instead keep growing, then
this is the problem. Pending indicates the number of messages waiting
for acknowledge/commit, current indicates the number of messages waiting
for a client to receive them.

Tom

Tom Barnes

unread,

Aug 14, 2001, 10:41:38 AM8/14/01

to Andrew Spyker

> Am I understanding this correctly? I can understand not trimming the file
> and reusing "empty" space the next time the server starts, but reading the
> whole file into memory seems to be braindead. I can't see how this will
> ever work in a server that has significantly high load and then crashes and
> must be restarted. Please tell me what I am missing!

The empty space in a file is not read into memory on boot.

The file in a running system does not grow unless there is no blank space left
(i.e. messages are being sent faster than they are received).

The JMS boot does use somewhat more memory than the running system, but
not significantly more. This is being addressed, possibly in an upcoming
service pack, but I don't think this is your issue (see my other post).

Tom

Andrew Spyker

unread,

Aug 14, 2001, 11:11:27 AM8/14/01

to

Tom,

I will check this out. Thanks for keeping me honest.

I do still have the question of if the file grows, what happens at startup.
It would still seem like this is a problem. I could imagine the database
going away, and therefore JMS's file store would queue the messages. If
this situation persists for a long enough time, and the server gets shut
down, it would still have the same problem, correct?

Does the server code read the entire file's contents into memory causing
this OutOfMemoryException?

If so, I guess this takes us back to the fact that WLS can only handle a
certain amount of undelivered messages at a time as it keeps them in memory.

(Important) Do only undelivered messages go into the file store, or do all
messages go into the store until delivered?

How does the server decide whether or not to put the messages into the file
store - what rules does it follow to figure out the Queue is full?

If there is a good reference I should look at let me know.

--

Andrew Spyker (asp...@nc.rr.com)
Software Engineer
http://spyker.dyndns.org

"Tom Barnes" <dev....@not.my.address.com> wrote in message
news:3B793132...@not.my.address.com...

Tom Barnes

unread,

Aug 14, 2001, 1:58:37 PM8/14/01

to Andrew Spyker

Andrew Spyker wrote:

Tom,
I will check this out. Thanks for keeping me honest.
I do still have the question of if the file grows, what happens at startup.
It would still seem like this is a problem. I could imagine the database
going away, and therefore JMS's file store would queue the messages. If
this situation persists for a long enough time, and the server gets shut
down, it would still have the same problem, correct?
Does the server code read the entire file's contents into memory causing
this OutOfMemoryException?

on boot server scans file for records, skipping "empty" space

server uses more memory at boot as it reads in all records before
"normalizing" them, but not a lot more

there are plans to add "paging" ability to JMS 6.1, perhaps in
an upcoming service-pack - this will also address the problem
of the boot using more memory

If so, I guess this takes us back to the fact that WLS can only handle a
certain amount of undelivered messages at a time as it keeps them in memory.
(Important) Do only undelivered messages go into the file store, or do all
messages go into the store until delivered?

in general persistent messages go into the file store, there are some optimizations
which sometimes avoid persistence - but this is internal and an application can
not depend on them

in order to provide a dependable QOS, messages are persisted before
a non-transactional send returns or before a transactional send is committed.
This is a general requirement of queueing, not specific to JMS.

How does the server decide whether or not to put the messages into the file
store - what rules does it follow to figure out the Queue is full?

you may set quotas on a per queue or per server basis, either based on
number of messages or total payload size of messages - this is independent
of the file store

If there is a good reference I should look at let me know.

The WL JMS FAQ and WL JMS Developers guide are good references.
The best general JMS reference I know is "Professional JMS".