Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

JMS File Storage OutOfMemoryError

6 views
Skip to first unread message

Marcelo Quintella

unread,
Apr 9, 2002, 4:49:24 PM4/9/02
to
We are using Durable subscribers and have File Storage set in our server. We
also have Paging Store set. We are gettin the exception below. The storage
file is 40Mb. Anyone can help?

Thanks,
Marcelo Quintella

java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:360)
at weblogic.jms.store.FileIOFile.write(FileIOFile.java:133)
at weblogic.jms.store.FileIOStream.write(FileIOStream.java:467)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:250)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:182)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
End server side stack trace

at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:360)
at weblogic.jms.store.FileIOFile.write(FileIOFile.java:133)
at weblogic.jms.store.FileIOStream.write(FileIOStream.java:467)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:250)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:182)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
>

Larry Presswood

unread,
Apr 9, 2002, 5:17:52 PM4/9/02
to
Is this 6.0 ??

JMS in 6.0 needs to have all contents of messages in memory so perhaps
you need to increase your jvm start parameters to give the jvm more
memory.

Oh Paging Store set must mean you are 6.1??

Still do you have sufficient jvm memory for your server?

Marcelo Quintella

unread,
Apr 9, 2002, 5:21:10 PM4/9/02
to
Our JVM parameters are -ms768m -mx768m

That should be enough...

Marcelo

"Larry Presswood" <lpres...@charter.net> wrote in message
news:3CB35A80...@charter.net...

Raja Mukherjee

unread,
Apr 9, 2002, 5:35:42 PM4/9/02
to
Is your durable subscribers de-queuing the messages at all?

.raja

"Marcelo Quintella" <marcelo....@sknt.com> wrote in message
news:3cb35b47$1...@newsgroups.bea.com...

Tom Barnes

unread,
Apr 9, 2002, 5:59:17 PM4/9/02
to Marcelo Quintella
My gut reaction is that 768m should be plenty as well, although there
are edge cases.

The OOM exception may not be related to the store or even JMS in particular, it
is
just that the file store call was the straw that broke the camel's back. A
tool like OptimizIt is
useful for getting a complete view of memory allocation on the server, and can
often quickly tell where the memory is going. There are nice pointers in the
performance
newsgroup on these types of tools. The performance newsgroup has more
experience
with memory settings, and there may be a per-thread setting that would also help
you there.

One question is why you have paging if your file-store is only at 40MB...
Perhaps
you are paging non-persistent messages? Or perhaps paging isn't occurring
(yet)...

Another question is how many subscribers?
and how many messages?
average message size?

Marcelo Quintella

unread,
Apr 9, 2002, 6:02:52 PM4/9/02
to
Aren't the messages de-queued once they are acknowledged? We are using
AUTO_ACKNOWLEDGE.

Marcelo


"Raja Mukherjee" <no...@nowhere.com> wrote in message
news:3cb3...@newsgroups.bea.com...

Marcelo Quintella

unread,
Apr 9, 2002, 6:06:41 PM4/9/02
to
I think that paging is not happening yet, because there is no page file in
the directory that we assigned in the Weblogic console.

I don't know the average size. Most are small, like 100K or so. But some
messages are pretty big (5Mb sometimes). But these are less often.

About 5 to 10 subscribers per topic. About 100 topics.

Marcelo

"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message
news:3CB36435...@replyinnewsgroup.com...

Tom Barnes

unread,
Apr 9, 2002, 6:19:52 PM4/9/02
to Marcelo Quintella

Marcelo Quintella wrote:

> I think that paging is not happening yet, because there is no page file in
> the directory that we assigned in the Weblogic console.
>

Persistent messages are paged directly into the persistent store, not the page
store. (The message is already has to go to the store or is already in
the store, so there is no reason to write it twice.) There are log messages
printed when paging activates and de-activates.


>
> I don't know the average size. Most are small, like 100K or so. But some
> messages are pretty big (5Mb sometimes). But these are less often.
>
> About 5 to 10 subscribers per topic. About 100 topics.
>

Given this scenario, the vast majority of memory JMS uses is just
for the message bodies. Are all messages persistent? Are you
sure it is just 40MB on the server? Note that
non-persistent messages won't hit the store, even for durable
subscriptions...

What are your JMS stats for server message counts and byte counts?
This may give you a quick and dirty estimate of memory usage.
There is a program on the dev2dev site that makes this easy
to get in one dump: "JMSStats.java".

One common design suggestion is to have clients compress messages
prior to submitting. Text/XML messages compress hugely.

Tom Barnes

unread,
Apr 9, 2002, 6:24:40 PM4/9/02
to Marcelo Quintella
There is a scenario where the ack won't occur:
The client app listener code is throwing a RuntimeException.

Also what sometimes can happen is the application has
an "orphaned" durable subscription that it is failing to service.
This durable subscription then sits there accumulating messages,
never allowing them to be cleaned up. The "JMSStats.java"
program enumerates durable subscriptions.

Marcelo Quintella

unread,
Apr 9, 2002, 6:38:18 PM4/9/02
to
All our messages are publiched with DeliveryMode.NON_PERSISTENT

I'll try the JMSStats.java to give you more details.

Marcelo

"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message

news:3CB36908...@replyinnewsgroup.com...

Marcelo Quintella

unread,
Apr 9, 2002, 6:40:24 PM4/9/02
to

"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message
news:3CB36A28...@replyinnewsgroup.com...

> Also what sometimes can happen is the application has
> an "orphaned" durable subscription that it is failing to service.
> This durable subscription then sits there accumulating messages,
> never allowing them to be cleaned up.

Even if the time to live is something like 3 minutes? Shouldn't the messages
be cleaned up after 3 minutes even if the durable subscription is failing?

Marcelo


Tom Barnes

unread,
Apr 10, 2002, 10:58:01 AM4/10/02
to Marcelo Quintella

Marcelo Quintella wrote:

No. Expired messages are deleted when they are accessed during
a subscribe operation and during boot, not spontaneously.

>
> Marcelo

Tom Barnes

unread,
Apr 10, 2002, 11:05:35 AM4/10/02
to Marcelo Quintella
Non-persistent messages do not go into the persistent store. So I do not
see how your persistent store could grow to 40MB. Your persistent
store should be small, as in this case it only records the existence of the
durable
subscriptions, not the messages.

If paging is active and enabled for non-persistent messages,
messages go into the seperately configured non-persistent paging store.
Make sure you have configured a non-persistent paging store -
(includes defining the file location, turning on paging, and setting
reasonable thresholds.)

I think it is quite likely you are running out of memory because your
subscribers are not keeping up, and you have not configured message
quotas. And according to your previous posts, paging has not turning on.

Tom Barnes

unread,
Apr 10, 2002, 11:07:21 AM4/10/02
to Tom Barnes
I'm curious, why are using durable subscriptions at all if your messages
are non-persistent with a three minute timeout? Are the subscriptions
shared between applications?

Marcelo Quintella

unread,
Apr 10, 2002, 1:40:44 PM4/10/02
to
Because we are having a problem that we are getting PeerGoneExceptions in
the client without the server really going down. Moreover, only the JMS goes
down, the t3 connection is still healthy and the application works just
fine. Only the subscribers stop receiving JMS messages.

This looked like network issues in the client side, but the fact that only
JMS went down was weird. After a lengthy support case with BEA we decided
that the only way to workaround this JMS instability would be to try to
reconnect the JMS subscribers every time that a PeerGoneException happened.

But we need that every message be delivered to the client until it logs out
of the system. So we use durable subscribers in case a message is published
while a subscriber is in this process of reconnection. We figured that 3
minutes would be more than enough time.

I have another message in this newsgroups with the subject of "JMS
Instability" where I describe the problem.

Marcelo

"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message

news:3CB45529...@replyinnewsgroup.com...

Marcelo Quintella

unread,
Apr 11, 2002, 9:55:34 AM4/11/02
to
Ok. But what I mean is that, we we have a short time to live, the messages
should get cleaned up at some point. Even if the subscriber failed to
acknowledge it. Is that right?

Marcelo

"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message

news:3CB452F9...@replyinnewsgroup.com...

Marcelo Quintella

unread,
Apr 11, 2002, 10:00:44 AM4/11/02
to
I turned off all our durable subscribers in production (they are now
no-durable). I still got the OutOfMemory with the stack trace pointing to
JMSStore.

So I removed the storage file from the JMS Server section in the
weblogic console. I hope that this will make the out of memory go away until
I can fix it in our test environment.

But (just out of curiosity), if I do not have any durable subscribers
and all my publish are done with DeliveryMode.NON_PERSISTENT mode, why
would the JMSStore be called at all?

Thanks,
Marcelo Quintella


java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:360)
at weblogic.jms.store.FileIOFile.write(FileIOFile.java:133)
at weblogic.jms.store.FileIOStream.write(FileIOStream.java:467)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:250)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:182)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
End server side stack trace

at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:360)
at weblogic.jms.store.FileIOFile.write(FileIOFile.java:133)
at weblogic.jms.store.FileIOStream.write(FileIOStream.java:467)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:250)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:182)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message
news:3CB454BF...@replyinnewsgroup.com...

Tom Barnes

unread,
Apr 11, 2002, 10:46:44 AM4/11/02
to Marcelo Quintella
They will get cleaned up the next time a durable subscriber re-attaches and
or
on the next reboot.

Tom Barnes

unread,
Apr 11, 2002, 11:38:53 AM4/11/02
to Marcelo Quintella

Marcelo Quintella wrote:

    I turned off all our durable subscribers in production (they are now
no-durable). I still got the OutOfMemory with the stack trace pointing to
JMSStore.

    So I removed the storage file from the JMS Server section in the
weblogic console. I hope that this will make the out of memory go away until
I can fix it in our test environment.

    But (just out of curiosity), if I do not have any durable subscribers
and all my publish are done with DeliveryMode.NON_PERSISTENT  mode, why
would the JMSStore be called at all?
 

As I wrote before.  The JMS store stores the existence of durable subscriptions, but not
non-persistent messages themselves.   The non-persistent paging store holds
non-persistent paged-out messages if non-persistent paging is configured and
the configured paging threshold has been reached.

If your messages are truly  non-persistent, none should  be ending up  in the
regular JMS store.   Period.  If your store is growing with each message,
it looks like your senders are generating persistent messages without realizing it.
This of course slows down performance considerably, and may be the reason
your subscribers couldn't keep up with your consumers.   Here is where
persistence is set in order of precedence:
    Default persistence:           Configurable on connection factory, defaults to "PERSISTENT"

    Override on Producer:      Set on producer via setDeliveryMode()

    Override on produce:       Set on some variants of send/publish method.

    Override on destination:    Configurable on destination via "DeliveryMode" setting.  (default is no-override)

    Override on server:          If no store is configured, message is downgraded to NON-PERSISTENT.
                                           This follows the JMS spec, I believe JMS 7.0 may have changed this
                                           behavior and throw a JMSException.    Note that durable subscribers can
                                           not be created on such a JMS server, as there is no place to store the existence
                                           of the durable subscription.

    Non-durable subscribers and temporary destinations:
                                           Always convert their messages to NON-PERSISTENT.
                                            (No need to store persistently as object itself is not persistent)

Marcelo Quintella

unread,
Apr 11, 2002, 1:05:49 PM4/11/02
to
OK.... although we did not have any durable subscribers we did sent out some
messages in the default delivery mode. So now I changed our factories to
have the default delivery mode of NOT_PERSISTENT. I think we were assuming
the default was that...

One question: sometimes we use the default factory
(weblogic.jms.client.JMSConnectionFactory, from
jndiContext.lookup("javax.jms.TopicConnectionFactory"). Is there a way to
override the default setting of this factory from the weblogic console? (To
avoid doing it in a per-publisher basis).

Or should we use a default factory of our own and set this property there?
Maybe that makes more sense....

Thanks,
Marcelo


"Marcelo Quintella" <marcelo....@sknt.com> wrote in message

news:3cb3...@newsgroups.bea.com...

Tom Barnes

unread,
Apr 11, 2002, 1:20:59 PM4/11/02
to Marcelo Quintella
>
> One question: sometimes we use the default factory
> (weblogic.jms.client.JMSConnectionFactory, from
> jndiContext.lookup("javax.jms.TopicConnectionFactory"). Is there a way to
> override the default setting of this factory from the weblogic console? (To
> avoid doing it in a per-publisher basis).
>

No.

>
> Or should we use a default factory of our own and set this property there?
> Maybe that makes more sense....
>

That is one way. Or you can override on the destination itself (as I wrote in
the other post),
or you could not configure the store at all (as I wrote in the other post).


Marcelo Quintella

unread,
Apr 11, 2002, 1:22:49 PM4/11/02
to
Yes. The two other options I understood. I just wanted to use a default
factory as well.

Thanks for your help. This might solve the storage issue. I'll get back if
it doesn't.

Thanks a lot.


This might solve our probles with

"Tom Barnes" <dev....@not.my.address.com> wrote in message
news:3CB5C5FB...@not.my.address.com...

0 new messages