Thanks,
Marcelo Quintella
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:360)
at weblogic.jms.store.FileIOFile.write(FileIOFile.java:133)
at weblogic.jms.store.FileIOStream.write(FileIOStream.java:467)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:250)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:182)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
End server side stack trace
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:360)
at weblogic.jms.store.FileIOFile.write(FileIOFile.java:133)
at weblogic.jms.store.FileIOStream.write(FileIOStream.java:467)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:250)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:182)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
>
JMS in 6.0 needs to have all contents of messages in memory so perhaps
you need to increase your jvm start parameters to give the jvm more
memory.
Oh Paging Store set must mean you are 6.1??
Still do you have sufficient jvm memory for your server?
That should be enough...
Marcelo
"Larry Presswood" <lpres...@charter.net> wrote in message
news:3CB35A80...@charter.net...
.raja
"Marcelo Quintella" <marcelo....@sknt.com> wrote in message
news:3cb35b47$1...@newsgroups.bea.com...
The OOM exception may not be related to the store or even JMS in particular, it
is
just that the file store call was the straw that broke the camel's back. A
tool like OptimizIt is
useful for getting a complete view of memory allocation on the server, and can
often quickly tell where the memory is going. There are nice pointers in the
performance
newsgroup on these types of tools. The performance newsgroup has more
experience
with memory settings, and there may be a per-thread setting that would also help
you there.
One question is why you have paging if your file-store is only at 40MB...
Perhaps
you are paging non-persistent messages? Or perhaps paging isn't occurring
(yet)...
Another question is how many subscribers?
and how many messages?
average message size?
Marcelo
"Raja Mukherjee" <no...@nowhere.com> wrote in message
news:3cb3...@newsgroups.bea.com...
I don't know the average size. Most are small, like 100K or so. But some
messages are pretty big (5Mb sometimes). But these are less often.
About 5 to 10 subscribers per topic. About 100 topics.
Marcelo
"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message
news:3CB36435...@replyinnewsgroup.com...
> I think that paging is not happening yet, because there is no page file in
> the directory that we assigned in the Weblogic console.
>
Persistent messages are paged directly into the persistent store, not the page
store. (The message is already has to go to the store or is already in
the store, so there is no reason to write it twice.) There are log messages
printed when paging activates and de-activates.
>
> I don't know the average size. Most are small, like 100K or so. But some
> messages are pretty big (5Mb sometimes). But these are less often.
>
> About 5 to 10 subscribers per topic. About 100 topics.
>
Given this scenario, the vast majority of memory JMS uses is just
for the message bodies. Are all messages persistent? Are you
sure it is just 40MB on the server? Note that
non-persistent messages won't hit the store, even for durable
subscriptions...
What are your JMS stats for server message counts and byte counts?
This may give you a quick and dirty estimate of memory usage.
There is a program on the dev2dev site that makes this easy
to get in one dump: "JMSStats.java".
One common design suggestion is to have clients compress messages
prior to submitting. Text/XML messages compress hugely.
Also what sometimes can happen is the application has
an "orphaned" durable subscription that it is failing to service.
This durable subscription then sits there accumulating messages,
never allowing them to be cleaned up. The "JMSStats.java"
program enumerates durable subscriptions.
I'll try the JMSStats.java to give you more details.
Marcelo
"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message
news:3CB36908...@replyinnewsgroup.com...
Even if the time to live is something like 3 minutes? Shouldn't the messages
be cleaned up after 3 minutes even if the durable subscription is failing?
Marcelo
Marcelo Quintella wrote:
No. Expired messages are deleted when they are accessed during
a subscribe operation and during boot, not spontaneously.
>
> Marcelo
If paging is active and enabled for non-persistent messages,
messages go into the seperately configured non-persistent paging store.
Make sure you have configured a non-persistent paging store -
(includes defining the file location, turning on paging, and setting
reasonable thresholds.)
I think it is quite likely you are running out of memory because your
subscribers are not keeping up, and you have not configured message
quotas. And according to your previous posts, paging has not turning on.
This looked like network issues in the client side, but the fact that only
JMS went down was weird. After a lengthy support case with BEA we decided
that the only way to workaround this JMS instability would be to try to
reconnect the JMS subscribers every time that a PeerGoneException happened.
But we need that every message be delivered to the client until it logs out
of the system. So we use durable subscribers in case a message is published
while a subscriber is in this process of reconnection. We figured that 3
minutes would be more than enough time.
I have another message in this newsgroups with the subject of "JMS
Instability" where I describe the problem.
Marcelo
"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message
news:3CB45529...@replyinnewsgroup.com...
Marcelo
"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message
news:3CB452F9...@replyinnewsgroup.com...
So I removed the storage file from the JMS Server section in the
weblogic console. I hope that this will make the out of memory go away until
I can fix it in our test environment.
But (just out of curiosity), if I do not have any durable subscribers
and all my publish are done with DeliveryMode.NON_PERSISTENT mode, why
would the JMSStore be called at all?
Thanks,
Marcelo Quintella
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:360)
at weblogic.jms.store.FileIOFile.write(FileIOFile.java:133)
at weblogic.jms.store.FileIOStream.write(FileIOStream.java:467)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:250)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:182)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
End server side stack trace
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:360)
at weblogic.jms.store.FileIOFile.write(FileIOFile.java:133)
at weblogic.jms.store.FileIOStream.write(FileIOStream.java:467)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:250)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:182)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message
news:3CB454BF...@replyinnewsgroup.com...
Marcelo Quintella wrote:
I turned off all our durable subscribers in production (they are now
no-durable). I still got the OutOfMemory with the stack trace pointing to
JMSStore.So I removed the storage file from the JMS Server section in the
weblogic console. I hope that this will make the out of memory go away until
I can fix it in our test environment.But (just out of curiosity), if I do not have any durable subscribers
and all my publish are done with DeliveryMode.NON_PERSISTENT mode, why
would the JMSStore be called at all?
As I wrote before. The JMS store stores the existence of durable
subscriptions, but not
non-persistent messages themselves. The non-persistent
paging store holds
non-persistent paged-out messages if non-persistent paging is
configured and
the configured paging threshold has been reached.
If your messages are truly non-persistent, none should be
ending up in the
regular JMS store. Period. If your store is growing
with each message,
it looks like your senders are generating persistent messages without
realizing it.
This of course slows down performance considerably, and may be the
reason
your subscribers couldn't keep up with your consumers.
Here is where
persistence is set in order of precedence:
Default persistence:
Configurable on connection factory, defaults to "PERSISTENT"
Override on Producer: Set on producer via setDeliveryMode()
Override on produce: Set on some variants of send/publish method.
Override on destination: Configurable on destination via "DeliveryMode" setting. (default is no-override)
Override on server:
If no store is configured, message is downgraded to NON-PERSISTENT.
This follows the JMS spec, I believe JMS 7.0 may have changed this
behavior and throw a JMSException. Note that durable
subscribers can
not be created on such a JMS server, as there is no place to store the
existence
of the durable subscription.
Non-durable subscribers and temporary destinations:
Always convert their messages to NON-PERSISTENT.
(No need to store persistently as object itself is not persistent)
One question: sometimes we use the default factory
(weblogic.jms.client.JMSConnectionFactory, from
jndiContext.lookup("javax.jms.TopicConnectionFactory"). Is there a way to
override the default setting of this factory from the weblogic console? (To
avoid doing it in a per-publisher basis).
Or should we use a default factory of our own and set this property there?
Maybe that makes more sense....
Thanks,
Marcelo
"Marcelo Quintella" <marcelo....@sknt.com> wrote in message
news:3cb3...@newsgroups.bea.com...
No.
>
> Or should we use a default factory of our own and set this property there?
> Maybe that makes more sense....
>
That is one way. Or you can override on the destination itself (as I wrote in
the other post),
or you could not configure the store at all (as I wrote in the other post).
Thanks for your help. This might solve the storage issue. I'll get back if
it doesn't.
Thanks a lot.
This might solve our probles with
"Tom Barnes" <dev....@not.my.address.com> wrote in message
news:3CB5C5FB...@not.my.address.com...