Weblogic JMS Peformance
Vincent
I recently ran some tests using Weblogic 8.1 and I have some numbers
that I would like to share.
I am using the grinder test tool and test cases as described in the
J2EE performance book and from the Weblogic JMS performance tuning
whitepaper.
Hardware:
Server hardware: 4 Pentium III Xeons (500 mhz), with 768 mb of ram.
O/S: Red Hat Linux 8.0, Kernel: 2.4.18-27.8.0smp
JDK: build 1.4.1_02-er-20030219 Server VM
VM option: -Xms128m -Xmx256m -XX:PermSize=32m -XX:SurvivorRatio=12
-Dweblogic.management.discover=false -Dweblogic.PosixSocketReaders=1
WLS: v8.1
Client client: 4 Pentium III Xeons (500 mhz), with 512 mb of ram
O/S: Red Hat Linux 8.0, Kernel: 2.4.18-27.8.0smp
JDK: build 1.4.1_02-er-20030219 Server VM
VM option: none
WLS: v8.1
JMS setup:
100 p2p queues
NO persistence delivery used
Receiver ack mode: WLSession.NO_ACKNOWLEDGE
Sender ack mode: AUTO
Message type is Bytes
Message size is 1kb
I have setup 100 producers and 100 consumers running on a client
machine. JMS connections are created once and JMS sessions are reused
per cycle. T3 connections were used using 100mbs network. Both
machines are on the same network segment and no firewall was used.
The best consumption rate I can achieve is 100 messages per second.
Is it a good number?
If I reduce the message to 100 byte.. The consumption rate would
increase to about 850 messages per second.
Does that mean we need to increase network bandwidth to use giga
network (assuming the problem is network bandwidth)? That might not
be possible because upgrade cost would be very expensive.
I haven't tried increasing the client side JMS thread pool..would this
help?
Is there anything I could do to improve the performance?
TIA,
Vincent
P.S.
I opened a case with with support but during our stress test, if
producer rate is much higher than consumer rate, I would get lots of
java.rmi.UnmarshalExceptions:
Start server side stack trace:
java.rmi.UnmarshalException: error unmarshalling
net.grinder.plugininterface.PluginException: Failed to send message,
nested exception: weblogic.jms.common.JMSException: error
unmarshalling arguments; nested exception is:
java.io.StreamCorruptedException: Unsupported class version 11.
Expected a value between 1 and 2 inclusive. Possible attempt to
access newer JMS version then current version.
Start server side stack trace:
java.rmi.UnmarshalException: error unmarshalling arguments; nested
exception is:
java.io.StreamCorruptedException: Unsupported class version 11.
Expected a value between 1 and 2 inclusive. Possible attempt to
access newer JMS version then current version.
at weblogic.jms.dispatcher.DispatcherImpl_WLSkel.invoke(Unknown
Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:407)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:356)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:123)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:351)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:178)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:151)
Caused by: java.io.StreamCorruptedException: Unsupported class version
11. Expected a value between 1 and 2 inclusive. Possible attempt to
access newer JMS version then current version.
at weblogic.jms.common.JMSUtilities.versionIOException(JMSUtilities.java:111)
at weblogic.jms.common.BytesMessageImpl.readExternal(BytesMessageImpl.java:673)
at weblogic.jms.frontend.FEProducerSendRequest.readExternal(FEProducerSendRequest.java:156)
at weblogic.common.internal.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:102)
at weblogic.rjvm.MsgAbbrevInputStream.readObject(MsgAbbrevInputStream.java:95)
... 9 more
End server side stack trace
; nested exception is:
java.io.StreamCorruptedException: Unsupported class version 11.
Expected a value between 1 and 2 inclusive. Possible attempt to
access newer JMS version then current version.
Start server side stack trace:
java.io.StreamCorruptedException: Unsupported class version 11.
Expected a value between 1 and 2 inclusive. Possible attempt to
access newer JMS version then current version.
at weblogic.jms.common.JMSUtilities.versionIOException(JMSUtilities.java:111)
at weblogic.jms.common.BytesMessageImpl.readExternal(BytesMessageImpl.java:673)
at weblogic.jms.frontend.FEProducerSendRequest.readExternal(FEProducerSendRequest.java:156)
at weblogic.common.internal.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:102)
at weblogic.rjvm.MsgAbbrevInputStream.readObject(MsgAbbrevInputStream.java:95)
at weblogic.jms.dispatcher.DispatcherImpl_WLSkel.invoke(Unknown
Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:407)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:356)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:353)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:123)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:351)
at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:178)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:151)
End server side stack trace
Tom Barnes
Date: Fri, 02 May 2003 11:15:07 -0400
From: Tom Barnes <ple...@replyinnewsgroup.com>
Reply-To: ple...@replyinnewsgroup.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.0.2) Gecko/20030208 Netscape/7.02
X-Accept-Language: en-us, en
MIME-Version: 1.0
Newsgroups: weblogic.developer.interest.jms
To: Vincent <vs...@myrealbox.com>
Subject: Re: Weblogic JMS Peformance
References: <c8855f85.03050...@posting.google.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
NNTP-Posting-Host: 216.148.48.18
X-Original-NNTP-Posting-Host: 216.148.48.18
X-Trace: newsgroups.bea.com 1051888511 216.148.48.18 (2 May 2003 08:15:11 -0800)
X-Original-Trace: 2 May 2003 08:15:11 -0800, 216.148.48.18
Organization: BEA NEWS SITE
Lines: 160
XPident: Unknown
Path: newsgroups.bea.com!not-for-mail
Xref: newsgroups.bea.com weblogic.developer.interest.jms:11885
Hi Vincent,
Back-of-the- envelope calculation for the receive data rates
alone:
100 msgs/sec * 100 receivers * 1000 bytes * 8 bits/byte
= 80 mbits/sec
Without even including any overhead in the calculation, it
looks like your network is the bottleneck. The fact that
100 byte messages produce roughly the same aggregate bit
rate (85m bits/sec) seems to support this theory. Are all
machines plugged directly into a switch? Or do some machines
share a hub? I would hazard a guess that your CPUs are not pegged -
this would be another clue that the network is the bottleneck.
If there is a network bottleneck there is little
point in tuning thread pool sizes - it won't help. If the
network isn't the bottleneck there are some tunables
that may help. Client thread pool (not JMS), use async
consumers, increase MessagesMaximum on CF,
default thread pool on server, and JMS thread pool on
server. (Increasing MessagesMaximum may aid perf
if network is bottleneck, as it reduces the use
of bandwidth somewhat.)
Tom
P.S. The unmarshal exception is due to a bug in the grinder,
specifically it is using the same message in multiple sender threads.
Multi-threading a message is not supported by the JMS spec
for various reasons - so the resulting behavior is undefined.
Zdrozny knows of the bug, so I suspect there should be a patch
on the tool's website, if not now then soon.
Vincent
Your guess is correct, the CPU is not pegged at all. I might have
found a problem with my grinder setup as I might have incorrectly put
a sleep() in my sender code. I will re-run my test and post my
result.
Vincent
Vincent
It's similar to the grinder test that I did earlier except with the
following changes:
1) no more grinder exceptions..;-)
2) each thread can use its own JMS connection
3) I make sure there is no calls to sleep() for senders/receivers
Here's the test case I use:
- 100 receivers and 100 senders are used
- each receiver and sender uses its own connections for a total 200
JMS connections
- session object is "recycled" for each sender/receiver
- message is 1k bytes
- receivers are async
Result:
Sender rate: 1853 messages per second
Receiver rate: 1850 messages per second
I rerun the same test on a different client machine. This time, the
100 receivers/senders are running on a single P4 2.0G with 512MB ram
desktop PC(win2k)(instead of a 4 CPU xeons running linux). The JMS
server is still running on the same server grade machine (4 PIII
xeons). The P4 is running on a different network segment and it's
routed by a router instead of a switch.
Result:
Sender rate: 1165 messages per second
Receiver rate: 643 messages per second
Now the shocking number:
I ran the same test but this time the senders/receivers are running on
a PIII 800mhz, 512MB ram desktop PC with win2k. It has the same
network setting as the P4 machine.
Result:
Sender rate: 850 message per second
Recieve rate: 16 messages per second
16 mps!!!!! Since I didn't use message paging, the server quickly ran
out of memory and I have to abort the test.
I reduce the numbers of senders and receivers to 10 each and the
number didn't improve by much:
Receive rate is 26 messages per second.
The server again ran out of memory quickly.
I reduced the number of sender/receive to 1 and got the following
result:
Sender rate: 77 messages per second
Receive rate: 77 messages per second
No out of memory error this time as the send/receive rate is more less
equal.
I then rerun the the test on the PIII with 100 senders/receivers but
this time the message size is reduced to 100 bytes. Result:
Sender rate: 694 mps
Receive rate: 274 mps.
I ran perfmon on the PIII machine and at no time the network
bandwidth/CPU/memory was maxed out..
P.S.
Tom, I think the network bandwidth calculation was not right in your
post. The mps was for all receivers so I don't think we need to
multiply # of receivers to get the total.
total bandwidth should be just (receiver mps) * message size (in bits)
Hopes this help!
Vincent
Tom Barnes
As I recall, the Sonic test harness is tuned for Sonic. (Hmmm.
I wonder why?). Sonic automatically configures quotas and
blocking sends. The test harness has a tendency to burn CPU
on quota failures and to max out send rates. So,
a couple of comments:
1) configure quotas on the BEA JMS server and a
long "blocking" send time on the connection factory
(configure
relatively low quotas so the senders do not
over-run the receivers by much - this is how the harness
expects to "throttle" the senders.)
2) with many threaded clients make sure that
you have enough internal threads on the client, this can
have a very large impact on a WL client (see the JMS
Performance white-paper, its a -D parameter)
3) make sure that you have not disabled native socket
IO on the server
The low rates you are seeing on one particular test
box are strange - I have no explanation, except perhaps number
(2) above.
Tom
BEA Internal Newsgroups
Newsgroups: weblogic.developer.interest.jms
References: <c8855f85.03050...@posting.google.com> <3EB28B7B...@replyinnewsgroup.com> <c8855f85.03050...@posting.google.com> <c8855f85.0305...@posting.google.com> <3EB66AA4...@replyinnewsgroup.com>
Subject: Re: Weblogic JMS Peformance
Lines: 132
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2800.1106
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
NNTP-Posting-Host: 216.148.48.18
X-Original-NNTP-Posting-Host: 216.148.48.18
Message-ID: <3eb7...@newsgroups.bea.com>
X-Trace: newsgroups.bea.com 1052240307 216.148.48.18 (6 May 2003 09:58:27 -0800)
X-Original-Trace: 6 May 2003 09:58:27 -0800, 216.148.48.18
Organization: BEA NEWS SITE
XPident: Unknown
Path: newsgroups.bea.com!not-for-mail
Xref: newsgroups.bea.com weblogic.developer.interest.jms:11915
ON linux, you may also want to play with the MTU size, that has major
impacts. 1500 seems to work pretty good. Large MTU has some known issue.
.raja
"Tom Barnes" <ple...@replyinnewsgroup.com> wrote in message
news:3EB66AA4...@replyinnewsgroup.com...