omniORB and Ice performance compared
Duncan Grisby
At the risk of further fanning the flamewar, I thought I would do some
performance comparisons between Ice and omniORB. ZeroC make a big deal
out of Ice's performance -- it's mentioned before anything else in the
description of Ice on the ZeroC web site -- so it's good to see how it
really performs. I don't want to imply that performance is the only
important thing -- whether Ice is better or worse than CORBA in other
respects is a separate issue.
To do the comparison, I used the latency and throughput demos from the
Ice distribution. It didn't take much modification to turn them into
CORBA programs.
The performance tests were done on a machine with an Intel Pentium 4
3.2 GHz hyper-threaded processor, running Fedora Core 3 Linux. Both
Ice and omniORB were compiled with g++ 3.4.4, with the -O3
optimisation level. The tests run on a single machine, between two
processes. I have only tested C++ so far. I might do some Python
comparisons later.
I used the latest stable releases of Ice and omniORB: 3.0.0 and 4.0.6
respectively.
Both Ice and omniORB are run with no special configuration, so they
use the default options for everything. For omniORB, I tried both the
default TCP transport and the Unix domain socket transport.
So, the results:
The latency test just pings an object. I ran each test three times and
picked the best result:
Ice tcp: time for 100000 pings: 6926.86ms
omniORB tcp: time for 100000 pings: 4566.58ms
omniORB unix: time for 100000 pings: 2373.28ms
omniORB is over 1.5 times faster using TCP, and more than 2.9 times
faster using Unix sockets.
The throughput test looks at the time taken to transfer large
sequences of various types. Each type is tested four ways: sending as
an argument, sending oneway as an argument, receiving as a return
value, and echoing.
The values shown are a throughput in MBit/s, calculated by the test.
Larger numbers are better.
Test Ice omniORB tcp omniORB unix
byte sequence
send 1180.1 2433.7 4788.8
oneway 1321.0 2504.5 4267.8
return 777.5 1184.1 1554.3
echo 950.7 1501.6 2250.3
string sequence
send 78.8 82.9 86.7
oneway 81.6 83.7 88.8
return 60.7 52.2 53.6
echo 66.2 64.4 67.9
struct: string and double
send 178.6 186.6 210.5
oneway 181.4 193.7 211.7
return 129.0 121.9 130.9
echo 145.8 150.0 159.9
struct: two longs and a double
send 611.4 1301.6 2118.0
oneway 567.7 1310.0 2059.1
return 486.7 1052.0 1520.8
echo 545.8 1159.4 1758.5
As you can see, omniORB mostly beats Ice, With fixed length types, it
beats it significantly. Ice wins on a couple of the string sequence
tests and one of the tests with a struct containing a string and a
double.
I think the results with strings are more an artifact of the way the
tests work than anything else. All the tests pre-build the sequences
they are going to transfer. Ice sensibly uses C++ standard strings to
represent the strings; CORBA just uses C-style char* arrays (plus
holder objects for deallocation). Constructing a C++ string is more
expensive than constructing a C string, but once it has been
constructed its length has been pre-calculated. Marshalling a sequence
of existing C++ strings therefore only requires one pass over each
string, while C-style strings require two passes. In a real
application, the strings would probably be constructed during
operation of the program, rather than right at the start, so I'm not
sure this difference would appear in many real situations.
I hope some people find this interesting.
Cheers,
Duncan.
--
-- Duncan Grisby --
-- dun...@grisby.org --
-- http://www.grisby.org --
Siegfried Rohdewald
...
> The tests run on a single machine, between two
> processes. I have only tested C++ so far. I might do some Python
> comparisons later.
>
How does the comparison look if the two processes do not run on the
same machine, but each process on a different machine ?
Siegfried Rohdewald.
Juergen Weber
that is very interesting.
Maybe somebody would care to repeat the tests using webservices ;-)
The other day I ran a test with a huge real world IDL (lots of nested
structures and sequences), request message size 1K, reply 64K using the
old JDK 1.3.1 implementation (server did nothing but fill the data
structures from dummy data). The round-trip between different machines
took about 70ms, which is about 10^4 slower than your ping numbers, but
still very fast, imho.
Juergen
Rob Ratcliff
machines and a big endian and a little endian machine just to see how
much byte swapping impacts the performance. (Of course CORBA uses
"receiver makes right" and Ice always uses little endian, so I'd expect
to see different trends from each approach.)
Steve Wampler
On Tue, 22 Nov 2005 12:12:55 +0000, Duncan Grisby wrote:
> Both Ice and omniORB are run with no special configuration, so they
> use the default options for everything. For omniORB, I tried both the
> default TCP transport and the Unix domain socket transport.
This decision concerns me, as the default configurations may
have vastly different philosophies. As an example, the
PostgreSQL database comes configured for operating on small,
slow box - you *have* to reconfigure it to get *any* sort of
decent performance on a typical machine really used for
databases these days. It would be much more useful to see
these comparisons done with both Ice and oO *fully* tuned
for performance.
How does oO 4.0.l6 compare against other ORBs? Specifically,
TAO (which Ice uses in their comparisons) and oO 3.x? If
oO 4.0.6's performance against TAO is consistent with
Ice's performance against TAO, then the above oO vs Ice
numbers have an increased 'validity potential'. If they
don't (for example, if TAO is faster...), then I'm still
left wondering whose version of the performance tests were
done 'right'.
Marc Laukien
performed. omniORB is definitely a well-engineered product, with some
very clever concurrency models. However, the concurrency models are also
not directly comparable to what Ice offers. For example, Ice uses a
leader/follower thread pool both for the server side and the client side
(for AMI callbacks). Whether or not the concurrency model difference is
relevant depends on your application.
I'd also like to point out that we have never claimed that Ice is faster
than omniORB. I know omniORB back from my CORBA days, and it has always
been known to be one of the fastest CORBA implementations.
-- Marc
Douglas C. Schmidt
++ On Tue, 22 Nov 2005 12:12:55 +0000, Duncan Grisby wrote:
++ > Both Ice and omniORB are run with no special configuration, so they
++ > use the default options for everything. For omniORB, I tried both the
++ > default TCP transport and the Unix domain socket transport.
++
++ This decision concerns me, as the default configurations may
++ have vastly different philosophies. As an example, the
++ PostgreSQL database comes configured for operating on small,
++ slow box - you *have* to reconfigure it to get *any* sort of
++ decent performance on a typical machine really used for
++ databases these days. It would be much more useful to see
++ these comparisons done with both Ice and oO *fully* tuned
++ for performance.
++
++ How does oO 4.0.l6 compare against other ORBs? Specifically,
++ TAO (which Ice uses in their comparisons) and oO 3.x? If
++ oO 4.0.6's performance against TAO is consistent with
++ Ice's performance against TAO, then the above oO vs Ice
++ numbers have an increased 'validity potential'. If they
++ don't (for example, if TAO is faster...), then I'm still
++ left wondering whose version of the performance tests were
++ done 'right'.
omniORB is widely considered as one of the fastest ORBs around. I
recommend you check out
http://www.atl.external.lmco.com/projects/QoS/compare/dist_oo_compare_ipc.html
http://nenya.ms.mff.cuni.cz/~bench/
for lots more information, benchmarks, and performance comparisons of
various middleware.
Thanks,
Doug
--
Dr. Douglas C. Schmidt Professor and Associate Chair
Electrical Engineering and Computer Science TEL: (615) 343-8197
Institute for Software Integrated Systems WEB: www.dre.vanderbilt.edu/~schmidt
Vanderbilt University, Nashville TN, 37203 NET: d.sc...@vanderbilt.edu