Glen Low wrote: > Another Java vs C++ performance shootout:
Hmm, most of the C++ code looks more like C. I don't know if this also has an impact on performance, but taking a Java vs C++ shootout should respect language-specific features which wasn't made here, alt least for C++.
-- To get my real email adress, remove the two onkas -- Hendrik Belitz - Abort, Retry, Fthagn? -
>Anyone care to comment about the appropriateness of his C++ code and >compiler settings?
This supposed benchmark has already been well picked apart on the Slashdot discussion site. The C++ is poor and the compiler used (gcc) known for portability, not fast executables.
As the author of the site states, he was tired of Java losing all the performance benchmarks, so he decided to fix it!
What would have been more interesting would have been something else more like a contest. Give a set of problems to people who are familiar with one or the other language. Have teams code solutions, and then test the solutions. This would test, in part, the ease of writing fast code in C++ or Java.
Also interesting, but not done would be to attempt to see the impact, if any, of a re-optimizing compiler. While a good C++ compiler should be able to optimize for a normal case, if a section of code appears at profile time to take one particular branch, then the branch prediction could be modified in real time. While this is presumably possible with C++ with some sort of compiler feedback loop from instrumented code, it is not part of my normal experience.
Thorsten Ottosen wrote: > 2. He's comparing two particular implementations, not two languages.
Well, that's a particularly meaningless remark. Obviously, languages do not have a speed, only implemented programs do!
> 3. He's comparing compile-time optimizations with run-time optimizations.
He's comparing how fast this one goes with how fast that one goes. I don't understand your distinction, nor why it would be meaningful to someone who wants an implementation which runs quickly.
> 4. His test are *very* simple, not really useful for what he's trying to do.
He's trying to demonstrate that equivalent Java is faster than C++, and since that's what his tests showed, they're very useful :-)
news:1087403244.506978@master.nyc.kbcfp.com... > Thorsten Ottosen wrote: > > 2. He's comparing two particular implementations, not two languages. > > Well, that's a particularly meaningless remark. > Obviously, languages do not have a speed, only > implemented programs do!
Of course languages have speed. Java came in like a whirlwind and left even faster, but C++ seems to stay here forever. So Java is a faster language than C++.
> Anyone care to comment about the appropriateness of his C++ code and > compiler settings?
When I'm benchmarking for speed, I use -O3 with g++, not -O2. That's definitly an error on his part.
I glanced at a few of the tests. For the most part, they are pretty low level; it's hard to say what significance they have with regards to real code.
The hash code tests actually test how good the hashing function is. The default algorithm used by g++ is definitly bad for large numbers of short strings (it will cluster near the front of the table), but I've not done enough analysis to be able to say exactly what is too large or too short.
It would be interesting to see what happens with more complex data structures, like std::vector (and push_back) vs. java.util.Vector. Java has a decided advantage in array handling, because of the absense of aliasing. As soon as you use any data structure other than an Array of a built-in type, however, you need to dynamically allocate each element in the structure. Relocating garbage collection generally results in much faster allocation than you can get with operator new in C++, but even the fastest allocation can't compete with no allocation.
Note too that while garbage collection allocation is typically much faster than manual allocation (perhaps an order of magnitude or so), this is partitially offset by the time you spend collecting. His programs allocate little enough that he probably never needs to collect.
The small size of the programs are a definite advantage for Java's runtime optimizer. A run-time optimizer can't spend forever optimizing, but it has access to actual run-time data, AND can specialize its optimization for the specific processor. (G++, and most other C++ compilers, have flags to allow this as well.) Given the small size, a complete analysis of the program is possible in the limited time available, and given the other additional data, I would find it rather worrisome if Java didn't do better. I'm more sceptical with regards to real programs, however. If there really is one small critical loop, Java should win; that's not the case for most of my programs, however.
And of course, g++ is far (very far) from state of the art optimizing, especially (so I've been told) for Intel architectures.
All that said, however, for many applications, there will be little or no difference in speed between Java and C++. Java has a number of advantages which make life easier for the optimizer, and for most applications, garbage collection will be faster than manual memory management, but the fact that everything has to be dynamically allocated can cost a lot.
In the end, of course, performance isn't necessarily the key argument. For most of the work I do, safety is. And that is a criteria where C++ is a clear winner; it is very, very difficult to write correct programs of any significant size in Java. (It's not really that easy in C++, but at least the language never actively prevents you from doing the right thing.)
-- James Kanze GABI Software Conseils en informatique orient�e objet/ Beratung in objektorientierter Datenverarbeitung 9 place S�mard, 78210 St.-Cyr-l'�cole, France, +33 (0)1 30 23 00 34
> Thorsten Ottosen wrote: >> 2. He's comparing two particular implementations, not two languages. > > Well, that's a particularly meaningless remark. > Obviously, languages do not have a speed, only > implemented programs do! > >> 3. He's comparing compile-time optimizations with run-time optimizations. > > He's comparing how fast this one goes with how fast that > one goes. I don't understand your distinction, nor why it > would be meaningful to someone who wants an implementation > which runs quickly. > >> 4. His test are *very* simple, not really useful for what he's trying to >> do. > > He's trying to demonstrate that equivalent Java is faster than C++, > and since that's what his tests showed, they're very useful :-)
Well, but without programming in proper C++ (which he doesn't), this tests are meaningless. Maybe he should ask a C++ developer to rewrite the C++ tests, so he can get more meaningful results. I'm sure, if I would write both cases (java and C++), the C++ programs would run much faster than the Java programs. Not because my C++'s so good, but my Java's so bad :-)
-- To get my real email adress, remove the two onkas -- Hendrik Belitz - Abort, Retry, Fthagn? -
| > 2. He's comparing two particular implementations, not two languages. | | Well, that's a particularly meaningless remark. | Obviously, languages do not have a speed, only | implemented programs do!
but if you want a fair picture you need to consider many implementations of both languages/libraries. Picking just two is hardly enough.
| > 3. He's comparing compile-time optimizations with run-time optimizations. | | He's comparing how fast this one goes with how fast that | one goes. I don't understand your distinction, nor why it | would be meaningful to someone who wants an implementation | which runs quickly.
that's true.
| > 4. His test are *very* simple, not really useful for what he's trying to do. | | He's trying to demonstrate that equivalent Java is faster than C++,
I guess we have different opinions about the meaning of "equivalent".
| and since that's what his tests showed, they're very useful :-)
But the test doesn't show that because they are too simple. The tests shows, runtime-optimizations perform very well on overly simplified code.
As an example, virtual machines can pseudo-inline virtual function calls by detecting really fast that we're dealing with the same object. That pays of when you're calling the same function on the same object 10^6 times.
However, it's been a long time since even parts of my programs would fit that discription.
The interestig question is how good the runtime-optimizer works with normal code. And that is not answered by his tests.
>> 4. His test are *very* simple, not really useful for what he's >> trying to do. > He's trying to demonstrate that equivalent Java is faster than C++, > and since that's what his tests showed, they're very useful :-)
But it's an obvious truism, well known for quite a long time, that for some specific operations, Java is faster than C++, and that for others, C++ is faster than Java. Even with much older, less optimized Java implementations. Speed, per se, has always been a weak point of C/C++, due to the way they don't) handle arrays. Speed, per se, is also a weak point of Java, due to the way it handles objects of user defined types. I expect that a good Ada compiler (if such exist) would leave both languages far behind.
In the meantime, it's well known: if you want Java to win your benchmarks, use lots of arrays of basic types. If you favor C++, use lots of simple, value oriented user defined types; I'll bet that in any benchmark making extensive use of complex, for example, C++ will win hands down.
Note that for the programs that use neither, like sieve or word count, the times are remarkably similar; probably due in fact to the differences in the quality of the optimizer, rather than any real differences in the language.
Another interesting test is object creation. Java looks to be about three or four times faster than C++. However, it is important to note that 1) the memory use pattern is optimal for a garbage collector, and 2) more importantly, Java must optimize this aspect to a maximum, in order to be usable, since all user defined objects must be allocated dynamically. Typically, in C++, I'd guess that at most something like 25% of all objects are dynamically allocated. (But this depends enormously on the application domain. For numeric calculations involving complex, for example, there may be no dynamically allocated objects in C++. See above -- what you test makes a difference.)
In the end, of course, speed is usually less important that correctness. And while Java is fine for smaller things (where I definitely prefer it to Perl), it doesn't scale; beyond a certain size, writing a correct program in Java becomes almost impossible (where as in C++, it's just very difficult).
-- James Kanze GABI Software Conseils en informatique orient�e objet/ Beratung in objektorientierter Datenverarbeitung 9 place S�mard, 78210 St.-Cyr-l'�cole, France, +33 (0)1 30 23 00 34
> But it's an obvious truism, well known for quite a long time, that for > some specific operations, Java is faster than C++, and that for > others, C++ is faster than Java. Even with much older, less optimized > Java implementations. Speed, per se, has always been a weak point of > C/C++, due to the way they don't) handle arrays. Speed, per se, is > also a weak point of Java, due to the way it handles objects of user > defined types. I expect that a good Ada compiler (if such exist) would > leave both languages far behind.
I do not have the same system the owner of the web page has. Mine is probably much slower. That said, I ran his Java code for the Ackerman test with a parameter of 8 (giving a result of 2045). It took 2 seconds to run, including the time needed to start up the JVM. An Ada example compiled with the GNAT compiler, which uses the GNU backend, took 0.17 seconds. This was run on my 3 year old laptop using Win98.
I do not have a C++ compiler to compare timings. I would be surprised if any C++ compiler required anything near 2 seconds to run this test for properly constructed C++.
> Speed, per se, has always been a weak point of C/C++, > due to the way they don't) handle arrays.
I was trying to find an article online to help me understand why this statement is true, but there are way to many "Java vs. C++" flame sites that are clogging my results. Would you mind posting a link explaining this?
> ... it doesn't scale; beyond a certain size, > writing a correct program in Java becomes almost impossible (where as in > C++, it's just very difficult).
Having little experience with mammoth projects, I also was curious about this. I'm sure this is a very in-depth issue, but what language features allow/impede large-scale development?
> Another Java vs C++ performance shootout: > > http://kano.net/javabench/ > > Anyone care to comment about the appropriateness of his C++ code and > compiler settings?
I've taken the liberty to adjust the C++ code the way I think it "should" be (that is: using STL properly). Of course there could still be remarks about the choices I've made (like using std::iostream rather than FILE* or file descriptors for the sake of performance), but I think the results give a better idea of the differences in performance. The test was conducted on Intel and GCC; I would like to test Comeau too, but unfortunately I don't have that.
First of all, how were these measurements made? The profiling information should be extracted somehow... I know that instrumenting the code is one approach. In this case, how can measurement of method call can be done without altering the binaries?
Can someone give me a hint?
Catalin
"Glen Low" <glen...@pixelglow.com> wrote in message
On 17 Jun 2004 10:46:35 -0400, ka...@gabi-soft.fr wrote:
>But it's an obvious truism, well known for quite a long time, that for >some specific operations, Java is faster than C++, and that for others, >C++ is faster than Java. Even with much older, less optimized Java >implementations. Speed, per se, has always been a weak point of C/C++, >due to the way they don't) handle arrays.
This is not so obvious to me. I always thought that array access is fast in C++ since it only needs a simple offset calculation or pointer arithmetic to access an array element. No range checks, nothing.
[snip]
>Another interesting test is object creation. Java looks to be about >three or four times faster than C++.
Maybe with the general purpose allocator that is normally used (malloc/free). But one often can implemennt special allocators for special situations which then are much faster. E.G: well known is the scenario when we have a container of objects all the same size.
------------------------------------------------ Michael Schlenger ------------------------------------------------
> Anyone care to comment about the appropriateness of his C++ code and > compiler settings?
Just FYI, I've already taken four of the bechmarks there and optimized them. I'll probably look at some more as time allows. All that I've looked at are now running faster than the Java implementations. I've submitted the code back to the author. One such implementation was 2.2 - 2.7 times faster than the java implementation while his original implementation was actually slower than java's. That one was the hash benchmark. I also noticed that one of the c++ implementations were not even proper (as in, didn't do what it was supposed to be doing) and was slower than the java implementation to boot. I corrected both of those.
Basically, all of the benchmarks appear to be absolute best corner case situations where hot spot can optimize and all appear to avoid ever paying any type of heap costs. The objinst benchmark, I simply used a commonly available collector and it quickly zoom ahead of the java implementation. Next, I plan on implementing an optimized heap implementation specifically tailored for the object life cycle used in the benchmark. Which is, highly transient, short lived objects whereby the destuctor is never used nor required.
Still several others, in an attempt to eliminate noise, I wanted to run a longer benchmark. Sadly, the java implementations were not able to run because I didn't have enough memory. The C++ implementation would take about 58M or something like that and completed in about 2s. The java implementation ran out of memory at 380M and failed to complete after around 6+ seconds. I was unable to add more memory to the VM with causing my system to start paging.
Oddly, the guy did mention that he considered proper C++ implementations to be "hand optimized" rather than recognizing that most of the benchmark code was simply horrible and improper implementations. I take it tha the believes his bad benchmark actually carries some level of validity. At any rate, he said he'd offer an update later this summer. So, check back to see what happens. ;)
> > Anyone care to comment about the appropriateness of his C++ code and > > compiler settings?
> When I'm benchmarking for speed, I use -O3 with g++, not -O2. That's > definitly an error on his part.
Actually, I tried several -O options here and for the few benchmarks that I tried different options, -O2 faired rather well. Mostly, -O3 made no difference. I may go back and make another pass with -O3 with the new code to see how things fair, just to make sure.
> I glanced at a few of the tests. For the most part, they are pretty low > level; it's hard to say what significance they have with regards to real > code.
Which is exactly why almost all of these represent ideal corner cases for runtime optimization but have little reflection of real world application. Worse, most of the C++ code is just horrible.
> The hash code tests actually test how good the hashing function is. The > default algorithm used by g++ is definitly bad for large numbers of > short strings (it will cluster near the front of the table), but I've > not done enough analysis to be able to say exactly what is too large or > too short.
Agreed. While using an equivilent hash implementation as what was being used in the java implementation, the c++ implementation is now 2.2 - 2.7 times faster than the java implementation. The cost was submitted back to the author.
> It would be interesting to see what happens with more complex data > structures, like std::vector (and push_back) vs. java.util.Vector. Java > has a decided advantage in array handling, because of the absense of > aliasing. As soon as you use any data structure other than an Array of a > built-in type, however, you need to dynamically allocate each element in > the structure. Relocating garbage collection generally results in much > faster allocation than you can get with operator new in C++, but even > the fastest allocation can't compete with no allocation.
> Note too that while garbage collection allocation is typically much > faster than manual allocation (perhaps an order of magnitude or so), > this is partitially offset by the time you spend collecting. His > programs allocate little enough that he probably never needs to collect.
Which is exactly true. Thusly, helping to invalidate benchmarks, such as objinst. When I grabbed a GC for C++ and compiled against, the c++ implementation became faster by about 30%. I plan on quickly writing my own heap implementation for that benchmark, which I suspect will be much, much faster yet. The GC C++ implementation was submitted back to the auther and I plan on submitting the other implementation as well.
> Anyone care to comment about the appropriateness of his C++ code and > compiler settings?
It appears he was testing the I/O system, and not actual number crunching.
For small loops on Pentium machines, it is possible for Java to be faster than native code. This is because some CPUs have deep caches. Some highly optimizing compilers in fact exploit this by generating P-code for small loops and using an interpreter that runs entirely within the cache.
On CPUs designed for multi-processor applications with shallower caches (such as SPARCs), Java sucks. I find it ironic that Sun produced a language that runs ideally on Intel.
I wonder why he didn't use the new gcj GNU compiler to compile the Java so both the C++ and Java would run natively?
> In the end, of course, performance isn't necessarily the key argument. > For most of the work I do, safety is. And that is a criteria where C++ > is a clear winner; it is very, very difficult to write correct programs > of any significant size in Java. (It's not really that easy in C++, but > at least the language never actively prevents you from doing the right > thing.)
You mentioned in several posts that Java does not scale well but you did not give an example. In which case, does Java not scale well? What prevent Java to write a large program? I don't think you have to write ONE large program to accomplish a task. Instead, you can write SMALL programs distributed together to accomplish the same task. In this case, does Java still have a scalable problem?
> Another Java vs C++ performance shootout: > > http://kano.net/javabench/ > > Anyone care to comment about the appropriateness of his C++ code and > compiler settings? >
I use both languages daily and my experience is:
1. When implement complex algorithms, C++ is far more elegant and easier than Java
2. When you need access to OS, forget about Java
3. Java provides a large runtime environment which C++ is lack of. So, it is easy to write some code in Java than C++
4. In general, C++ runs faster than Java. Forget about this benchmark test. C++ and Java like plane and car. No doubt, it is much quicker to go to a Wal-Mart from home by a car than by a plane.
For small and lightweight processes, Java is preferred. For mission critical and heavy-duty processes, C++ is the best choice.
> But it's an obvious truism, well known for quite a long time, that for > some specific operations, Java is faster than C++, and that for others, > C++ is faster than Java. Even with much older, less optimized Java > implementations.
I've looked at two of his examples: ackermann and hash, and these illustrate your point nicely. The first example, the ackermann seems indeed much faster in Java. A little googling told me that this is because C and C++ compilers cannot implement tail-call elimination as well as Java. With n = 12 the Java implementation is more than twice as fast on my Athlon 2.5 GHz/ 256MB / Linux 2.4 / Java HotSpot 1.4.2 / GCC 3.3.1
The hash example however is less impressive:
- I cannot reproduce the results the author claims. If I run his binaries Java is already losing with a factor 5 if n = 100k in both client and server mode.
- With a better hash function and itoa() instead of sprintf you can further speedup the cpp version (see the example code at the end). With n = 1M the speedup is almost 50%. A further improvement would be the removal of the costly second strdup() call which seems unnecessary. But for some mysterious reason this gives faulty results at some values of n. Perhaps someone can explain what is happening here.
- The Java version can handle about n = 600k but after that it throws an out of memory exception. C++ can handle n = 4M without swapping. Rather embarrassing.
> | > 2. He's comparing two particular implementations, not two > | > languages.
> | Well, that's a particularly meaningless remark. > | Obviously, languages do not have a speed, only > | implemented programs do!
> but if you want a fair picture you need to consider many > implementations of both languages/libraries. Picking just two is > hardly enough.
Yes and no. If I have to evaluate which language to use for a project, and speed is that important, then I'll restrain my evaluation to the implementations which are available, or which I can get threw purchasing. Here, that means Sun's JDK and VC++ for Windows, Sun CC or g++ for Solaris. The fact that better implementations exist (for both languages) is irrelevant to me if I can't use them.
In theory, all Turing complete languages are equal. A perfect compiler will determine the actual semantics, and generate the best code for those semantics. In practice, of course, nobody's ever written a perfect compiler, and nobody is going to do so anytime soon. Different languages make different things easier or harder for the optimizer. C/C++'s built in array semantics, with arrays decaying to pointers, is a classical example of a language making this excessively difficult; Java's lack of aliasing helps optimization considerably. On the other hand, C/C++ allow the user to specify some optimizations (like placing objects on the stack, rather than allocating them dynamically) that are incredibly difficult for a compiler to find (in the general case).
Obviously, if you want to show Java faster, you make extensive use of arrays, and avoid user defined types. If you want to show C++ faster, just make extensive use of user defined types, either in short lived arrays (so creation and deletion are more important than access) or as local variables.
> | He's comparing how fast this one goes with how fast that one goes. I > | don't understand your distinction, nor why it would be meaningful to > | someone who wants an implementation which runs quickly.
> that's true.
> | > 4. His test are *very* simple, not really useful for what he's > | > trying to do.
> | He's trying to demonstrate that equivalent Java is faster than C++,
> I guess we have different opinions about the meaning of "equivalent".
The few bits of code I looked at were more or less equivalent. On the other hand, of course, what he has shown is that for some programs, equivalent Java is faster than C++. But that's something that any reasonable person would have expected anyway, once the technology became mature. C/C++'s aliasing and lack of garbage collection result in known performance problems.
I don't know whether his choices were intentional, or simple an unintentional result of trying to make things simple. It's simpler, in very small programs, to just use built-in types, for example. And Java is particularly efficient with built-in types.
> | and since that's what his tests showed, they're very useful :-)
> But the test doesn't show that because they are too simple. The tests > shows, runtime-optimizations perform very well on overly simplified > code.
It depends. From his tests, I think it safe to say that if you are developing on an Intel based Linux box, and your typical applications are between ten and twenty lines, Java will probably outperform C++ most of the time. There are certainly other cases where Java will outperform C++, but his tests don't indicate what they are.
> As an example, virtual machines can pseudo-inline virtual function > calls by detecting really fast that we're dealing with the same > object. That pays of when you're calling the same function on the > same object 10^6 times.
Static analysis can also do this in certain cases. But it's harder, and I'm pretty sure that g++ doesn't do it.
> However, it's been a long time since even parts of my programs would > fit that discription.
The same is true for me, but there are applications where this is the case.
> The interestig question is how good the runtime-optimizer works with > normal code. And that is not answered by his tests.
The interesting question is what do you consider normal code:-). I suspect that no two people will have perfectly identical answers.
-- James Kanze GABI Software Conseils en informatique orient�e objet/ Beratung in objektorientierter Datenverarbeitung 9 place S�mard, 78210 St.-Cyr-l'�cole, France, +33 (0)1 30 23 00 34
| glen...@pixelglow.com (Glen Low) wrote in message | <news:9215d7ac.0406151559.2b45e595@posting.google.com>... | > Another Java vs C++ performance shootout: | | > http://kano.net/javabench/ | | > Anyone care to comment about the appropriateness of his C++ code and | > compiler settings? | | When I'm benchmarking for speed, I use -O3 with g++, not -O2. That's | definitly an error on his part.
But -O3 may be counter-productive in some cases, it tries to inline as much as it can -- even those you did not mark inline.
-- Gabriel Dos Reis g...@cs.tamu.edu Texas A&M University -- Computer Science Department 301, Bright Building -- College Station, TX 77843-3112
If one wants to get real number crunching speed one has to use SIMD instructions. In my experience many functions can be SIMD optimized to run 2-5X faster than "regular" code. This can be quite easily done from within C++ because generally it allows access to all kinds of hardware capabilities (e.g. one can use SIMD C intrinsics for Intel CPUs). I don't believe SIMD will be supported by Java any time soon at the level beyond toy examples.
Generally, C/C++ code quite straightforwardly translates to machine instructions, which is not the case with Java if we use interpreter. Java can achieve comparable speed only if it uses Just-in-time (JIT) compiler, which basically translates Java byte code to machine code . And, theoretically, JIT compiler can do some dynamic optimizations - by doing so it can outperfom statically optimized code (in principle, that is). But C++ code can be compiled to Java byte code as well - and therefore it can also take advantage of dynamic optimizations. Thus, comparison of C++ progamming language and a JIT compiler is not apple to apple comparison.
In addition, one of the inherent performance problems of the Java programming language is that it heavily uses heap memory allocations and RTTI, both of which are expensive. Even translation to machine code will not remove the penalties.
