I've decided to break down and actually do the comparison that seems to be continually in demand. However, in the name of fairness and conservation of (my) resources, I'm not going to do it alone. Here's my plan:
The goal is to compare Delphi, BCB and VC++ at the code level from a performance perspective. I've made some very simplistic passes at this in the pass, but to do it correctly requires more "real world" code examples. Additionally, the comparison should be divided into several areas (string, integer, floating point that sort of thing) and several programming styles or approaches. Off the top of my head I thought maybe 'literal translations', 'simplest code' and 'fastest possible'. For example, given a string based snippet, the literal translations would both (Pascal and C++) be pchar based, use while's etc with the goal being that they be as similar a possible. For the simplest code approach, the OP code would be string based, use built in functions etc. For the fastest possible approach there would be no constraints, just let 'er rip.
It would be nice to have several examples for each combination of coding approach and target area so as you can see this is not a small project. Consequently, I think that Code Central would provide the most appropriate venue for this effort with one entry for each routine/algorithm. Code Central now allows threaded commentary/review by others and so provides an easy means of allowing group participation on each entry. Thus the current contents of the entry would eventually represent the consensus view for each approach. With appropriate designation anybody could contribute the initial problem with the understanding that they would be responsible for updating the entry with the latest versions for each language and compiler and each programming level. If anyone wants to submit an algorithm but doesn't want to handle the babysitting then they can send it to me and I'll handle the drudgework.
What say you? Given the amount of oxygen expended in just the most recent go-round in the C++ vs OP debate surely there are some who would be willing to participate in this? What I most need is one or more people to champion the C++ side, and of course those "real" world examples. Also there are a few niggling details to hash out.
Are you comparing languages or tools. If it's languages then your plan seems fine, but if it's tools, is performance just a measure of how fast the examples run? It's not a great deal of comfort to have an application that runs 10-15% faster (if it does) if you have to wait an extra month to get it.
Not being negative, just a 2CW.
Grif
Robert Lee wrote in message <38A17EFD.CF06E...@nwu.edu>... >I've decided to break down and actually do the comparison that seems to >be continually in demand. However, in the name of fairness and >conservation of (my) resources, I'm not going to do it alone
I find this very interesting, as I am planning on including a section on performance in my Visual C++ vs Delphi paper (see Jake's Delphi Site at http://www.xnet.com/~johnjac ) and was planning on doing my own performance comparisons. For it to be useful I think you are right that such a comparison should include both the normal types of code that would be written by programmers in their everyday programming, and the most-highly-optimized-as-we-can-make-it code. I was planning on starting such a comparison this coming weekend (for me it will be a 4-day weekend), for Delphi 5 Pro and Visual C++ 6 Pro. For C++, string testing code should cover CString, null-terminated character arrays, and the STL string class, because those are the three most commonly used strings.
If we can agree on code snippets that are fair comparisons across tools, I'll use those code snippets to run tests on my PC. I wonder if we shouldn't find out what code the big testing labs use for this stuff. Does anyone know this?
Does anyone else want to write up the results, or should I do it?
Also, we could get the Visual C++ guys over in the Microsoft forums to participate too, once the tests are defined. It could even be a contest or challenge.
Robert Lee wrote in message <38A17EFD.CF06E...@nwu.edu>... >The goal is to compare Delphi, BCB and VC++ at the code level from a >performance perspective. I've made some very simplistic passes at this >in the pass, but to do it correctly requires more "real world" code >examples.
Depends on your definition of real world code examples. The real world code I am working on simply cannot be written in VC++ because it's functionally lacking. If you choose to benchmark a subset the language such that VC++ can compile it, you are eliminating one of BCB's biggest strengths from the competition. Everyone knows VC++ is faster than BCB. But I don't care how fast it is if it isn't ANSI C++. Well, let me rephrase. I don't care how fast it is if it can't compile my code. In that respect, VC++ is worthless to me.
I am interested in helping. I not very good with C/C++ but I can help with some Delphi code. I think that separating the comparisons into categories is a good idea.
Something like this:
Category Subcategory Code Example #1 (with some kind of description - maybe pseudocode) Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
Code Example #2 (with some kind of description - maybe pseudocode) Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
for instance:
String Manipulation Category String Searching Subcategory Example#1 - Description & pseudocode Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
Example#2 - Description & pseudocode Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
String Concatenation Subcategory Example#1 - Description & pseudocode Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
Example#2 - Description & pseudocode Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation ...
> Also there are a few niggling details to hash out.
Some questions: 1) How would the entries map to Code Central submissions? One submission per Category, SubCategory, Example, or Coding approach? When someone wants to review the various comparisons from this project, there should be a way to "filter out" all submissions relating to it in Code Central. How this query is done and its results will depend on how the CC entries have been organized. I think that if a "system" is devised to use CC then a single submitter should be used to ensure that all submissions are consistent with the system. Maybe John can add another field to his CC database for people who want to submit entries under their own custom categories.
2) How should the literal translation be approached? Would this be a "brute force" implementation of the psuedocode? Will the OP version written first and literally translated into C++ or vice versa?
> I've decided to break down and actually do the comparison that seems to > be continually in demand. However, in the name of fairness and > conservation of (my) resources, I'm not going to do it alone.
Robert,
I am interested in helping. I don't have much experience with C/C++ but I can help with Delphi code. I think that separating the comparisons into categories is a good idea.
Something like this:
Category Subcategory Code Example #1 (with some kind of description - maybe pseudocode) Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
Code Example #2 (with some kind of description - maybe pseudocode) Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
for instance:
String Manipulation Category String Searching Subcategory Example#1 - Description & pseudocode Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
Example#2 - Description & pseudocode Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
String Concatenation Subcategory Example#1 - Description & pseudocode Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation
Example#2 - Description & pseudocode Literal Object Pascal translation / Literal C++ Translation Simplest OP implementation / Simplest C++ implementation Fastest OP implementation / Fastest C++ implementation ...
> Also there are a few niggling details to hash out.
Some questions: 1) How would the entries map to Code Central submissions? One submission per Category, SubCategory, Example, or Coding approach? When someone wants to review the various comparisons from this project, there should be a way to "filter out" all submissions relating to it in Code Central. How this query is done and its results will depend on how the CC entries have been organized. I think that if a "system" is devised to use CC then a single submitter should be used to ensure that all submissions are consistent with the system. Maybe John can add another field to his CC database for people who want to submit entries under their own custom categories.
2) How should the literal translation be approached? Would this be a "brute force" implementation of psuedocode? Should the OP version written first and literally translated into C++ or vice versa?
3) Will binaries be tested on more than one kind of processor? PII, PIII, Athlon...
> I am interested in helping. I don't have much experience with C/C++ > but I can help with Delphi code. I think that separating the comparisons > into categories is a good idea.
> Something like this:
While getting that many examples would be great I expect that it will be a bit more spotty. Consequently, it is likely that each example will be its own sub catagory.
> > Also there are a few niggling details to hash out. > Some questions: > 1) How would the entries map to Code Central submissions? > One submission per Category, SubCategory, Example, or Coding approach?
One per Example was what I was thinking.
> When someone wants to review the various comparisons from this project, > there should be a way to "filter out" all submissions relating to it in Code > Central. How this query is done and its results will depend on how the CC > entries have been organized.
Yes this can be handled simply by placing the appropriate keywords in the description.
I think that if a "system" is devised to use
> CC then a single submitter should be used to ensure that all submissions are > consistent with the system.
I don't think that will be necessary as the system doesn't need to be that complex:
Description: C++ vs Delphi project Catagory: Strings SubCatagory: Quick Sort;
> 2) How should the literal translation be approached? > Would this be a "brute force" implementation of psuedocode? > Should the OP version written first and literally translated into C++ or > vice versa?
I was thinking of simply starting with someones off the shelf code in either language and then attempting a literal translation to the other. If need be the original could be adjusted to bring the two more in line.
> 3) Will binaries be tested on more than one kind of processor? > PII, PIII, Athlon...
I can test on all of those, the PII vs PII distinction is hardly worth making though. Probably, we'll use a PII as a baseline and make note of any significant deviations on Athlon as needed.
Should BASM be allowed? BASM is part of the Delphi product, but some C++ compilers allow inline assembly also. The goal would be to compare the performance of the code the compiler produces, no?
I'll do a VB version for the comparsion...as long as it isn't too involved. I only have limited time for 'extracurricular' coding. <g>
Email me the Delpi code when you get it done and I'll redo it in VB and send it back. It would be interesting to see how VB stacks up against Delphi and C++
Robert Lee wrote: > I've decided to break down and actually do the comparison that seems to > be continually in demand. However, in the name of fairness and > conservation of (my) resources, I'm not going to do it alone. Here's my > plan:
> The goal is to compare Delphi, BCB and VC++ at the code level from a > performance perspective. I've made some very simplistic passes at this > in the pass, but to do it correctly requires more "real world" code > examples. Additionally, the comparison should be divided into several > areas (string, integer, floating point that sort of thing) and several > programming styles or approaches. Off the top of my head I thought > maybe 'literal translations', 'simplest code' and 'fastest possible'. > For example, given a string based snippet, the literal translations > would both (Pascal and C++) be pchar based, use while's etc with the > goal being that they be as similar a possible. For the simplest code > approach, the OP code would be string based, use built in functions etc. > For the fastest possible approach there would be no constraints, just > let 'er rip.
> It would be nice to have several examples for each combination of coding > approach and target area so as you can see this is not a small project. > Consequently, I think that Code Central would provide the most > appropriate venue for this effort with one entry for each > routine/algorithm. Code Central now allows threaded commentary/review > by others and so provides an easy means of allowing group participation > on each entry. Thus the current contents of the entry would eventually > represent the consensus view for each approach. With appropriate > designation anybody could contribute the initial problem with the > understanding that they would be responsible for updating the entry with > the latest versions for each language and compiler and each programming > level. If anyone wants to submit an algorithm but doesn't want to > handle the babysitting then they can send it to me and I'll handle the > drudgework.
> What say you? Given the amount of oxygen expended in just the most > recent go-round in the C++ vs OP debate surely there are some who would > be willing to participate in this? What I most need is one or more > people to champion the C++ side, and of course those "real" world > examples. Also there are a few niggling details to hash out.
> Should BASM be allowed? BASM is part of the Delphi product, but some C++ > compilers allow inline assembly also. The goal would be to compare the > performance of the code the compiler produces, no?
No assembler. The point is to compare the compilers.
> Depends on your definition of real world code examples. The real world code > I am working on simply cannot be written in VC++ because it's functionally > lacking. If you choose to benchmark a subset the language such that VC++ > can compile it, you are eliminating one of BCB's biggest strengths from the > competition.
A definite problem, but only on the "literal translation" part of the comparison. Presumably, equivalent code could be developed and compared.
The point of this exercise is not to compare entire development platforms, but rather the compiler output. Yes, everyone knows that VC produces faster code than BCB. But how much faster, it is commonly tossed out that VC is *much* faster. I personally don't think that is true. I think it is *much* faster at a small handful of things and the same or only marginally faster on a bunch of others. The only way to settle this is with a solid comparison, rather than a pathetic and usually highly benchmark that some knocked off in 10 minutes.
> Are you comparing languages or tools. If it's languages then your plan seems > fine, but if it's tools, is performance just a measure of how fast the > examples run? It's not a great deal of comfort to have an application that > runs 10-15% faster (if it does) if you have to wait an extra month to get > it.
Tools really. And I agree with your assessment. Nonetheless there is always that "well, MS VC is *so* much faster" Followed by a bunch of idle speculation, rumor and inuendo. The point of this exercise is to eliminate that discussion by actually generating some public facts.
> I find this very interesting, as I am planning on including a section on > performance in my Visual C++ vs Delphi paper
Yes I thought there would be a tie-in there.
> If we can agree on code snippets that are fair comparisons across tools, > I'll use those code snippets to run tests on my PC.
Basically, I thought we try to win this by volume. Lots of examples in various areas. I dislike the one number benchmarks because unless that benchmark is actually *your* program is could be irrelevant. Also by having the whole thing in CodeCentral anybody would have access and the ability to comment or complain about any perceived unfairness.
> Also, we could get the Visual C++ guys over in the Microsoft forums to > participate too, once the tests are defined. It could even be a contest or > challenge.
Robert Lee wrote: > "Chris Uzdavinis (TeamB)" wrote:
> > Depends on your definition of real world code examples. The real world code > > I am working on simply cannot be written in VC++ because it's functionally > > lacking. If you choose to benchmark a subset the language such that VC++ > > can compile it, you are eliminating one of BCB's biggest strengths from the > > competition.
> A definite problem, but only on the "literal translation" part of the > comparison. Presumably, equivalent code could be developed and > compared.
One thing that could be done without a lot of effort is compile a bunch of those example applications that come with the MS compiler using both it and the Borland one. While not many of them are speed tests, one could at least get an idea of how big the executables generated by each compiler are without worrying about the VCL's overhead - some are WinAPI, while others use MFC, but that comes with both systems anyway. Another thing to consider is DirectX: the SDKs come with the source for some utilities that are meant to test the speed of graphics hardware, so it might be interesting to see if compiling them with MS/Borland results in different figures (MS supply libraries for BCB, so there shouldn't be too many problems).
Another idea might be to write some examples that thrash the NAG C++ numerics library, taking into account not only speed, but also the accuracy of the results produced by both compilers.
If nothing else, fiddling with all this stuff might produce some optimisation settings that help users of both compilers get the best possible results for their applications.
I don't see any pseudo-code examples posted yet, and my (4-day) weekend starts tonight, so I'm going to get started myself and then post the results here and in my paper. Hopefully that can be a starting point for expanding the tests.
> I don't see any pseudo-code examples posted yet,
You don't because I haven't <g>. Actually, I've got one just about ready. It is Linear regression test that I stripped out of one of my simulators. I'll try and get it up by the end of the day. Basically, I'm thinking that each version should be a unit with a 'test' procedure that can be called and timed. That way one test rig can handle any of the tests. Also I need to throw together a quickie timing mechanism that can is more accurate than GetTickCount and can be implemented in all enviroments without pulling in all of the Windows API.
> so I'm going to get started myself and then post the results > here and in my paper. Hopefully that can be a starting point for expanding > the tests.
Sounds good, we can organize as we go.
Based on my own prelim. test from about a year ago. It is very easy to skew the results merely by including or excluding specific functions. This is the "trick" that unscrupulus VC'ers have used to make Borland look really bad at FP. They always bury an ATAN in it somewhere that VC code does comparatively fast (but with some sloppiness) but is still so slow that it swamps all the remaining FP computations. This is the reason that I wanted to base this on "real" world code. What I'm trying to say is that if you just cook up your benchmark, your are likely to also be cooking your biases into the results.
An ASCII copy of the book of Genesis, King James Version, is used as the reference text. (This is public doman text.) Test 1: Find and replace the word "the" with "that". Test 2: Find and replace the letter "e" with "a". Test 3: Count the number of times the word "god" (case insensitive) appears in the text. Test 4: sort the lines in the text alphabetically (forward and as a second test, backward) Test 5: remove sentences with the word "god" (case insensitive) in them. Test 6: make the entire text uppercase. Test 7: all tests in one loop
Say, 1000 iterations? Don't count the time used to load the text into memory.
> One thing that could be done without a lot of effort is compile a bunch of those > example applications that come with the MS compiler using both it and the Borland > one.
Good for BCB vs VC but doesn't address the Delphi leg of the issue. Plus it leaves no room for anything but the literal translation and whose to say that MS hasn't tweaked these with respect to there own compiler.
> Another idea might be to write some examples that thrash the NAG C++ numerics > library, taking into account not only speed, but also the accuracy of the results > produced by both compilers.
John Jacobson <john...@xnet.com> wrote in message news:38a30e3e@dnews... > How about this for the strings tests?
[complex string manipulation spec snipped]
Your post moved me to articulate my thoughts on bechmarking VC++ versus Delphi versus BCB.
Almost by definition, this [snipped test] would test something, but I'm not sure what! I could imagine many "cheats" to optimise this (convert the whole thing to lowercase first, keep a list of occurences of words beginning with "th" as you read the thing in, maintain two collections one which sorts forward, and one backwards etc.) but what would it prove? That one C++ programmer was better than one Delphi programmer?
Personally, I use C++ for the server side of things (COM etc.) and Delphi/BCB/VB for the GUI, and this works out really well. Nobody is going to select Delphi or BCB as a development tool because it is faster (or slower) than VC++ for a particular artificial test - things like training costs, availability of knowlegeable staff etc. will be much more important.
Bottom line: can't we push Delphi (and BCB) as being complementary to VC++??? If not, we are losing the Windows environment which, despite the current Linux fad, is still the only game in town.
> Your post moved me to articulate my thoughts on bechmarking VC++ versus > Delphi versus BCB.
> Almost by definition, this [snipped test] would test something, but I'm not > sure what!
Unquestioningly any given test is of only marginal significance. The goal here is to have multiple tests so that the individual tests aren't so heavily weighted. This particular test is roughly a mix of several replacment schemes.
I could imagine many "cheats" to optimise this (convert the
> whole thing to lowercase first, keep a list of occurences of words > beginning with "th" as you read the thing in, maintain two collections one > which sorts forward, and one backwards etc.) but what would it prove?
With the entire process being open, the idea is to construct all versions with the same fundamental algorithm. If say the C++ version is faster then that algorithm could readily be tried in OP. Besides, that is only the 1/3 of each test. The other parts are more restricted, for instance the "simplest" category might be *really* simple in say OP but probably alot slower than the simplest C++ version.
> Personally, I use C++ for the server side of things (COM etc.) and > Delphi/BCB/VB for the GUI, and this works out really well. Nobody is going > to select Delphi or BCB as a development tool because it is faster (or > slower) than VC++ for a particular artificial test
Again, the idea here is to general a non-artificial test. It *is* worth little on its own, but this is just one piece. Eventually there will be many pieces with many (hopefully) from real world problems.
- things like training
> costs, availability of knowlegeable staff etc. will be much more important.
these are important short term. Long term is a different story. Many people use C++ (and VC++) because they believe the are "supposed" to. I see this all the time. I often run into a lot of flak for using pascal for my simulators. "Oh my god! you should use a "real" language like C++ or fortran, your sims would be so much faster". I then proceed to explain that this is simply not true, that I pay very little if anything for my choice and my development time is dramatically less. They typically remain unconvinced. What I would really like to do is be able to throw some numbers at them.
> Bottom line: can't we push Delphi (and BCB) as being complementary to > VC++???
Complementary? Do you mean comparable? I definitely think we can make a statement about how they compare to VC++. Borland has been loathe to do this, probably because they wouldn't "win" However, what if they loose by only 5%? Would you pick VC over BCB and all its RAD tools because of such a small difference?
> Complementary? Do you mean comparable? I definitely think we can make a > statement about how they compare to VC++. Borland has been loathe to do > this, probably because they wouldn't "win" However, what if they loose > by only 5%? Would you pick VC over BCB and all its RAD tools because of > such a small difference?
No, you misunderstand me. I don't think performance is an issue, and I've never had a (human) client who did. What it all comes down to is that:
- The Borland products (Delphi BCB) are demonstrably superior for building GUIs.
- The Borland products are not so demonstrably superior for N-tier. The reliance on CORBA prevents adoption in many (ectually most) business environments where COM, DCOM and other MS technologies have been adopted.
- VC++ is clearly superior to BCB (by virtue of it's excellent debugger [Borland, please copy]) for developing server and COM apps.
So Borland have (from my point of view, and I suspect from the point of view that anyone who has had the misfortune of using VB or programming with MFC) the GUI, and MS have the server. My question is why don't Borland leverage the GUI advantage?
> - The Borland products are not so demonstrably superior for N-tier. The > reliance on CORBA prevents adoption in many (ectually most) business > environments where COM, DCOM and other MS technologies have been adopted.
Delphi does COM. -- Please respond only in the newsgroup. I will not respond to newsgroup messages by e-mail.
"NeilB" wrote: >(snip) >- The Borland products are not so demonstrably superior for N-tier. The >reliance on CORBA prevents adoption in many (ectually most) business >environments where COM, DCOM and other MS technologies have been adopted.
FWIW, I do lots of COM/DCOM stuff in Delphi with little or no problems at all - I don't see where Delphi is preventing me from adopting these technologies.
> The other parts are more restricted, for > instance the "simplest" category might be *really* simple in say OP but > probably alot slower than the simplest C++ version.
Robert Lee wrote: > Good for BCB vs VC but doesn't address the Delphi leg of the issue.
Well, your message text was about VC++ vs. BCB - Delphi wasn't actually mentioned in the bit I was replying to.
>Plus it leaves no room for anything but the literal translation and >whose to say that MS hasn't tweaked these with respect to there own >compiler.
I don't think the MS examples are "tweaked" in any way because they're supposed to be didactic - if anything, they'd hopefully sacrifice efficiency for clarity, although one never knows with MS!
> > Another idea might be to write some examples that thrash the NAG C++ numerics > > library, taking into account not only speed, but also the accuracy of the results > > produced by both compilers.
> Yes, this could be a good starting point.
Or perhaps a freeware offering would be more accessible. There's a fairly comprehensive set of C and C++ source code numerics routines at www.brent.worden.org/products/numericsc.html and www.brent.worden.org/products/numericscpp.html. I don't usually do heavy numeric stuff in OP, but it shouldn't be too hard to convert the C ones to OP if there isn't anything comparable available for Delphi. Note I haven't used these, so I don't know how good they are - they just came up in a quick web search for freeware numerics libs. Even if they're not particularly efficient, using the same set of algorithms with all languages should provide a good starting point for some interesting benchmarks.
