Thanks to Sanghyeon for running the benchmarks. To summarize his results, it seems like IP is pretty good at: comparing simple types, loops, function calls and float arithmetic. It performs pretty poorly at almost everything else! I guess its time to convert critical code sections to Boo.
I agree, Boo is a nice language, but given its foundation as a statically-typed language it's not exactly a one-to-one comparison.
Of course you can always use the duck type capabilities to gain a similar dynamic effect, but even a Boo duck type doesn't provide a fair one-to-one as you still have to declare it as a duck type, and once you do, the performance advantage gained from the static compilation is obviously lessened.
Anybody happen to know of a similar benchmark comparison between CPython and the duck type-based Boo equivalent? And has anybody taken the time to quantify how much benefit is gained in developer productivity due to the rapid prototyping and interactive capabilities provided to the .NET developer at runtime?
If for no other reason (and there are quite a few other reasons), the fact that with IronPython I can quickly and easily move from idea > through the thought process > and into a working prototype in about the same amount of time it takes to open up Visual Studio and create a new project is worth each and every additional cycle, cycles which can easily be gained back by writing the production app in C# (or Boo, for that matter) once things have solidified into something a bit more production worthy.
Just food for thought, but if we're going to compare apples and oranges lets at least take a bite out of the apple and peel the orange before making a determination as to which one is preferable, and why.
On 4/21/07, Arman Bostani <ar...@twinsun.com> wrote:
> Thanks to Sanghyeon for running the benchmarks. To summarize his > results, it seems like IP is pretty good at: comparing simple types, > loops, function calls and float arithmetic. It performs pretty poorly > at almost everything else! I guess its time to convert critical code > sections to Boo.
I also I agree with you two. Boo is an extremelly nice and productive static language, and its syntax is so similar to Python's that waisting time deciding what to choose for a given task is almost non-sense. I think the decision should be made based on platform and libraries to be used mainly and, in any case, converting from python to Boo is pretty straightforward.
Note that performance-wise, Boo is still completely unoptimized, as its author recently recognized. It has for sure the potential to become as fast as C#, but so far its development has been concentrated in completeness and correction (as is the case with IP).
For me, the difference between python's dynamicity and Boo is simply that python allows for a more exploratory way of development. But if performance is paramount, moving part or all the program to Boo is extremelly simple.
Luis
On Apr 21, 10:04 am, "M. David Peterson" <xmlhac...@gmail.com> wrote:
> I agree, Boo is a nice language, but given its foundation as a > statically-typed language it's not exactly a one-to-one comparison.
> Of course you can always use the duck type capabilities to gain a > similar dynamic effect, but even a Boo duck type doesn't provide a > fair one-to-one as you still have to declare it as a duck type, and > once you do, the performance advantage gained from the static > compilation is obviously lessened.
> Anybody happen to know of a similar benchmark comparison between > CPython and the duck type-based Boo equivalent? And has anybody taken > the time to quantify how much benefit is gained in developer > productivity due to the rapid prototyping and interactive capabilities > provided to the .NET developer at runtime?
> If for no other reason (and there are quite a few other reasons), the > fact that with IronPython I can quickly and easily move from idea > > through the thought process > and into a working prototype in about > the same amount of time it takes to open up Visual Studio and create a > new project is worth each and every additional cycle, cycles which can > easily be gained back by writing the production app in C# (or Boo, for > that matter) once things have solidified into something a bit more > production worthy.
> Just food for thought, but if we're going to compare apples and > oranges lets at least take a bite out of the apple and peel the orange > before making a determination as to which one is preferable, and why.
> On 4/21/07, Arman Bostani <a...@twinsun.com> wrote:
> > Thanks to Sanghyeon for running the benchmarks. To summarize his > > results, it seems like IP is pretty good at: comparing simple types, > > loops, function calls and float arithmetic. It performs pretty poorly > > at almost everything else! I guess its time to convert critical code > > sections to Boo.
I'd like to point out one fact about these benchmarks that seems to have been missed. Seo's numbers are for running IronPython 1.1 on Mono - not on the Microsoft .NET implementation. The Mono team has done excellent work with that project, but today the performance of the .NET implementation is still significantly better and is a much better platform for benchmarking IronPython's performance.
I reran pybench on my laptop tonight on the RTM version of .NET that ships with Windows Vista - and has been available for XP for quite some time now. I compared with CPython-2.5.1 (the latest version).
Running on .NET, I find that IronPython is faster on some tests and CPython is faster on others. We know that we still have performance work to do. After all, IronPython just reached its 1.1 release and almost all of our recent work has been about compatibility and completeness. However, the story is already quite interesting. Out of the 51 tests in pybench, CPython is more than 2x faster on 10 of the tests and IronPython is more than 2x faster on 9 of the tests. Depending on what your code does, either implementation could run faster.
The most interesting cases to me are the 5 tests where CPython is more than 3x faster than IronPython and the other 5 tests where IronPython is more than 3x faster than CPython. CPython's strongest performance is in dictionaries with integer and string keys, list slicing, small tuples and code that actually throws and catches exceptions. IronPython's strongest performance is in calling builtin functions, if/then/else blocks, calling python functions, deep recursion, and try/except blocks that don't actually catch an exception.
The places where IronPython is performing strongest are where it can use the .NET code generation and JIT optimization most effectively. The places where it is slowest are mainly in runtime library implementation of core datatypes. CPython's list and dictionary datatypes are written in C and have been hand-tuned for Python-style workloads over the past 10 years. IronPython's datatypes are much newer and clearly need more tuning as the implementation matures.
The only two tests that really stand out as showing a deep performance issue are the two exception handling ones. These are the only tests where either implementation is more than 4x faster than the other. IronPython is 10x faster on the try/catch without an exception and CPython is 30x faster when an exception is actually raised. This is a deliberate design decision within .NET to make code that doesn't throw exceptions run faster - even if that means slowing down code that does throw exceptions. I'm fairly confident this was the right decision - and even remember the day long ago when Guido was discussing Python's exception system and explained that he'd accept almost any slow-down to the exceptional case in return for removing a single instruction from the non-exceptional path.
Thanks - Jim
My results --------------------------------------------------------------------------- ---- PYBENCH 2.0 --------------------------------------------------------------------------- ---- * using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int el)] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.clock
Your point regarding the performance difference between Mono and MS.NET is an important one. It's one of those things that should have been a lot more obvious than it was. None-the-less, thanks for bringing it to the surface!
One question: While I recognize the fact that for obvious reasons, ngen'ing the IronPython assemblies as part of the distribution isn't an option (don't worry, I'm not suggesting that it should or even could be an option ;-), has anyone ran these same benchmarks against an ngen'd version of the assemblies to compare the results? I realize that performance gains are no guarantee, but I have found that more often than not the performance gains are fairly significant, and worth the extra effort to ngen during the setup processes in *most* cases. And in this case, given the fact that the CPython runtime is compiled to native code and IronPython is still in all of its CIL glory during the start-up process I wonder if its worth gaining a more apples-to-apples comparison by running the same tests with ngen'd IP assemblies?
I would suggest the same for Mono, but as of the last time I checked, there is currently no support for AOT compiling .NET 2.0 assemblies (though I will quickly check to verify, as its been a month or two since I last checked)
Worth pointing out: With these updated numbers, and given your statement,
IronPython just reached its 1.1 release and almost all of our recent work
> has been about compatibility and completeness.
I must admit, I am certainly looking forward to the result when "speed and optimization" is added to the mix. :)
Thanks for all of your hard work!
On 4/22/07, Jim Hugunin <Jim.Hugu...@microsoft.com> wrote:
> I'd like to point out one fact about these benchmarks that seems to have > been missed. Seo's numbers are for running IronPython 1.1 on Mono - not > on the Microsoft .NET implementation. The Mono team has done excellent work > with that project, but today the performance of the .NET implementation is > still significantly better and is a much better platform for benchmarking > IronPython's performance.
> I reran pybench on my laptop tonight on the RTM version of .NET that ships > with Windows Vista - and has been available for XP for quite some time > now. I compared with CPython-2.5.1 (the latest version).
> Running on .NET, I find that IronPython is faster on some tests and > CPython is faster on others. We know that we still have performance work to > do. After all, IronPython just reached its 1.1 release and almost all of > our recent work has been about compatibility and completeness. However, the > story is already quite interesting. Out of the 51 tests in pybench, CPython > is more than 2x faster on 10 of the tests and IronPython is more than 2x > faster on 9 of the tests. Depending on what your code does, either > implementation could run faster.
> The most interesting cases to me are the 5 tests where CPython is more > than 3x faster than IronPython and the other 5 tests where IronPython is > more than 3x faster than CPython. CPython's strongest performance is in > dictionaries with integer and string keys, list slicing, small tuples and > code that actually throws and catches exceptions. IronPython's strongest > performance is in calling builtin functions, if/then/else blocks, calling > python functions, deep recursion, and try/except blocks that don't actually > catch an exception.
> The places where IronPython is performing strongest are where it can use > the .NET code generation and JIT optimization most effectively. The places > where it is slowest are mainly in runtime library implementation of core > datatypes. CPython's list and dictionary datatypes are written in C and > have been hand-tuned for Python-style workloads over the past 10 > years. IronPython's datatypes are much newer and clearly need more tuning > as the implementation matures.
> The only two tests that really stand out as showing a deep performance > issue are the two exception handling ones. These are the only tests where > either implementation is more than 4x faster than the other. IronPython is > 10x faster on the try/catch without an exception and CPython is 30x faster > when an exception is actually raised. This is a deliberate design decision > within .NET to make code that doesn't throw exceptions run faster - even if > that means slowing down code that does throw exceptions. I'm fairly > confident this was the right decision - and even remember the day long ago > when Guido was discussing Python's exception system and explained that he'd > accept almost any slow-down to the exceptional case in return for removing a > single instruction from the non-exceptional path.
Jim Hugunin wrote: > I'd like to point out one fact about these benchmarks that seems to have been missed. Seo's numbers are for running IronPython 1.1 on Mono - not on the Microsoft .NET implementation. The Mono team has done excellent work with that project, but today the performance of the .NET implementation is still significantly better and is a much better platform for benchmarking IronPython's performance.
> I reran pybench on my laptop tonight on the RTM version of .NET that ships with Windows Vista - and has been available for XP for quite some time now. I compared with CPython-2.5.1 (the latest version).
> Running on .NET, I find that IronPython is faster on some tests and CPython is faster on others. We know that we still have performance work to do. After all, IronPython just reached its 1.1 release and almost all of our recent work has been about compatibility and completeness. However, the story is already quite interesting. Out of the 51 tests in pybench, CPython is more than 2x faster on 10 of the tests and IronPython is more than 2x faster on 9 of the tests. Depending on what your code does, either implementation could run faster.
> The most interesting cases to me are the 5 tests where CPython is more than 3x faster than IronPython and the other 5 tests where IronPython is more than 3x faster than CPython. CPython's strongest performance is in dictionaries with integer and string keys, list slicing, small tuples and code that actually throws and catches exceptions. IronPython's strongest performance is in calling builtin functions, if/then/else blocks, calling python functions, deep recursion, and try/except blocks that don't actually catch an exception.
> The places where IronPython is performing strongest are where it can use the .NET code generation and JIT optimization most effectively. The places where it is slowest are mainly in runtime library implementation of core datatypes. CPython's list and dictionary datatypes are written in C and have been hand-tuned for Python-style workloads over the past 10 years. IronPython's datatypes are much newer and clearly need more tuning as the implementation matures.
> The only two tests that really stand out as showing a deep performance issue are the two exception handling ones. These are the only tests where either implementation is more than 4x faster than the other. IronPython is 10x faster on the try/catch without an exception and CPython is 30x faster when an exception is actually raised. This is a deliberate design decision within .NET to make code that doesn't throw exceptions run faster - even if that means slowing down code that does throw exceptions. I'm fairly confident this was the right decision - and even remember the day long ago when Guido was discussing Python's exception system and explained that he'd accept almost any slow-down to the exceptional case in return for removing a single instruction from the non-exceptional path.
> Thanks - Jim
> My results > --------------------------------------------------------------------------- ---- > PYBENCH 2.0 > --------------------------------------------------------------------------- ---- > * using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int > el)] > * disabled garbage collection > * system check interval set to maximum: 2147483647 > * using timer: time.clock
Jim Hugunin wrote: > I'd like to point out one fact about these benchmarks that seems to have been missed. Seo's numbers are for running IronPython 1.1 on Mono - not on the Microsoft .NET implementation. The Mono team has done excellent work with that project, but today the performance of the .NET implementation is still significantly better and is a much better platform for benchmarking IronPython's performance.
And on close examination the results are impressive. Nice one.
The anomaly with raising exceptions is very weird though (IronPython is around 30 times slower!) :
> I reran pybench on my laptop tonight on the RTM version of .NET that ships with Windows Vista - and has been available for XP for quite some time now. I compared with CPython-2.5.1 (the latest version).
> Running on .NET, I find that IronPython is faster on some tests and CPython is faster on others. We know that we still have performance work to do. After all, IronPython just reached its 1.1 release and almost all of our recent work has been about compatibility and completeness. However, the story is already quite interesting. Out of the 51 tests in pybench, CPython is more than 2x faster on 10 of the tests and IronPython is more than 2x faster on 9 of the tests. Depending on what your code does, either implementation could run faster.
> The most interesting cases to me are the 5 tests where CPython is more than 3x faster than IronPython and the other 5 tests where IronPython is more than 3x faster than CPython. CPython's strongest performance is in dictionaries with integer and string keys, list slicing, small tuples and code that actually throws and catches exceptions. IronPython's strongest performance is in calling builtin functions, if/then/else blocks, calling python functions, deep recursion, and try/except blocks that don't actually catch an exception.
> The places where IronPython is performing strongest are where it can use the .NET code generation and JIT optimization most effectively. The places where it is slowest are mainly in runtime library implementation of core datatypes. CPython's list and dictionary datatypes are written in C and have been hand-tuned for Python-style workloads over the past 10 years. IronPython's datatypes are much newer and clearly need more tuning as the implementation matures.
> The only two tests that really stand out as showing a deep performance issue are the two exception handling ones. These are the only tests where either implementation is more than 4x faster than the other. IronPython is 10x faster on the try/catch without an exception and CPython is 30x faster when an exception is actually raised. This is a deliberate design decision within .NET to make code that doesn't throw exceptions run faster - even if that means slowing down code that does throw exceptions. I'm fairly confident this was the right decision - and even remember the day long ago when Guido was discussing Python's exception system and explained that he'd accept almost any slow-down to the exceptional case in return for removing a single instruction from the non-exceptional path.
> Thanks - Jim
> My results > --------------------------------------------------------------------------- ---- > PYBENCH 2.0 > --------------------------------------------------------------------------- ---- > * using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int > el)] > * disabled garbage collection > * system check interval set to maximum: 2147483647 > * using timer: time.clock
>> I'd like to point out one fact about these benchmarks that seems to have been missed. Seo's numbers are for running IronPython 1.1 on Mono - not on the Microsoft .NET implementation. The Mono team has done excellent work with that project, but today the performance of the .NET implementation is still significantly better and is a much better platform for benchmarking IronPython's performance.
> And on close examination the results are impressive. Nice one.
> The anomaly with raising exceptions is very weird though (IronPython is > around 30 times slower!) :
>> I reran pybench on my laptop tonight on the RTM version of .NET that ships with Windows Vista - and has been available for XP for quite some time now. I compared with CPython-2.5.1 (the latest version).
>> Running on .NET, I find that IronPython is faster on some tests and CPython is faster on others. We know that we still have performance work to do. After all, IronPython just reached its 1.1 release and almost all of our recent work has been about compatibility and completeness. However, the story is already quite interesting. Out of the 51 tests in pybench, CPython is more than 2x faster on 10 of the tests and IronPython is more than 2x faster on 9 of the tests. Depending on what your code does, either implementation could run faster.
>> The most interesting cases to me are the 5 tests where CPython is more than 3x faster than IronPython and the other 5 tests where IronPython is more than 3x faster than CPython. CPython's strongest performance is in dictionaries with integer and string keys, list slicing, small tuples and code that actually throws and catches exceptions. IronPython's strongest performance is in calling builtin functions, if/then/else blocks, calling python functions, deep recursion, and try/except blocks that don't actually catch an exception.
>> The places where IronPython is performing strongest are where it can use the .NET code generation and JIT optimization most effectively. The places where it is slowest are mainly in runtime library implementation of core datatypes. CPython's list and dictionary datatypes are written in C and have been hand-tuned for Python-style workloads over the past 10 years. IronPython's datatypes are much newer and clearly need more tuning as the implementation matures.
>> The only two tests that really stand out as showing a deep performance issue are the two exception handling ones. These are the only tests where either implementation is more than 4x faster than the other. IronPython is 10x faster on the try/catch without an exception and CPython is 30x faster when an exception is actually raised. This is a deliberate design decision within .NET to make code that doesn't throw exceptions run faster - even if that means slowing down code that does throw exceptions. I'm fairly confident this was the right decision - and even remember the day long ago when Guido was discussing Python's exception system and explained that he'd accept almost any slow-down to the exceptional case in return for removing a single instruction from the non-exceptional path.
>> Thanks - Jim
>> My results >> --------------------------------------------------------------------------- ---- >> PYBENCH 2.0 >> --------------------------------------------------------------------------- ---- >> * using Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int >> el)] >> * disabled garbage collection >> * system check interval set to maximum: 2147483647 >> * using timer: time.clock
2007/4/22, Jim Hugunin <Jim.Hugu...@microsoft.com>:
> I'd like to point out one fact about these benchmarks that seems to have been missed. Seo's numbers are for running IronPython 1.1 on Mono - not on the Microsoft .NET implementation. The Mono team has done excellent work with that project, but today the performance of the .NET implementation is still significantly better and is a much better platform for benchmarking IronPython's performance.
I never tried to claim otherwise. Perhaps I should have been more explicit.
> I reran pybench on my laptop tonight on the RTM version of .NET that ships with Windows Vista - and has been available for XP for quite some time now. I compared with CPython-2.5.1 (the latest version). > (snip)
Can you provide the raw pybench result files? I can host it on my site side-by-side with Mono results.
I saw nothing wrong with your original post. I had just noticed that others on the list were taking these numbers as indicative of expected IronPython performance anywhere and I wanted to be sure that they could also see the numbers on .NET for the rest of the story.
Trying to understand these numbers and use the relative performance to help improve all of the different implementations out there can only be a good thing. Often these kinds of discrepancies can help identify low-hanging fruit for performance optimizations. I know that we're going to be looking at our dictionaries and our slicing operation in much more detail during the next performance push. I also remember that IronPython-0.6's poor performance on try/raise/except was noted by the Mono team - at the time, Mono was 10x slower than the already slow .NET here. I believe that this was the original motivation for the performance improvements to that Mono path that now make it even faster than .NETs.
In the spirit of more data, I've attached my raw results files for you in case they help with your comparisons. My results are on a ThinkPad X60 laptop with an intel T2400 1.83 GHz core duo processor and 1.5GB of RAM running Windows Vista.
-----Original Message----- From: users-boun...@lists.ironpython.com [mailto:users-boun...@lists.ironpython.com] On Behalf Of Sanghyeon Seo Sent: Sunday, April 22, 2007 7:35 AM To: Discussion of IronPython Subject: Re: [IronPython] pybench results for CPython and IronPython
2007/4/22, Jim Hugunin <Jim.Hugu...@microsoft.com>: > I'd like to point out one fact about these benchmarks that seems to have been missed. Seo's numbers are for running IronPython 1.1 on Mono - not on the Microsoft .NET implementation. The Mono team has done excellent work with that project, but today the performance of the .NET implementation is still significantly better and is a much better platform for benchmarking IronPython's performance.
I never tried to claim otherwise. Perhaps I should have been more explicit.
> I reran pybench on my laptop tonight on the RTM version of .NET that ships with Windows Vista - and has been available for XP for quite some time now. I compared with CPython-2.5.1 (the latest version). > (snip)
Can you provide the raw pybench result files? I can host it on my site side-by-side with Mono results.
I assume that IronPython is implemented with thread-safe data structures (i.e. there's no GIL). If so, then IronPython is at a disadvantage on single-threaded benchmarks like pybench.
Just for fun, I tested System.Collections.SortedList against its thread safe (Synchronized) version. The latter was 5 times slower!
In light of this simple test, its possible that 1) IP is very efficient at implementing thread safety 2) .NET Synchronized collections are horribly slow 3) my assumption on IP's implementation are wrong.
I think #2 is the winner here - the synchronized versions of the classes are much slower than just using a simple Monitor. But IronPython's data structures are thread safe and we just use Monitor's for most of them (there are some more tuned classes in the system as well but there aren't too many of those). ________________________________________ From: users-boun...@lists.ironpython.com [users-boun...@lists.ironpython.com] On Behalf Of Arman Bostani [ar...@twinsun.com] Sent: Tuesday, April 24, 2007 11:42 AM To: us...@lists.ironpython.com Subject: Re: [IronPython] pybench results for CPython and IronPython
I assume that IronPython is implemented with thread-safe data structures (i.e. there's no GIL). If so, then IronPython is at a disadvantage on single-threaded benchmarks like pybench.
Just for fun, I tested System.Collections.SortedList against its thread safe (Synchronized) version. The latter was 5 times slower!
In light of this simple test, its possible that 1) IP is very efficient at implementing thread safety 2) .NET Synchronized collections are horribly slow 3) my assumption on IP's implementation are wrong.
|
