Language Shootout on 28 Dec 2007
after....@gmail.com
compiled. It would also certainly be better if most of the implicit
string copies were not done (remember, string is immutable).
dmac...@gmail.com
Ruby 1.8 is slow. Ruby 1.9 has a bytecode virtual machine (YARV) which
is as fast (or even faster) than Python. Ruby 1.9 was released some
days ago.
Marius Gedminas
that matter, assuming they exist. I left the Java world before it had
generics, and never had the opportunity to look at C# (other than some
idle browsing of Tomboy's source code). While I'm very happy at how
Python lets me express my thought concisely, that doesn't mean I want
to ignore what's happening with the other languages out there. It
just means that I'm more likely to learn from a blog post on one of
the planets I read than go out and try to look for good C# or Java
related websites.
ro...@kennke.org
http://openjdk.java.net or packaged in Ubuntu, Fedora and Gentoo
(among others) as IcedTea (http://icedtea.classpath.org).
Andy
exactly written in a way that encourages efficiency (indeed anyone
used to RegExp in any other language including Java not even to
mention PERL et al will find it cumbersome and disappointing).
Benchmarks tend to show how fast something runs benchmarks and after
the implementors optimize for the benchmark... More interesting is
looking at the areas that Java6 has improved the VM and how mono can
match (such as acquiring light locks when there is no contention) and
taking platform advantages and improving on them (Java classloading is
truly cracked but Mono/.Net non-class-unloading was [is?] more
cracked). It is not surprising that mono would be much tighter and
then slower. In fact the two may be related. A well tuned garbage
collector takes twice the amount of memory to match (or possibly
slightly exceed) the performance of manually collected memory. Java 5
saw a drastic re-engineering of Sun's GC (especially parallel) and
Java 6 saw some rework of that plus a lot of work in the JIT and
threading (esp locking).
pascal....@gmail.com
http://alioth.debian.org/forum/forum.php?thread_id=5548&forum_id=999
Miguel de Icaza
> http://openjdk.java.net or packaged in Ubuntu, Fedora and Gentoo
> (among others) as IcedTea (http://icedtea.classpath.org).
Thanks! IcedTea seems to be the one that works, the OpenJDK does not
seem ready to run yet.
Miguel de Icaza
> http://alioth.debian.org/forum/forum.php?thread_id=5548&forum_id=999
That would improve things, but would not measure the default deployment model.
Miguel de Icaza
> It isn't just RegEx support in Mono, but the regex api from .NET isn't
> exactly written in a way that encourages efficiency (indeed anyone
> used to RegExp in any other language including Java not even to
> mention PERL et al will find it cumbersome and disappointing).
I agree. The API seems to have been designed by someone that was
given a feature list, and it does not seem to have been "usability-tested".
The fact that I have to look up the docs every time I need to use it means
that there was not enough focus on incorporating it into the language.
I for one,
would love to see Perl-like operators in C# for regex matching.
> Benchmarks tend to show how fast something runs benchmarks and after
> the implementors optimize for the benchmark... More interesting is
> looking at the areas that Java6 has improved the VM and how mono can
> match (such as acquiring light locks when there is no contention) and
> taking platform advantages and improving on them (Java classloading is
> truly cracked but Mono/.Net non-class-unloading was [is?] more
> cracked).
I agree that we need to find ways to improve Mono. And I guess this is good
food for research (light locks).
Am only familiar with Java folks complaining about the class loading system,
but I do not understand or pretend to understand the issue.
What seems to be the problem you experience with class unloading? In Mono
and .NET, class unloading is only performed by unloading an AppDomain, not
by selectively unloading classes.
> It is not surprising that mono would be much tighter and
> then slower. In fact the two may be related. A well tuned garbage
> collector takes twice the amount of memory to match (or possibly
> slightly exceed) the performance of manually collected memory. Java 5
> saw a drastic re-engineering of Sun's GC (especially parallel) and
> Java 6 saw some rework of that plus a lot of work in the JIT and
> threading (esp locking).
I still believe that Java is bloated, but there is a way of finding out, someone
with access to .NET should benchmark Java vs .NET as they have comparable
GCs.
Miguel
rodne...@gmail.com
have surpassed Lua (my personal favorite that was once a speed demon
of the 3rd tier). Keep up the good work.
Rodney
Andy Oliver
> Hello,
> >
> I agree. The API seems to have been designed by someone that was
> given a feature list, and it does not seem to have been "usability-tested".
>
> The fact that I have to look up the docs every time I need to use it means
> that there was not enough focus on incorporating it into the language.
> I for one,
> would love to see Perl-like operators in C# for regex matching.
>
Yes. David Bau I think did it or at least told me he did. He was of
the opinion that if you had never used regexp before in
any other language then you'd like it, if you had then you
wouldn't.... I don't know that I agree, but he's a smart guy anyhow.
It is hard to read/hear/say/write "I would love to see Perl-like"
without the simultaneous twinge of remembering the pain of debugging
someone else's
Perl. However boy is it lovely when you have to (un)mangle some text
and often I'm shocked when I write what I think is going to be the
ultimate
performance sucker and it runs instantly...
Java's regexp is not as good as Perl's but much nicer than .NETs. I
would bet per your point however that this is at the trivial level,
but w/o touching the
API I don't think you'll match here.
>
> I agree that we need to find ways to improve Mono. And I guess this is good
> food for research (light locks).
>
> Am only familiar with Java folks complaining about the class loading system,
> but I do not understand or pretend to understand the issue.
>
> What seems to be the problem you experience with class unloading? In Mono
> and .NET, class unloading is only performed by unloading an AppDomain, not
> by selectively unloading classes.
>
I'm probably talking out of my ass on this but...
Some background, skip if you like: Primarily, I came to both
appreciate and lament .NET's assembly loading in trying to get IKVM
and Mono to run JBoss (my background). A few of us
had a "pipe dream" that we occasionally messed around with. The idea:
A full JavaEE appserver that you could use Microsoft
development tools to develop for (if you liked), write aspects and
interceptors in .NET around Java classes or Java around
C# if you prefer... It appealed because it traversed and peed in both
giants' Sanka[http://en.wikipedia.org/wiki/Sanka] (Microsoft and Sun,
Java and .NET) more
than for practical matters. Just imagine the stink it would have made
had I gotten it to work and passed Sun's JavaCTK running on .NET :-P .
Mostly it was an excuse to fart around with Mono and C# on company time ;-).
One biggy that I punted on early...hot deployment/dynamic deployment
(scan directory for stuff....load it, rescan and unload or reload it).
So my problem is more that it seemed difficult to both load a class
that could be "pass[ed] by reference", dynamically load and
yet leave it unloadable... Granted you only care about this kind of
minutia if you're doing a lot of remoting
or running an application server or something. On one hand, I really
like the way .NET handles classloading and
versioning in particular (at least the MS VM as I understand it, the
last time I used Mono it couldn't do some of the tricks yet but it
was awhile back). On the other hand, it makes dealing with a large
potentially dynamically loaded,
potentially remotely loaded application server difficult to do without
leaking class definitions (not that Java does this nicely either).
For our little pipe dream we thought "okay no big deal you'll just
have to deal with restarting the appserver and no RMI classloading or
network loading
fun".
Consider if you have 10 different working groups in your company and
they have some rough interface agreement. So
in an "enterprise" application you may have these 10 working groups
writing a web layer and some services/business/data stuff. Obviously,
they'll all
have their own management structures and be incapable of using the
same level of dependencies or having the same release cycle. It is
possible in Java (but admittedly not all that commonly done do to the
complexity) to structure the classloaders such that you have a
hierarchy. Interface agreement occurs at a trivial
level (value objects and service interfaces) such that you pass by
value but if two apps/services have two versions of the same library
at some
deeper level that is fine. JBoss does this admitedly rather poorly.
MyApp.ear CL1
-------MyInterfaces.jar CL1
-------SomeLibrary.jar CL1
-------MyAppStuff.jar CL2
------------------MyDependencyVersion1.jar CL2
-------MyOtherAppStuff.jar CL3
------------------MyDependencyVersion2.jar CL3
In this hierarchy assume that MyOtherAppStuff and MyAppStuff have some
kind of dependency. CL1 can share down but CL2 and CL3 do not share
up. Thus
MyAppStuff and MyOtherAppStuff can share the SomeLibrary and
MyInterfaces without letting MyDependency get in their way.
Every JavaEE appserver handles this differently. In JavaEE most
appserver support some kind of hot deployment as well, such that we
could redeploy MyAppStuff
and MyOtherAppStuff independently. The house of cards falls if you
redeploy MyInterfaces (indeed in the above example you simply cannot
as it is in the parent classloader).
So by my design I just wanted to get JBossAS to run on IKVM (which I
did but with thoroughly underwhelming performance whereas I could
start JBoss in 30 second in Java or
5 minutes on .NET, but it wasn't clear whether it was Mono or
Classpath at fault -- and I had to statically load everything which
sucked). It looked like mapping jar files to .NET assemblies and
enabling side-by-side execution but it was brand new in Mono at the
time and I didn't see how unloading was going to work and no way how
remote loading was going to work and all this w/o requiring anyone to
custom-spin their .jar/assemblies (note that the above is very similar
to a diagram explaining side-by-side execution). Granted I only spent
a little bit of time digging into it as it became obvious that the
"classpath sucks" issue was too large to get anywhere (sorry to my
pals on #classpath but it was true).
However I never figured out how you'd do "hot deployment" of multiple
versions of a library in .NET w/o leaking even with s-b-s execution...
Could be I'm missing something. Prolly making an ass out of myself
with my swisscheese memory.
> > It is not surprising that mono would be much tighter and
> > then slower. In fact the two may be related. A well tuned garbage
> > collector takes twice the amount of memory to match (or possibly
> > slightly exceed) the performance of manually collected memory. Java 5
> > saw a drastic re-engineering of Sun's GC (especially parallel) and
> > Java 6 saw some rework of that plus a lot of work in the JIT and
> > threading (esp locking).
>
> I still believe that Java is bloated, but there is a way of finding out, someone
> with access to .NET should benchmark Java vs .NET as they have comparable
> GCs.
>
I run Linux only cept for occasional Outlook integration testing for
Meldware (http://buni.org) on VMWare. I think that I once did some
.NET testing on my wife's computer but its too random for perf testing
(with the viruses and virus checkers and weird junk she runs on
there), but rebooting made me cry.
They don't yet have comparable GCs though at least by available online
references. There is no reason to believe that the .NET's GC would
outperform the Sun or JRocket
VMs as the Java VMs have multiple algorythims that by wikipedia at
least .NET doesn't have yet. The .NET has only the concurrent
GC which tends to underperform the parallel GCs on 4core+ machines (I
only really deal with performance on big systems so your mileage may
vary). There is reason to
believe that the .NET IL language will compile faster/better than the
Java equiv because it is better designed/tighter/etc. There is also
reason to believe that the class libraries
make a larger difference in microbenching than the VM/JIT/GC do anyhow
:-/. To me it isn't that interesting unless you're running a bunch of
threads and processing gobs of
data...but that may be an excessive single-mindedness on my part.
Even the MS VM doesn't yet have the GC that I'd likely want to deploy
for a high volume app... No doubt they'll be there with the next rev.
IT took until Java5 for sun to get that part right -- albeit probably
like only 30 of us in the world know how to configure it correctly:
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp. With
satisfying options like -XX:+UseParallelOldGC and
-XX:+UseConcMarkSweepGC where can you go wrong? ;-)
-Andy
> Miguel
>
casperbang
is just too much hatred/NIH towards C# in that camp to make any real
progress."
Indeed, that's sadly true... it seems like there is no getting through
to some people regarding the need for components over a class library.
For what it matters though, I would love to see what you think of the
closure proposals. (I'm for Colebournes FCM, as it would bring
delegates into Java land)
/Casper
pascal....@gmail.com
) is 2x slower with AOT activated !
bogdan....@gmail.com
It's funny to always see these strong comments against Sun and Sun's
JVM. This comes especially after they actually did moved the Java to
GPL - unlike mono -, and it's not at all easy for them, since they
weren't Free from the beginning. Imagine how many third party
companies need to be contacted and asked about every line from some
old libraries they might have developed at some point in time in let's
say OpenSolaris. Just think a little bit at OpenSolaris, OpenOffice,
NetBeans or even Java - these are huge moves towards Free software and
you shouldn't try to minimize them. Even if now there are some parts
which are not yet solved, I have no doubt, that in the next year they
will be solved.
Another thing about the "bad Java" language this time, without
closures, etc. etc., is that you might want to take a look at groovy
(http://groovy.codehaus.org/). Yup, it compiles to java bytecode, and
it has all the things you want - including support for production
ready IDEs (like Eclipse, NetBeans or IntelliJ Idea) which I strongly
doubt that mono has, from more than one vendor. If you can't find or
do something in groovy I am sure that Guillaume Laforge can point you
out, or heh, you can even make a feature request.
And who cares that your product (mono) is working a tiny little bit
faster than the other (java)? What I am interested in is not being
vendor locked and having a plenitude of libraries, frameworks and
application servers to choose from. At these mono is light years
behind...
Have fun,
Bogdan
tleslie
"stellar productivity that can be achieved with Ruby on Rails is
hardly matched"
within the context of comparison to C#.
That is, what would be an example of something that
"Ruby on Rails" would blow C#/Mono away on, with respect to
productivity,
and is there issues (and improvements) required in C# to bridge this?
I guess I am skeptic. Perhaps the answer, like with Perl - being a
quick duct
tape sol'n to problems (as Larry puts it), is why perhaps RoR can be
productive?,
but extended to large projects, that productivity probably reverses?
I know nothing of RoR, but hear lots a about it ... I shudder to think
that I want
to research and learn another language, I am a strong supporter of
learning one
language well especially if that one language can do just about
anything you want to
do, i.e. web/asp, console/scripting, OS independent apps, and browser
plugin/scripting (a.la. moon light).
For the one time i can't use c# - linux kernel programming, at least
c# is close to C.
(and personally I know C from decades ago anyways).
I guess I am just wondering if RoR would be any benefit to me given
your statement that
"stellar productivity that can be achieved with Ruby on Rails is
hardly matched",
just makes me think I am missing something perhaps by not checking out
RoR.
-tl
vincentk
"""
I wonder what the policy is for the language shootout to use external
libraries. If its ok, I should contribute my port, if its not, the
Java test should be rewritten to be a fully managed implementation.
"""
As the benchmark rules point out [1], it's OK to use GMP:
"""
In addition to language specific multiprecision arithmetic, we will
accept programs that use GMP.
"""
So please consider posting your implementation ;-)
[1] http://shootout.alioth.debian.org/gp4/benchmark.php?test=pidigits&lang=all#about
Miguel de Icaza
>
> So please consider posting your implementation ;-)
I could not find a way of contributing the program, would you mind
contacting the authors and submitting my sample?
Miguel
Miguel de Icaza
> Can you explain or give an example of
> "stellar productivity that can be achieved with Ruby on Rails is
> hardly matched"
> within the context of comparison to C#.
My point is that raw performance power is not all you need in a
language, and I cited a number of languages that are much slower than
say C or C++, but yet, they yield incredible productivity gains.
I compare Java and C# because they are roughly on the same "tier", and
productivity-wise, I believe that the limitations in Java play in C#'s
favor.
> That is, what would be an example of something that
> "Ruby on Rails" would blow C#/Mono away on, with respect to
> productivity,
Again, I was not comparing those two tiers, why you would pick a
specific tier goes beyond a handful of language features.
> and is there issues (and improvements) required in C# to bridge this?
There are a few different elements; Rails is not only taking
advantage of the dynamic nature of Ruby, it also increases
productivity by choosing well know-patterns and forcing a particular
coding style over full configuration. If you can adapt your
workflow, your databases and your coding to the setup provided by
Rails, you can be very productive.
ASP.NET MVC in addition brings some of the features from Rails (MVC,
url routing) that some people like, and other systems like Subsonic
bring some of the ActiveRecord-like capabilities that Rails uses
extensively; But finally Rails is not a one-feature thing, its a
combination of them. They might be useful to you, or not. It all
depends on the case.
But again, i was not comparing those two, Rails is a framework, C# is
a language.
Miguel
Miguel de Icaza
> JVM. This comes especially after they actually did moved the Java to
> GPL - unlike mono -, and it's not at all easy for them, since they
> weren't Free from the beginning.
Well, mono has been free software/open source since the beginning. Am
not sure what you are trying to say here, that we are not?
> Imagine how many third party
> companies need to be contacted and asked about every line from some
> old libraries they might have developed at some point in time in let's
> say OpenSolaris. Just think a little bit at OpenSolaris, OpenOffice,
> NetBeans or even Java - these are huge moves towards Free software and
> you shouldn't try to minimize them. Even if now there are some parts
> which are not yet solved, I have no doubt, that in the next year they
> will be solved.
I was not trying to minimize those efforts; But the comment from Alvaro came
across as "Java is better than Mono" and I wanted to point out that
they might be,
but so far, a year after its release we have yet to see an official
release that is
open source (some folks have pointed out to IcedTea already).
> Another thing about the "bad Java" language this time, without
> closures, etc. etc., is that you might want to take a look at groovy
> (http://groovy.codehaus.org/).
I was comparing Java and C# as languages, not the virtual machines.
I for instance expect JRuby to have working closures, but that is not the point,
the point is that Java *as a language* is in trouble.
Some people seem to be fine fleeing Java and going to something else,
although it seems that the sanctioned platform these days is JavaFX.
> Yup, it compiles to java bytecode, and
> it has all the things you want - including support for production
> ready IDEs (like Eclipse, NetBeans or IntelliJ Idea) which I strongly
> doubt that mono has, from more than one vendor.
You are confusing the steak with the pepper sauce.
Mono the virtual machine does not need an IDE, its a runtime environment,
Groovy is a programming language.
As for IDEs that support C# there are a number of them, Visual Studio is
a popular choice, SharpDevelop (an open source effort) is another one.
You have a few cross platform solutions Monodevelop (an open source effort,
and you are correctly to point out, it is not on the same league as IntelliJ or
NetBeans or Eclipse).
Then you got some commercial IDEs like SlickEdit (it has C# and intellisense
support in a cross platform way) and X-Develop which supports Mono natively.
Finally, there are some plugins for Eclipse that support C#, but I do not use
Eclipse, so I do not know how good they are.
> If you can't find or
> do something in groovy I am sure that Guillaume Laforge can point you
> out, or heh, you can even make a feature request.
And again, I was not talking about Groovy or Java, or for that matter MacOS
or my wife's dog. I was talking about Java the language.
> And who cares that your product (mono) is working a tiny little bit
> faster than the other (java)?
Well, I for one, care, and a few other people care.
But this came as a reply to Alvaro's claim that Java came in #10, and Mono
in #18. Which is true with the given tests, but they are not comparing apples
to apples as I pointed out on my blog entry.
So to me, it is a perception issue: was Mono *that* much slower? Well, as it
turns out, it is not.
> What I am interested in is not being
> vendor locked and having a plenitude of libraries, frameworks and
> application servers to choose from. At these mono is light years
> behind...
Am not sure that you quite understand open source, there is not much
'vendor locking' when all the code you use is open source.
But then again, you seem like an individual that can easily confuse his
wife for a hat.
Miguel.
Miguel de Icaza
> ) is 2x slower with AOT activated !
Yes, I would not be surprised.
AOT code generates Position Independent Code (PIC) which is slower (as
it takes away a
register). What you loose in registers, you make up in memory usage
reduction when
multiple applications are running simultaneously.
Miguel.
Miguel de Icaza
> would bet per your point however that this is at the trivial level,
> but w/o touching the API I don't think you'll match here.
I find the Java one as confusing as the .NET one.
But in retrospect, I can not remember by hearth the C version of the regexp
libraries anyways, so I claim that all of these are unnatural.
> Granted I only spent
> a little bit of time digging into it as it became obvious that the
> "classpath sucks" issue was too large to get anywhere (sorry to my
> pals on #classpath but it was true).
I see what you were trying to do. Am not sure, this also feels
unnatural to me,
its like trying to walk with shoes made of marble, possible, but I do not see
the reason to.
What I would probably have done was to embrace the AppDomain model from
.NET which does the application/class/interface isolation for you. Now, how t
to integrate that with JBoss is a different story, and considering
that I have an
allergy to the mess that is J2EE (well, that and the doctor prescribed
that I stay
away from overengineered designs to prevent a hearth attack) am not sure that
I can give you much feedback.
> However I never figured out how you'd do "hot deployment" of multiple
> versions of a library in .NET w/o leaking even with s-b-s execution...
> Could be I'm missing something. Prolly making an ass out of myself
> with my swisscheese memory.
You use AppDomains, these are like processes, but instead of being full OS
processes, they are "virtual" processes created by the CLI VM.
> The .NET has only the concurrent
> GC which tends to underperform the parallel GCs on 4core+ machines (I
> only really deal with performance on big systems so your mileage may
> vary).
This is called "Server GC" in .NET (not supported in Mono), see the gcServer
flag for your .config file.
Anyways, I still believe it would be useful to test on Windows.
miguel
Ciprian M.
memory allocation should work better than a classic C/C++ malloc-new
cause of better cache hit ratio, the big impact on slowdown should be
on startup of Mono and in JITing the MSIL.
Mono miss on front of Java a decent GC (the allocating of multiple
objects in a simple program works more than 10 times in Java JVM or in
MS .NET runtime).
Another miss is the JIT, from my knowledge, Mono does not check the
host machine capabilities to use best instructions of the machine,
like SSE2 for floating loops on the 32 bits machines, MMX for memory
copy, but use an i386 + coprocessor. Even sounds hacky, probably the
JIT have to use them as mater of benchmarks and speed sake, that is
the industry standard way that the software works.
Mono has some benchmarks that it lose deeply: XMLMark, this shootout,
GC tests (cause of usage of Boehms GC), foreach loops (mostly fixed
today). One person suggest me: on MS .NET the parsing engine is
written in C (MS XML parser), you suggest that BigInteger is better
written in C.
Why not Mono take in account that benchmark and use the best on mater
of usability solution:
- map BigInteger's CLR on top of a C based library that does it speedy
number crouching
- map System.XML on top of LibXML2 or other very fast XML parsing
library
- finish (I know that is a waited feature) the Compacting GC to be on
the same class as Java on allocations/deallocations
- make the JIT to have some optimised libraries and runtime on the
popular instruction sets (MMX, SSE2) and replace it if the host
machine supports it
At the end will not be a speed killer for sure, but will make it look
more aggressively on a market that expects from Mono to be a close
competitor for .NET, not the little cousin of it.
Andy Oliver
> libraries anyways, so I claim that all of these are unnatural.
> > Granted I only spent
> > a little bit of time digging into it as it became obvious that the
> > "classpath sucks" issue was too large to get anywhere (sorry to my
> > pals on #classpath but it was true).
>
> I see what you were trying to do. Am not sure, this also feels
> unnatural to me,
> its like trying to walk with shoes made of marble, possible, but I do not see
> the reason to.
>
> What I would probably have done was to embrace the AppDomain model from
> .NET which does the application/class/interface isolation for you. Now, how t
> to integrate that with JBoss is a different story, and considering
> that I have an
> allergy to the mess that is J2EE (well, that and the doctor prescribed
> that I stay
> away from overengineered designs to prevent a hearth attack) am not sure that
> I can give you much feedback.
>
Fair with regards to the programming model. However, .NET is only now
catching up to the kind of full XA compliant, OR mapped stuff.
It is easy to argue "lightweight" until you have a bigger problem to
solve (2 databases, 1 messaging system, that all need to work in one
reliable
transaction). These things exist already in open source and for
little or no money in Java. They are only now really reaching any
kind of production level in
.NET.
By all means don't use distributed transactions or isolation or
dynamic classloading or any of that stuff if you don't need to...but I
guess I live in a world where
such things are necessary from time to time.
> > However I never figured out how you'd do "hot deployment" of multiple
> > versions of a library in .NET w/o leaking even with s-b-s execution...
> > Could be I'm missing something. Prolly making an ass out of myself
> > with my swisscheese memory.
>
> You use AppDomains, these are like processes, but instead of being full OS
> processes, they are "virtual" processes created by the CLI VM.
>
I'll have to look closer at how this evolved.
> > The .NET has only the concurrent
> > GC which tends to underperform the parallel GCs on 4core+ machines (I
> > only really deal with performance on big systems so your mileage may
> > vary).
>
> This is called "Server GC" in .NET (not supported in Mono), see the gcServer
> flag for your .config file.
>
> Anyways, I still believe it would be useful to test on Windows.
>
I hope to see the evolution of the higher end GC stuff in Mono. While
it does take 2x as much memory
as native code, when it is done well the concurrency can cause the
single thread performance to exceed "normal" manually
managed native.
-andy
> miguel
>
san...@gmail.com
Isaac Gouy
On Dec 29, 6:09 pm, "Miguel de Icaza" <miguel.de.ic...@gmail.com>
wrote:
> > So please consider posting your implementation ;-)
>
> I could not find a way of contributing the program, would you mind
> contacting the authors and submitting my sample?
Here are FAQ links on how to contribute source code attachments
http://shootout.alioth.debian.org/gp4/faq.php#play
Isaac Gouy
On Dec 29, 6:26 pm, "Miguel de Icaza" <miguel.de.ic...@gmail.com>
wrote:
-snip-
> > faster than the other (java)?
>
> Well, I for one, care, and a few other people care.
doesn't use GMP
2) "If you run all the tests ..."
- 'startup' just prints 'hello world', it's intended to give some
sense of the difference in startup times for different language
implementations, nothing more.
- 'meteor-contest' is different, we accept any algorithm for meteor-
contest, that's why we don't include meteor-contest in the rankings.
- 'chameneos-redux' is new, we don't have a Java implementation or a
C# implementation yet.
3) "It is interesting to see in the benchmarks the progression ..."
I'm pleased that you find it interesting - being curious and
interested and trying to understand really is the attitude to have
when looking at the benchmarks game.
"Fortran" - license restrictions prevent us from publishing
measurements of commercial (but freely available) Fortran
implementations :-(
4) "Ruby 1.9 will improve things..."
See http://shootout.alioth.debian.org/gp4sandbox/benchmark.php?test=all&lang=yarv
5) "Mono is a language-independent VM..."
And among the Extra language implementations we show - F#, IronPython,
Zonnon.
And we show other JVM languages - CAL, Groovy, Rhino, Nice, JRuby,
Scala
Isaac Gouy
> > faster than the other (java)?
>
> Well, I for one, care, and a few other people care.
Miguel de Icaza
> memory allocation should work better than a classic C/C++ malloc-new
> cause of better cache hit ratio, the big impact on slowdown should be
> on startup of Mono and in JITing the MSIL.
Well, there are a few elements that come into play that you might not
have accounted for:
1. C# and Java have to add extra code to protect for instance against
accidental out-of-bounds access (arrays bound checking), so this
inherently slows down the generated code (this is handled by the
low-level array-access instructions).
2. When you are JITing, you have to balance code quality for JIT time,
as you are doing things when the user expects immediate responses.
In Mono you can mitigate some of this by forcing the JIT to use a higher
optimization level (-O=all) and pay the price or precompiling (--aot, but
you loose a register).
3. Mono's optimizer is not as complete as GCC's code optimizer, and
in many instances we are aware of ways to fix it (our "linear-il" branch is
all about bringing some well-known optimizations that we were not able
to do with the current model to the JIT).
Now, JITs have an advantage that static compilers do not have, but to
this day we do not take advantage of this as much as we should. JITs
are able to generate CPU-specific code depending on the CPU they are
running at runtime. So if certain instructions are available, it would use
them (for example, we do that today with cmov, but we could take advantage
of more; Again, there are other issues to consider: testing is an important
issue, because all of a sudden your test matrix becomes larger).
> Mono miss on front of Java a decent GC (the allocating of multiple
> objects in a simple program works more than 10 times in Java JVM or in
> MS .NET runtime).
I would be interested in seeing your tests, I do not believe that we are ten
times slower than those.
But with GCs, you can always construct a test that will help a particular
GC algorithm. For instance, Mono's conservative GC in some scenarios
is better than .NET's. Now, the .NET GC is a lot more advanced, but there
is a lot more than meets the eye.
> Another miss is the JIT, from my knowledge, Mono does not check the
> host machine capabilities to use best instructions of the machine,
> like SSE2 for floating loops on the 32 bits machines, MMX for memory
> copy, but use an i386 + coprocessor. Even sounds hacky, probably the
> JIT have to use them as mater of benchmarks and speed sake, that is
> the industry standard way that the software works.
Correct. There is already code in Mono to handle SSE2 when running on
x86-64, but the code has never been "backported" to x86.
> Mono has some benchmarks that it lose deeply: XMLMark, this shootout,
> GC tests (cause of usage of Boehms GC), foreach loops (mostly fixed
> today). One person suggest me: on MS .NET the parsing engine is
> written in C (MS XML parser), you suggest that BigInteger is better
> written in C.
Have you ran XMLMark recently, or are you still using the 2-year old reports
on XMLMark? Because we did a lot of work all around and things have
improved significantly.
Also, a large performance speedup in XMLMark comes from using a new
XML API. Newer versions of XMLMark now use XLinq and just switching
this implementation on the .NET end give better performance numbers (so
this is an improvement that comes not from raw tuning of the VM or class
libraries, but an improvement from using a different approach).
The shootout numbers are only useful when you compare apples to apples,
they are never useful when they compare two different implementations of
the same problem.
At one point, they had a "threading" test, Mono's implementation used actual
threads, but someone contributed "cooperative-threading" implementations
that had none of the disadvantages of real threads (no need to synchronize)
for other languages.
> Why not Mono take in account that benchmark and use the best on mater
> of usability solution:
We do not mind looking at them, but its a matter of priorities, is improving
BigInteger performance more important than implementing a popular feature,
or supporting a popular platform, or improving other general areas, or
bringing new APIs, or reducing memory usage?
Some efforts take a long time to implement; the linear-il branch has been
under development for a year, and it will bring big performance improvements,
but its not something that can be done in a week, but the improvements will
benefit every single use case, as opposed to improving only a handful
of tests.
> - map BigInteger's CLR on top of a C based library that does it speedy
> number crouching
BigInteger is really an internal class that we use for implementing some
security bits (that is why the test has to link against
Mono.Security), you should
think of it really as an internal API, that someone just decided to
use to simplify
their particular test.
As I showed with my port, using GMP is trivial, it required less than
5 minutes to
"port" the Java version to C# to use GMP.
> - map System.XML on top of LibXML2 or other very fast XML parsing
> library
You can not map System.XML to libxml2; You could either google for the
reasons (this was discussed years ago) or you could try to do it, which would
probably be more educational.
> - finish (I know that is a waited feature) the Compacting GC to be on
> the same class as Java on allocations/deallocations
I believe you might be confusing features; a Compacting GC is independent
of some other features, and I can only second guess (because I do not understand
what you mean) what this particular feature is.
Would you mind testing Java vs Mono 1.2.6 and pinpoint where you think
allocations are 10 times faster?
> - make the JIT to have some optimised libraries and runtime on the
> popular instruction sets (MMX, SSE2) and replace it if the host
> machine supports it
Yeah, its a good idea.
> At the end will not be a speed killer for sure, but will make it look
> more aggressively on a market that expects from Mono to be a close
> competitor for .NET, not the little cousin of it.
There is a lot more when it comes to competing than raw power;
For example, with the Silverlight chess game, we get incredibly similar
performance boosts in the .NET vs JS tests; So similar that you can not
tell that our VM is different.
So again, its a matter of priorities.
Miguel.
Miguel de Icaza
>
> And among the Extra language implementations we show - F#, IronPython,
> Zonnon.
>
> And we show other JVM languages - CAL, Groovy, Rhino, Nice, JRuby,
> Scala
Oh, I wonder how I can get to see that page, I was not able to find it.
Miguel de Icaza
I get an error when I click the link, it says that it does not allow
Anonymous submissions.
Consider this email my request to get my port.cs for the pidigits test
submitted.
It gives a 4.7x performance improvement to C#, and also reduces memory
usage, as described on my blog.
Isaac Gouy
On Dec 30, 12:34 pm, "Miguel de Icaza" <miguel.de.ic...@gmail.com>
wrote:
http://shootout.alioth.debian.org/
so
http://shootout.alioth.debian.org/gp4sandbox/
Isaac Gouy
On Dec 30, 12:36 pm, "Miguel de Icaza" <miguel.de.ic...@gmail.com>
wrote:
-snip-
> Anonymous submissions.
Please try again.
I
berti_...@hotmail.com
where you can compare more languages against each other (even
languages where not all benchmark tests have been implemented).
(http://shootout.alioth.debian.org/sandbox/benchmark.php?
test=all&lang=all&xfullcpu=1&xmem=0&xloc=0&binarytrees=1&chameneosredux=1&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&nbody=1&nsieve=1&nsievebits=1&partialsums=1&pidigits=1&recursive=1®exdna=1&revcomp=1&spectralnorm=1&hello=1&sumcol=1&threadring=1&calc=Calculate)
In this list you can also find gcj, the java frontend for the gcc
compiler, which does produce native executables instead of class
files. When you compare the timings of the native compiled gcj
benchmark programs with the results of the managed java 6 server
programs, you will find something very interesting and illogical at
first:
http://shootout.alioth.debian.org/sandbox/benchmark.php?test=all&lang=gcj&lang2=java
The native executables nearly always perform better than the managed
ones as expected, except in the regex-dna and the pidigits test. And
when you look at the implementation of the pidigits test for gcj you
will see that this time it is a pure java implementation of the
pidigits test without the use of external libraries like gmp.
http://shootout.alioth.debian.org/sandbox/benchmark.php?test=pidigits&lang=gcj&id=0
And as you can see, the native compiled pure java version using the
biginteger class is about 20 times slower than the "tuned" version
which uses the gmp library. Strangely the regex-dna test being native
compiled by gcj runs about 56 times slower than the manged java 6
server one even though it is native compiled. Both versions use the
same source code. Does that mean, that Sun has optimized their regex
implementation in a special way since gcj is using as far as i know
the classpath libraries ?
Spam...@gmx.de
Language shootout has two main aspects:
1) Flaws in the Great Computer Language shootout itself:
* GCC has hand-tuned compiler parameters for every single program. If
you look at mono there is not a single optimization parameter
specified for neither the compiler nor the VM itself. Usually you can
get 10%-50% speed improvement just by picking the right compiler and
VM parameters for that program and machine
* Lots (All?) of the C# applications are just plain Java applications
with a C# syntax. How many applications use performance/memory
features such as pointers or structures?
* All benchmarks create output to stdout console (sometimes lots of
it). Every single one gets "tainted" with the potential speed problems
outputting to stdout.
* On the whole GCLS-Page you cannot find out which codepage stdout
uses on the system. Mono's speed may be affected heavily on choosing
the "right" or "wrong" one.
2) Mono is not particulary suited for (this kind of) microbenchmarks:
* It's not optimized toward console output (especially with "wrong"
codepages).
* Some key (common) base functions are relying on too general code
paths. In fact there doesn't seem too much interest in changing that
(e.g. there have been some proposals from me and someone else on the
mono-list to optimize IntXX.ToString() which is e.g. heavily used by
most of the benchmarks for a performance AND memory gain).
* Mono IS slow in several situations (In lots of microbenchmarks I
tend to see MS.Net to be about 2-2.5 times faster than mono). Some of
that comes from the VM (like non-optimal register allocation), from
unification of the codebase (to minimize the amount of code) and
suboptimal system resource use.
Eyal Alaluf
Java benchmarks are notoriously difficult to manage. We hit this issue
all the time. The VM is too complex for simple benchmarks to work.
Anyhow, when checking the binarytree benchmark the performance in Java
I get 4.53 seconds with -server and 1.91 seconds with -server -Xms256m
-Xmx256m.
The results vary by a factor of 2.4(!) Which makes the benchamrk
results difficult to verify and qualify.
And all of this is before handling the VM warmup issues which is
another factor of variance in Java.
Eyal.
Isaac Gouy
On Dec 30, 4:05 pm, berti_krue...@hotmail.com wrote:
-snip-
> compiler, which does produce native executables instead of class
> files.
JVM seems faster on Intel P4 than on AMD Sempron so also look at
http://shootout.alioth.debian.org/gp4sandbox/benchmark.php?test=all&lang=gcj
Isaac Gouy
On Dec 31, 8:46 am, Eyal Alaluf <ey...@mainsoft.com> wrote:
> Hi, Miguel.
>
> Java benchmarks are notoriously difficult to manage. We hit this issue
> all the time. The VM is too complex for simple benchmarks to work.
> Anyhow, when checking the binarytree benchmark the performance in Java
> I get 4.53 seconds with -server and 1.91 seconds with -server -Xms256m
> -Xmx256m.
> The results vary by a factor of 2.4(!) Which makes the benchamrk
> results difficult to verify and qualify.
binary trees
1.0 Java 6 -Xms64m #2 2.95 40,316 587
2.2 Java 6 -server #2 6.52 23,444 587
http://shootout.alioth.debian.org/gp4sandbox/benchmark.php?test=binarytrees&lang=all
> And all of this is before handling the VM warmup issues which is
> another factor of variance in Java.
http://shootout.alioth.debian.org/gp4sandbox/miscfile.php?file=dynamic&title=Java%20Dynamic%20Compilation
Isaac Gouy
On Dec 30, 6:01 pm, SpamSa...@gmx.de wrote:
> IMO the reason that Mono doesn't score very high on the Great ComputerLanguageshootouthas two main aspects:
> 1) Flaws in the Great ComputerLanguageshootoutitself:
> * GCC has hand-tuned compiler parameters for every single program. If
> you look at mono there is not a single optimization parameter
> specified for neither the compiler nor the VM itself. Usually you can
> get 10%-50% speed improvement just by picking the right compiler and
> VM parameters for that program and machine
have asked for particular compiler options. The sad thing is so few
people contributed C# programs that I ended up writing basic programs
- contribute better programs!
> * Lots (All?) of the C# applications are just plain Java applications
> with a C# syntax. How many applications use performance/memory
> features such as pointers or structures?
> * All benchmarks create output to stdout console (sometimes lots of
> it). Every single one gets "tainted" with the potential speed problems
> outputting to stdout.
> * On the whole GCLS-Page you cannot find out which codepage stdout
> uses on the system. Mono's speed may be affected heavily on choosing
> the "right" or "wrong" one.
and ask the benchmarks game admins to run it for you - ask on the
benchmarks game discussion forum where people will see your request
http://shootout.alioth.debian.org/gp4sandbox/faq.php#help
Miguel de Icaza
> have asked for particular compiler options. The sad thing is so few
> people contributed C# programs that I ended up writing basic programs
> - contribute better programs!
This makes sense, I have provided a couple.
> > * All benchmarks create output to stdout console (sometimes lots of
> > it). Every single one gets "tainted" with the potential speed problems
> > outputting to stdout.
>
> stdout is redirected to /dev/null
Yeah, but even while redirected Mono still outputs it, its the kernel that
discards the data, so the overhead is still there.
Am not sure myself how much of an issue this is.
Also, the console is usually slower than outputing the code to a file, or
to stdout, because the console class is guaranteed to be thread safe,
so every WriteLine takes a lock.
Not a major issue in my opinion, but still worth researching.
Miguel
Ciprian M.
> I believe you might be confusing features; a Compacting GC is independent
> of some other features, and I can only second guess (because I do not understand
> what you mean) what this particular feature is.
>
> Would you mind testing Java vs Mono 1.2.6 and pinpoint where you think
> allocations are 10 times faster?
>
making a comparison against one "static" JIT as has Mono. The program
to test that two runtimes, take in account only allocations and
hopefully expect that GC allocation will make the difference.
I was wrong, but still there is a somehow big difference. To consider
an actual Mono, I've took the SVN version, hopefully it doesn't add
regressions in the allocation code base.
To not give a chance for Java's JIT to remove useless code I had use
one static reference, that was need to be used for the next step, so
the JIT of Java hopefully will only optimise the code.
In rest, the code consists in the incrementing and copy simple values.
Java code:
public class ProgramRunner {
int val = 3;
static ProgramRunner q = new ProgramRunner();
void testAlloc()
{
for(int i = 0; i< 10000000; i++)
{
ProgramRunner p = new ProgramRunner();
p.val = q.val + 1;
q = p;
}
}
public static void main(String[] args) {
long start = System.currentTimeMillis();
new ProgramRunner().testAlloc();
long end = System.currentTimeMillis();
System.out.println("Time to finish:" + (end - start) + " " + q.val);
}
}
C# code:
using System;
class Program
{
int val = 3;
static Program q = new Program();
void allocTest()
{
for(int i = 0; i< 10000000; i++)
{
Program p = new Program();
p.val = q.val + 1;
q = p;
}
}
static void Main(string[] args)
{
int Start = Environment.TickCount;
new Program().allocTest();
int Stop = Environment.TickCount;
Console.WriteLine("Total time:{0} {1}",(Stop-Start), q.val);
}
}
Java commands for compile run:
Sun Java JRE 1.6.03
javac ProgramRunner.java
java -server ProgramRunner
javac ProgramRunner.java
java ProgramRunner
C# + Mono 1.2.6
gmcs -o+ Program.cs
mono --optimize=all Program.exe
gmcs -o+ Program.cs
mono Program.exe
That corresponds typically with the thin applications / high level
profile applications.
is mostly dependent of VM's allocation library
Best Java (client) time:
Time to finish:149 10000003
Worst java(client) time:
Time to finish:262 10000003
Best Java server time:
Time to finish:186 10000003
Worst Java server time:
Time to finish:230 10000003
Best Mono time:
Total time:477 10000003
Worst Mono time:
Total time:544 10000003
Best Mono (optimized) time:
Total time:473 10000003
Worst Mono (optimized) time:
Total time:786 10000003
Both Mono and Java does not show differences between unoptimized/
optimized times on a wide margin, the meaning is that is not JIT
dependent, but the library of allocation may do the difference. Mono
Optimized may take a bit too long to do the JIT, which makes even a
small program to run so slow.
For a sake of persons that want to have a measure of that tests, the
test machine was an up-to-date Ubuntu Linux 7.10 with Gnome, 32 bits,
host machine with 1.5 G RAM, CPU Pentium M Donthan at 1.73 GHz. I am
curious on matter of time, how will behave on a powerful workstation
which has SUSE ;), 64 bits, AMD Quad Core, that I have around, but
that when I will have time for playing.
Hope it helps,
Ciprian Mustiata
Miguel de Icaza
> Both Mono and Java does not show differences between unoptimized/
> optimized times on a wide margin, the meaning is that is not JIT
> dependent, but the library of allocation may do the difference. Mono
> Optimized may take a bit too long to do the JIT, which makes even a
> small program to run so slow.
These results are more inline with what I had expected, and the whole time
is spent inside the GC;
Here is a neat trick though: small objects in .NET have a smaller overhead
than full objects, replace the "class" with a struct (you will have to
initialize
"val" in a constructor) and the time goes from 514 with Mono to 80 with
mono on my machine.
Miguel
herve...@gmail.com
I'm sorry but I must say that l must disagree with some points of your
post:
- <i>Sun's open source one... is nowhere to be found yet:</i> that's
not the case, you just have to go to <a href="http://
openjdk.java.net/">OpenJDK</a> to found the sources on Mercurial, and
even the source bundles under GPLv2 (for the part that is opened,
which is all Hotspot+ compiler, and more than 95% of the Class
Library). They even provide a <href="http://nb-openjdk.netbeans.org/
get-and-build.html">tutorial</a> to build it by binding the opened
part with the remaining closed binaries. If you want something
entirely opened, you can go to <a href="http://icedtea.classpath.org/
wiki/Main_Page">IcedTea</a>, which replace the missing parts of
OpenJDK with Classpath GPL code... So what about your "nowhere to be
seen" ?. I wished Microsoft had opened some parts of .NET, but it will
probably will never be the case...
- Generics : I think that you misquoted what <a href="http://
weblogs.java.net/blog/arnold/archive/2005/06/
generics_consid_1.html">Ken Arnold</a> said. For him (as I
understand), the problem is not Java implementation of generics, but
the <b>fact that Java choose to add generics in the first place</b>.
As for me, I think it's just all right. I never do rocket science with
generics, I just use them to advertise what is to be found in
Collection parameters and return types, and I think it is very simple
for that. My point of view: if you want to do more than that in any
language, it is hard, and I don't think that C# is better than others
for that.
- catastrophic closure proposals: what is bad about controversy and
open discussion ? This is how the Java community works. This is not
how the .NET community works, because Microsoft decides first.
- lack of a strong language designer: hmm, I don't see anything harm
in asking people what they want, and judging by the result in the
example you linked, it is OK.
Hervé
Miguel de Icaza
> - Generics : I think that you misquoted what <a href="http://
> weblogs.java.net/blog/arnold/archive/2005/06/
> generics_consid_1.html">Ken Arnold</a> said. For him (as I
> understand), the problem is not Java implementation of generics, but
> the <b>fact that Java choose to add generics in the first place</b>.
That might have been Ken's point, but coming from C#, where generics
actually *make sense*, and where generics do not have all the artificial
limitations that come from type erasure it is pretty obvious that generics
were botched in Java.
Mastering generics in C# is very easy, there is really not much to it, it
works the way you expect them to.
> As for me, I think it's just all right. I never do rocket science with
> generics, I just use them to advertise what is to be found in
> Collection parameters and return types, and I think it is very simple
> for that. My point of view: if you want to do more than that in any
> language, it is hard, and I don't think that C# is better than others
> for that.
Implementing generics correctly means that you will actually use them
beyond using them in a handful of collections. There will be an actual
incentive to build on top of them
LINQ builds extensively on generics, I have a hard time picturing the
Java community producing something like LINQ at this point, not
without going back to the drawing board and fixing the mess that they
have in their hands.
> - catastrophic closure proposals: what is bad about controversy and
> open discussion ? This is how the Java community works. This is not
> how the .NET community works, because Microsoft decides first.
Nothing bad with the discussion, love it.
But the two main proposals are essentially train wrecks, just like generics
in Java is a train wreck. If discussion and controversy were ok, why did
Java generics get incorporated in that state? I can only imagine that the
debate did not go far enough.
And I bet that Sun did in the end say "we are doing generics this way",
this was probably driven by Sun's desire to avoid making changes to the
VM. Looking at the voting for JSR 14, you have to wonder if anyone
actually tried the code before they voted.
In fact, I believe that the issue with generics is that it was the "power plant"
example from the Bike Shed paper: http://www.bikeshed.com/
Basically, the proposal was so large that everyone assumed that whoever
proposed had done all the work and checked that it worked.
> - lack of a strong language designer: hmm, I don't see anything harm
> in asking people what they want, and judging by the result in the
> example you linked, it is OK.
Learn C# 3, use it for a project and it will become obvious to you.
It is hard to discuss color rendering with someone that only has a
monochromatic monitor.
Miguel.