[Caml-list] Is OCaml fast?

942 views
Skip to first unread message

Thanassis Tsiodras

unread,
Nov 22, 2010, 8:21:08 AM11/22/10
to caml...@yquem.inria.fr
I apologize beforehand if this is not the forum to ask.

I am on the fence about whether to learn OCaml or not, and while
reading an article called "Why OCaml"
(http://www.cs.ubc.ca/~murphyk/Software/Ocaml/why_ocaml.html), I saw
that OCaml was praised for the speed of the executables it generates -
and was referred to, speed-wise, as "second to none", except C and
C++.

However, when I actually went to the Language Shootout page suggested
in the article, I found out that OCaml is not 2nd, it is 13th, behind
languages like Haskell and C#...
(http://shootout.alioth.debian.org/u32/which-programming-languages-are-fastest.php)

Is it just hype, then? Or am I missing something?

--
What I gave, I have; what I spent, I had; what I kept, I lost. (Old Epitaph)

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Gregory Bellier

unread,
Nov 22, 2010, 8:36:27 AM11/22/10
to Thanassis Tsiodras, caml...@yquem.inria.fr
If you want speed then you should learn assembly or choose C as a second
choice.
It depends on the kind of apps you'd like to write. Even though O'Caml is
fast, it's not the first criteria I have in mind which would be security :
no segfault, no need to handle horrible stuff like in C, ...


2010/11/22 Thanassis Tsiodras <ttsi...@gmail.com>

> I apologize beforehand if this is not the forum to ask.
>
> I am on the fence about whether to learn OCaml or not, and while
> reading an article called "Why OCaml"

> (http://www.cs.ubc.ca/~murphyk/Software/Ocaml/why_ocaml.html<http://www.cs.ubc.ca/%7Emurphyk/Software/Ocaml/why_ocaml.html>),

Thomas Fischbacher

unread,
Nov 22, 2010, 8:42:41 AM11/22/10
to Gregory Bellier, caml...@yquem.inria.fr

Gregory Bellier wrote:

> If you want speed then you should learn assembly or choose C as a second
> choice.

Certainly not assembly. Modern microprocessors just see assembly as a
kind-of high level language which they interpret in a funny way, doing
all sorts of re-schedulings, register renamings, delayed branching,
speculative execution etc. Producing code that can take advantage of
this is best left to the compiler - unless one has to deal with
instruction set extensions for which there is no good compiler
support yet.

> It depends on the kind of apps you'd like to write. Even though O'Caml
> is fast, it's not the first criteria I have in mind which would be
> security : no segfault, no need to handle horrible stuff like in C, ...

I'd say the OCaml native compiler is reasonably fast for pretty much all
applications that need the speed of compiled code. But I'd say the same
for GHC and SBCL, say. There are some good reasons to take a closer look
at OCaml, but these are related to other qualities of the language.

In my view, its greatest benefit is that it makes working with closures
simple while being fast and by far not as intimidating to (prospective)
PhD students as Scheme/Lisp (but that only because most of them have
been spoilt by imperative languages beforehand).

--
best regards,
Thomas Fischbacher
t.fisc...@soton.ac.uk

Sylvain Le Gall

unread,
Nov 22, 2010, 8:43:32 AM11/22/10
to caml...@inria.fr
Hello,

On 22-11-2010, Thanassis Tsiodras <ttsi...@gmail.com> wrote:
> I apologize beforehand if this is not the forum to ask.
>
> I am on the fence about whether to learn OCaml or not, and while
> reading an article called "Why OCaml"
> (http://www.cs.ubc.ca/~murphyk/Software/Ocaml/why_ocaml.html), I saw
> that OCaml was praised for the speed of the executables it generates -
> and was referred to, speed-wise, as "second to none", except C and
> C++.
>
> However, when I actually went to the Language Shootout page suggested
> in the article, I found out that OCaml is not 2nd, it is 13th, behind
> languages like Haskell and C#...
> (http://shootout.alioth.debian.org/u32/which-programming-languages-are-fastest.php)

Shootout benchmarks doesn't always allow to set some variables that
could greatly improve the speed of OCaml.

>
> Is it just hype, then? Or am I missing something?
>

You could write very fast application with OCaml -- even beating C code
in some case. But what OCaml is really helpful for, is that you can do
algorithmic optimizations that improve the speed. It means that OCaml
is terse enough to allow you to write complex algorithm without making
too much bugs.

All in all: development time + execution time is fast.

But if you spend 10x development time on the same C code, you will
obviously get something faster in C.

Regards,
Sylvain Le Gall

Dario Teixeira

unread,
Nov 22, 2010, 8:55:10 AM11/22/10
to caml...@yquem.inria.fr, Thanassis Tsiodras
Hi,

> I am on the fence about whether to learn OCaml or not, and while
> reading an article called "Why OCaml"
> (http://www.cs.ubc.ca/~murphyk/Software/Ocaml/why_ocaml.html),
> I saw that OCaml was praised for the speed of the executables it
> generates - and was referred to, speed-wise, as "second to none",
> except C and C++.

Yes, Ocaml is fast (more on that later), but you should consider speed
as just the icing in the cake. Besides all the cool language features,
what makes me recommend Ocaml to people who -- like you -- may be on
the fence, is that among the not so mainstream languages, Ocaml is
perhaps the one people are more likely to stick with for actual work
after they get past the learning stage. Remember that though Ocaml
favours the functional paradigm, it is flexible enough to support those
(few) situations where it actually makes sense to be imperative and/or
use an object oriented approach, and where a purely functional solution
is just intellectual masturbation.

> However, when I actually went to the Language Shootout page suggested
> in the article, I found out that OCaml is not 2nd, it is 13th, behind
> languages like Haskell and C#...
> (http://shootout.alioth.debian.org/u32/which-programming-languages-are-fastest.php)
>
> Is it just hype, then? Or am I missing something?

There's lies, damn lies, and shootout statistics. Getting good performance
in Haskell is often something of a black art that baffles even gurus. In
contrast, Ocaml has very good "performance predictability". Moreover, it
is possible to get good performance from fairly idiomatic Ocaml programmes,
whereas in other languages you may need to write ugly code which looks like
nothing one would write for elegance and sanity.

Best regards,
Dario Teixeira

Gerd Stolpmann

unread,
Nov 22, 2010, 9:04:42 AM11/22/10
to Thanassis Tsiodras, caml...@yquem.inria.fr
Am Montag, den 22.11.2010, 15:21 +0200 schrieb Thanassis Tsiodras:
> I apologize beforehand if this is not the forum to ask.
>
> I am on the fence about whether to learn OCaml or not, and while
> reading an article called "Why OCaml"
> (http://www.cs.ubc.ca/~murphyk/Software/Ocaml/why_ocaml.html), I saw
> that OCaml was praised for the speed of the executables it generates -
> and was referred to, speed-wise, as "second to none", except C and
> C++.
>
> However, when I actually went to the Language Shootout page suggested
> in the article, I found out that OCaml is not 2nd, it is 13th, behind
> languages like Haskell and C#...
> (http://shootout.alioth.debian.org/u32/which-programming-languages-are-fastest.php)
>
> Is it just hype, then? Or am I missing something?

I think the shootout is not a good data source. There are definitely
some very poor Ocaml results there, so I'd guess the shootout got
recently more attention by enthusiasts of other languages, and the
current Ocaml programs there are not very good. (I remember Ocaml was #1
at the shootout a few years ago, faster than C.) So maybe a good
opportunity to post better Ocaml solutions there?

Gerd
--
------------------------------------------------------------
Gerd Stolpmann, Bad Nauheimer Str.3, 64289 Darmstadt,Germany
ge...@gerd-stolpmann.de http://www.gerd-stolpmann.de
Phone: +49-6151-153855 Fax: +49-6151-997714
------------------------------------------------------------

Sylvain Le Gall

unread,
Nov 22, 2010, 9:22:29 AM11/22/10
to caml...@inria.fr
On 22-11-2010, Gerd Stolpmann <in...@gerd-stolpmann.de> wrote:
> Am Montag, den 22.11.2010, 15:21 +0200 schrieb Thanassis Tsiodras:
>> I apologize beforehand if this is not the forum to ask.
>>
>> I am on the fence about whether to learn OCaml or not, and while
>> reading an article called "Why OCaml"
>> (http://www.cs.ubc.ca/~murphyk/Software/Ocaml/why_ocaml.html), I saw
>> that OCaml was praised for the speed of the executables it generates -
>> and was referred to, speed-wise, as "second to none", except C and
>> C++.
>>
>> However, when I actually went to the Language Shootout page suggested
>> in the article, I found out that OCaml is not 2nd, it is 13th, behind
>> languages like Haskell and C#...
>> (http://shootout.alioth.debian.org/u32/which-programming-languages-are-fastest.php)
>>
>> Is it just hype, then? Or am I missing something?
>
> I think the shootout is not a good data source. There are definitely
> some very poor Ocaml results there, so I'd guess the shootout got
> recently more attention by enthusiasts of other languages, and the
> current Ocaml programs there are not very good. (I remember Ocaml was #1
> at the shootout a few years ago, faster than C.) So maybe a good
> opportunity to post better Ocaml solutions there?
>

Maybe it is time to join forces and update the shootout for OCaml?

If some people are interested here, I can setup a repository on the forge
to update all this tests.

If I get 2 people interested, I will setup a VCS repository + project on
the forge, this afternoon. Drop me an email + preferred VCS (among svn,
git, darcs) and your account login on http://forge.ocamlcore.org. I will
take care, when ready, to made this code available in the shootout once
finished.

Regards,
Sylvain Le Gall

bluestorm

unread,
Nov 22, 2010, 9:37:00 AM11/22/10
to Gerd Stolpmann, caml...@yquem.inria.fr
On Mon, Nov 22, 2010 at 3:04 PM, Gerd Stolpmann <in...@gerd-stolpmann.de>wrote:

> I think the shootout is not a good data source. There are definitely
> some very poor Ocaml results there, so I'd guess the shootout got
> recently more attention by enthusiasts of other languages, and the
> current Ocaml programs there are not very good. (I remember Ocaml was #1
> at the shootout a few years ago, faster than C.) So maybe a good
> opportunity to post better Ocaml solutions there?
>

As Sylvain noticed, some (in not most) of the OCaml poor performances in the
shootout are actually not due to bad OCaml programs, but to arbitrary
restrictions in the shootout rules. For example, one of the bad-performing
benchmark for OCaml is the binary-tree benchmark, where it is nearly four
times slower than C, but on closer inspection you discover that this is due
to the arbitrary choice to forbid any change of the GC parameters. With
appropriate GC parameters, the very same OCaml program is exactly as fast as
C.

http://shootout.alioth.debian.org/u32/performance.php?test=binarytrees

« Note: these programs are being measured with *the default initial heap
size* - the measurements may be very different with a larger initial heap
size or GC tuning. »
C version : 12.11 secs
OCaml version : 47.22 secs
OCaml version with GC parameters tuned ("interesting alternative" section) :
12.67 secs


Therefore, there is nothing that can be changed to the OCaml submission for
this benchmark to improve performances, except changing the default GC
parameters; while this might be a good idea in general, changing it only for
the sake of shootout-obsessed people is ridiculous.

Török Edwin

unread,
Nov 22, 2010, 10:01:57 AM11/22/10
to bluestorm, Gerd Stolpmann, caml...@yquem.inria.fr

Isn't it possible for the GC to realise its doing too many collections
and increase the minor heap size on its own?
Or isn't it possible to determine at compile time an approximation for
a good heap size, and use that?

Best regards,
--Edwin

Gerd Stolpmann

unread,
Nov 22, 2010, 10:02:11 AM11/22/10
to bluestorm, caml...@yquem.inria.fr
Am Montag, den 22.11.2010, 15:36 +0100 schrieb bluestorm:
> On Mon, Nov 22, 2010 at 3:04 PM, Gerd Stolpmann
> <in...@gerd-stolpmann.de> wrote:
> I think the shootout is not a good data source. There are
> definitely
> some very poor Ocaml results there, so I'd guess the shootout
> got
> recently more attention by enthusiasts of other languages, and
> the
> current Ocaml programs there are not very good. (I remember
> Ocaml was #1
> at the shootout a few years ago, faster than C.) So maybe a
> good
> opportunity to post better Ocaml solutions there?
>
>
> As Sylvain noticed, some (in not most) of the OCaml poor performances
> in the shootout are actually not due to bad OCaml programs, but to
> arbitrary restrictions in the shootout rules. For example, one of the
> bad-performing benchmark for OCaml is the binary-tree benchmark, where
> it is nearly four times slower than C, but on closer inspection you
> discover that this is due to the arbitrary choice to forbid any change
> of the GC parameters. With appropriate GC parameters, the very same
> OCaml program is exactly as fast as C.
>
>
> http://shootout.alioth.debian.org/u32/performance.php?test=binarytrees
>
>
> « Note: these programs are being measured with the default initial
> heap size - the measurements may be very different with a larger

> initial heap size or GC tuning. »
> C version : 12.11 secs
> OCaml version : 47.22 secs
> OCaml version with GC parameters tuned ("interesting alternative"
> section) : 12.67 secs
>
>
>
>
> Therefore, there is nothing that can be changed to the OCaml
> submission for this benchmark to improve performances, except changing
> the default GC parameters; while this might be a good idea in general,
> changing it only for the sake of shootout-obsessed people is
> ridiculous.

It's in deed an unfair comparison: In C they use the Apache runtime
which provides memory pools. This is something that does not extend to
most real world programs.

Because it's ridiculous anyway: Encode the tree in an array. Not really
idiomatic, but in C they also do not use the idiomatic memory management
(malloc/free).

Goswin von Brederlow

unread,
Nov 22, 2010, 11:00:20 AM11/22/10
to Edwin, Gerd Stolpmann, caml...@yquem.inria.fr
Török Edwin <edwin...@gmail.com> writes:

Maybe. The GC would have to see that with a larger minor heap it has
sufficient space so that most values die before the collection. It could
see that most values die after 1 major sweep and increasing the minor
heap would likely keep them out of the major heap alltogether.

> Or isn't it possible to determine at compile time an approximation for
> a good heap size, and use that?

Usualy that is dependent on the input.

MfG
Goswin

Fabrice Le Fessant

unread,
Nov 22, 2010, 11:40:02 AM11/22/10
to Gerd Stolpmann, caml...@yquem.inria.fr
On Mon, Nov 22, 2010 at 3:04 PM, Gerd Stolpmann <in...@gerd-stolpmann.de> wrote:
> So maybe a good opportunity to post better Ocaml solutions there?

I spent some time improving OCaml solutions, and most of the time, my
solutions were refused: the organizers don't let you unroll loops, fix
GC parameters, etc. One strength of OCaml is that it runs fairly fast
immediatly, but you can make it run even faster with some tuning,
something that you cannot do with many other languages. Such tuning is
not allowed there, so, it is not a good place for OCaml hackers. If
the organizers had been consistent, they would have forbidden Haskell
programmers from adding strictness in their programs, but then, there
programs would have been 2x slower than OCaml programs.

Well, actually, things might improve in the future, if we manage to
improve the performance of the compiler itself, and have more
"official" libraries. For example, most parallel solutions in OCaml
currently use fork() and sockets for communication. Using an official
shared-memory library, we could directly improve the performances of
many OCaml solutions.

--Fabrice

Fabrice Le Fessant

unread,
Nov 22, 2010, 11:46:56 AM11/22/10
to Török Edwin, Gerd Stolpmann, caml...@yquem.inria.fr
2010/11/22 T�r�k Edwin <edwin...@gmail.com>:

> Isn't it possible for the GC to realise its doing too many collections
> and increase the minor heap size on its own?

Indeed, it could notice that a lot of data is being moved to the major
heap, and double its size in consequence, until a maximum limit is
reached.

The problem is that it is the kind of things that are application
dependent, and should be put in the program itself (the program would
have a trigger on each minor heap collection, and, depending on the
moved bytes, would increase the size of the minor heap). The problem
is that the Shootout does not allow that, so the winner is the
language whose runtime allocates the most memory at the beginnning...

--Fabrice

Oliver Bandel

unread,
Nov 22, 2010, 12:02:10 PM11/22/10
to caml...@yquem.inria.fr
Zitat von "Gerd Stolpmann" <in...@gerd-stolpmann.de>:
[...]

> (I remember Ocaml was #1
> at the shootout a few years ago, faster than C.) So maybe a good
> opportunity to post better Ocaml solutions there?
[...]

Yes I also remember that.
I hope that the new OCaml compilers did not
make OCaml lessperformance by enhancing other features.

And I don't realy think so.

But were the old code-snippets emoved, or what was going on,
that OCaml degraded that much?

From my experience - normally using ocamlc for most of my OCaml
programs - it's even fast enough with this bytecode.
If it will be not fast enough, I have an option as a joker:
compiling to native code. :)

Doing this is rather psychologically, as I like to have a potential
for enhancement. :)

If you really always need the best performance, of course compiling
to native code will be best choice.
But at lest for my work I have not found out really bottlenecks,
and always was faster than doing things in Perl or Python or so.

Learning OCaml IMHO does make sense in any case. :)

But be aware one big disadvantage: if you have learned Ocaml,
this may absolutely kill your motivation in learning other languages
afterwards.So,if you need some other languages for your job, learn
them first, and learn OCaml after this, for your pleasure.

But you may never have fun at work then, until your boss allows you to
use OCaml.

And this is not a joke, I mean it as I wrote it!

Ciao,
Oliver

David Rajchenbach-Teller

unread,
Nov 22, 2010, 12:32:24 PM11/22/10
to Oliver Bandel, caml...@yquem.inria.fr
I can confirm that old code-snippets were removed (and that both faster solutions and environment variable tweaks were rejected).

On Nov 22, 2010, at 6:02 PM, Oliver Bandel wrote:

> Zitat von "Gerd Stolpmann" <in...@gerd-stolpmann.de>:
> [...]
>> (I remember Ocaml was #1
>> at the shootout a few years ago, faster than C.) So maybe a good
>> opportunity to post better Ocaml solutions there?
> [...]
>
> Yes I also remember that.
> I hope that the new OCaml compilers did not
> make OCaml lessperformance by enhancing other features.
>
> And I don't realy think so.
>
> But were the old code-snippets emoved, or what was going on,
> that OCaml degraded that much?
>

_______________________________________________

Oliver Bandel

unread,
Nov 22, 2010, 12:49:05 PM11/22/10
to David Rajchenbach-Teller, caml...@yquem.inria.fr

..hmhhh..

..looks like they are biased...


... not that we are not ;)

..but... as the GC-stuff is available FROM WITHING the language,
in the standard-lib, this is nothing added on later.


And I think it should also be allowed to be used.

To reject environment variables, I can see as acceptable in this case,
but rejecting the GC-stuff does not make sense, because, as just
mentioned, it is avalable by the programmer from within the code.

What about compiling parameters?
I mean: in C you can use -O for optimization.
This should also be forbidden then.... Is it?

There are so much possibilities to influence the results,
that blocking Gc-module is idiotic, IMHO.


Ciao,
Oliver


P.S.:
I looked at one of the C-makefiles:

usr/bin/gcc -pipe -Wall -O3 -fomit-frame-pointer -march=native
-fopenmp -D_FILE_OFFSET_BITS=64 -I/usr/include/apr-1.0 -lapr-1 -lgomp
binarytrees.gcc-7.c -o binarytrees.gcc-7.gcc_run
rm binarytrees.gcc-7.c


So, -O3 is allowed.
AFAIK with O3 and higher, inline does work.
__inline__ must be forbidden as well as -O3

Optimization should be switched off completely, if
OCaml's optimizations are also not allowed.

Zitat von "David Rajchenbach-Teller" <David....@univ-orleans.fr>:

David Rajchenbach-Teller

unread,
Nov 22, 2010, 12:54:25 PM11/22/10
to Oliver Bandel, caml...@yquem.inria.fr
Note: I'm not saying that they are biased. It's quite possible they did the same thing for other languages, too, I didn't take the time to check.

On Nov 22, 2010, at 6:23 PM, Oliver Bandel wrote:

>
> ...hmhhh..
>
> ...looks like they are biased...
>
>
> .... not that we are not ;)

Message has been deleted

Oliver Bandel

unread,
Nov 22, 2010, 1:33:15 PM11/22/10
to caml...@yquem.inria.fr
Hi,

Zitat von "Thanassis Tsiodras" <ttsi...@gmail.com>:

> I apologize beforehand if this is not the forum to ask.
>
> I am on the fence about whether to learn OCaml or not, and while
> reading an article called "Why OCaml"
> (http://www.cs.ubc.ca/~murphyk/Software/Ocaml/why_ocaml.html), I saw
> that OCaml was praised for the speed of the executables it generates -
> and was referred to, speed-wise, as "second to none", except C and
> C++.

[...]


I remember one argument on that topic, many years ago,
maybe rom Xavier leoy or other core developers, and it goes like that:

Benchmarks on such small problem solving is ONE comparision, what
makes sense, but it must be added by comparisions of "optimizations"
of the code in large applications.

The lower the level of the öanguage the harder it is to keep track of
possible optimizations on a very big program.

But the higher the language level is, the easier it is to
keep that in mind.

So: when optmizing small snippets of code this helps optimizing
performance in maybe critical sections, but to get an overview on all
interdependencies is close to unmanagable with lower leveled languages.

I have not seen a comparison of performance in this repect,
but I also would think higher level languages might be a better choice here.


Any comparisions on that scale of programming, which you know of?

Then please throw in a link here.

Thanks.

Ciao,
Oliver

Török Edwin

unread,
Nov 22, 2010, 1:33:45 PM11/22/10
to Fabrice Le Fessant, Gerd Stolpmann, caml...@yquem.inria.fr
On Mon, 22 Nov 2010 17:46:49 +0100
Fabrice Le Fessant <fab...@lefessant.net> wrote:

> 2010/11/22 Török Edwin <edwin...@gmail.com>:


> > Isn't it possible for the GC to realise its doing too many
> > collections and increase the minor heap size on its own?
>
> Indeed, it could notice that a lot of data is being moved to the major
> heap, and double its size in consequence, until a maximum limit is
> reached.

I did some benchmarks on 2 CPUs, and here are the wall clock times
for GC minor heap size:

"minorheap"," phenomII",,," core2",,,"cycle diff %"
," time",," cycles","time",," cycles",
32," 17.76","17.74",56.8," 48.60","48.86",58.48,2.96
64," 16.81","16.80",53.78,"44.79","44.79",53.75,-0.06
128," 15.41","15.38",49.26,"41.38","40.76",49.28,0.04
256," 14.39","14.35",45.98,"39.75","38.98",47.24,2.74
512," 12.97","13.15",41.79,"35.75","35.59",42.8,2.42
1024," 12.89","12.94",41.33,"33.57","33.13",40.02,-3.17
2048," 11.05","11.09",35.42,"29.26","29.12",35.03,-1.1
4096," 9.79"," 9.81",31.36,"26.21","25.96",31.3,-0.19
8192," 8.71"," 8.85",28.1," 23.36","23.34",28.02,-0.28
16384," 7.89"," 7.86",25.2," 21.41","21.54",25.77,2.26
32768," 7.55"," 7.55",24.16,"20.74","20.88",24.97,3.35

(minorheap is in KWords, time is in seconds, cycles is divided by 10^9)

Increasing minor heap beyond that yielded no improvement (number of
minor heap collections stayed the same).

phenomII has 64 Kb L1 data cache, 512Kb L2 cache, 6144Kb L3 cache
(shared), runs at 3.2Ghz. That would be 516k words if only 1 core used.

core2 has 32 Kb L1 data cache, 4MB L2 cache, runs at 1.2Ghz.
That would be 840k words if only 1 core used.

Both used exact same binaries on 64-bit Linux, ocaml 3.11.2.

Despite the difference in CPUs and heap sizes the number of CPU cycles
spent for a given size of the minor heap is about the same (within 3%
difference).

Not sure what the max should be for the minor heap increase, but based
on this benchmark increasing size of minor heap never slows down the
program. Even when size of minor heap exceeds what fits in the cache.
I guess there is another microbenchmark that can show the opposite
though.

Is there some real world benchmark for OCaml's GC that can be used
to get some better values for minor heap size based on CPU's L2/L3 size
(as suggested in the 'Optimizing garbage collection' thread)?


>
> The problem is that it is the kind of things that are application
> dependent,

Yes, maybe optimal size of minor heap doesn't depend on CPU's cache
size but on the application, in which case a heuristic for
increasing/decreasing the heap size may be better?

> and should be put in the program itself (the program would
> have a trigger on each minor heap collection, and, depending on the
> moved bytes, would increase the size of the minor heap). The problem
> is that the Shootout does not allow that, so the winner is the
> language whose runtime allocates the most memory at the beginnning...

If tuning minor heap can double performance of the program, then the GC
should have some heuristics to tune it automatically. Sure applications
which are not happy with that tuning should tune it themselves.

Best regards,
--Edwin

Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted

Eray Ozkural

unread,
Nov 22, 2010, 6:28:34 PM11/22/10
to David Rajchenbach-Teller, caml...@yquem.inria.fr, Oliver Bandel
It's always funny to look at benchmarks made by amateurs. We get it all the
time in parallel computing, when some sharp programmer decides to implement
a parallel algorithm in Java. :) I keep asking myself questions like do you
also program ciphers in Flash? Graph algorithms in PHP?

In my experience, ocaml is pretty fast, quite comparable to C++, for
computation-intensive tasks and of course excellent at traditional
functional programming tasks. Since I use both kinds of algorithms in my
research, ocaml is better than C++ for me (despite the great boost libs),
except for shared memory parallelism, which is still non-existent in ocaml.

Try implementing something that has a large time/memory complexity like a
data mining algorithm, and you'll see the difference from Haskell/Java/Blah.
Just run the algorithms in Weka, the famous data mining suite written in
Java, and compare them to any sensible implementation of the same
algorithms. Measure the difference, it's unbelievable how awful Java is at
any kind of serious computing! I think some conservative humans from the
future sent the designers of Java back in time to stop the singularity.

Cheers,

--
Eray Ozkural, PhD candidate. Comp. Sci. Dept., Bilkent University, Ankara
http://groups.yahoo.com/group/ai-philosophy
http://myspace.com/arizanesil http://myspace.com/malfunct

Jeff Schultz

unread,
Nov 22, 2010, 6:56:12 PM11/22/10
to Oliver Bandel, caml...@yquem.inria.fr
On Mon, Nov 22, 2010 at 06:23:38PM +0100, Oliver Bandel wrote:
> ...looks like they are biased...

> .... not that we are not ;)

> To reject environment variables, I can see as acceptable in this case,
> but rejecting the GC-stuff does not make sense, because, as just mentioned,
> it is avalable by the programmer from within the code.

I haven't looked at the "benchmark game" or its predecessor for many
years, but all this really says is that the programs it measures are
sufficiently unlike real applications that the OCaml developers
haven't been tempted to tune the GC defaults for them.

The conclusion to draw would be that the shootout results are a
poor, and probably misleading, evidence base for selecting a
programming language for execution time performance. Measured
programming language implementation performance gets faster or
slower over the years due to its own rule changes and language and
library implementation changes, but I doubt that this conclusion
changes much ;-)


The one thing the shootout programs are useful for is to get a feel
for a variety of different languages' expressivity for a common set
of "small" problems.


Jeff Schultz

Isaac Gouy

unread,
Nov 22, 2010, 9:00:16 PM11/22/10
to caml...@inria.fr
Thanassis Tsiodras <ttsiodras <at> gmail.com> writes:

-snip-


> Is it just hype, then? Or am I missing something?

Note the "Why Ocaml?" date - December 2002.

Isaac Gouy

unread,
Nov 22, 2010, 9:05:13 PM11/22/10
to caml...@inria.fr
David Rajchenbach-Teller <David.Teller <at> univ-orleans.fr> writes:


> I can confirm that old code-snippets were removed (and that both faster
solutions and environment
> variable tweaks were rejected).


Even back in 2001, Doug Bagley had noted all the things that were
wrong with the tasks on his "The Great Computer Language Shootout".

During autumn 2004, on Brent Fulgham's website, new tasks were added
and old Doug Bagley tasks removed.

By 2005 only 2 tasks remained from that old Doug Bagley website.

By August 2008 none of those old Doug Bagley tasks were measured for
the current benchmarks game.

Isaac Gouy

unread,
Nov 22, 2010, 9:10:14 PM11/22/10
to caml...@inria.fr
Gerd Stolpmann <info <at> gerd-stolpmann.de> writes:

-snip-


> I think the shootout is not a good data source. There are definitely
> some very poor Ocaml results there, so I'd guess the shootout got
> recently more attention by enthusiasts of other languages, and the
> current Ocaml programs there are not very good. (I remember Ocaml was #1
> at the shootout a few years ago, faster than C.) So maybe a good
> opportunity to post better Ocaml solutions there?


Even back in 2001, on Doug Bagley's website called "The Great Computer Language
Shootout" Ocaml was not #1-

http://web.archive.org/web/20010602051335/www.bagley.org/~doug/shootout/craps.shtml

Doug Bagley stopped updating his website in Fall 2001.


Even back in 2004, when Brent Fulgham started his website using the same
programs, OCaml was not #1

http://web.archive.org/web/20040717184911/shootout.alioth.debian.org/craps.php

Isaac Gouy

unread,
Nov 22, 2010, 9:15:11 PM11/22/10
to caml...@inria.fr
bluestorm <bluestorm.dylc <at> gmail.com> writes:

-snip-


> With appropriate GC parameters, the very same OCaml program is exactly as fast
as C.
>
>
> http://shootout.alioth.debian.org/u32/performance.php?test=binarytrees
>

> « Note: these programs are being measured with the default initial heap size -


the measurements may be very different with a larger initial heap size or GC
tuning. »
>
>
> C version : 12.11 secs
> OCaml version : 47.22 secs
> OCaml version with GC parameters tuned ("interesting alternative" section) :
12.67 secs

And of course you know because that GC tuned OCaml program is shown on the
benchmarks game website ;-)

http://shootout.alioth.debian.org/u32/program.php?test=binarytrees&lang=ocaml&id=1

Isaac Gouy

unread,
Nov 22, 2010, 9:20:12 PM11/22/10
to caml...@inria.fr
Dario Teixeira <darioteixeira <at> yahoo.com> writes:
-snip-
> There's lies, damn lies, and shootout statistics.
-snip-

After all, facts are facts,
and although we may quote one to another with a chuckle
the words of the Wise Statesman, 'Lies--damned lies--and statistics,'
still there are some easy figures the simplest must understand,
and the astutest cannot wriggle out of.

Leonard Henry Courtney, 1895

:-)

Fabrice Le Fessant

unread,
Nov 23, 2010, 4:15:44 AM11/23/10
to caml...@yquem.inria.fr
Maybe you should read "Tainted Truth: The Manipulation of Fact In
America" by Cynthia Crossen ?

--Fabrice

Isaac Gouy wrote, On 11/23/2010 03:20 AM:
> Dario Teixeira <darioteixeira <at> yahoo.com> writes:
> -snip-
>> There's lies, damn lies, and shootout statistics.
> -snip-
>
> After all, facts are facts,
> and although we may quote one to another with a chuckle
> the words of the Wise Statesman, 'Lies--damned lies--and statistics,'
> still there are some easy figures the simplest must understand,
> and the astutest cannot wriggle out of.
>
> Leonard Henry Courtney, 1895

_______________________________________________

Jon Harrop

unread,
Nov 23, 2010, 10:50:59 AM11/23/10
to caml...@inria.fr
And almost all of the "Haskell" solutions (reverse-complement,
spectral-norm, Mandelbrot, n-body, fannkuch-redux, k-nucleotide, regex-dna)
abuse GHC's FFI in order to work around Haskell.

The k-nucleotide benchmark in Haskell even uses malloc! Ketil Malde crafted
a much better solution but noted:

"This is considered cheating, since it is the easy and natural way to do
it." -
http://www.haskell.org/haskellwiki/Shootout/Knucleotide

Cheers,
Jon.

> -----Original Message-----
> From: caml-lis...@yquem.inria.fr [mailto:caml-list-
> bou...@yquem.inria.fr] On Behalf Of Christophe TROESTLER
> Sent: 23 November 2010 10:38
> To: igo...@yahoo.com
> Cc: caml...@inria.fr
> Subject: Re: [Caml-list] Re: Is OCaml fast?


>
> On Tue, 23 Nov 2010 02:03:48 +0000, Isaac Gouy wrote:
> >
> > > C version : 12.11 secs
> > > OCaml version : 47.22 secs
> > > OCaml version with GC parameters tuned ("interesting alternative"
> > section) : 12.67 secs
> >
> > And of course you know because that GC tuned OCaml program is shown
> > on the
> > benchmarks game website ;-)
> > http://shootout.alioth.debian.org/u32/program.php?test=binarytrees&
> > lang=ocaml&id=1
>

> Since you are here, please explain why C can use memory pools and vec
> tor instructions but tuning the GC of OCaml ― although it is part of
> the standard library ― is considered an "alternative".
>
> Best,
> C.

Jon Harrop

unread,
Nov 23, 2010, 11:08:42 AM11/23/10
to Christophe TROESTLER, caml...@inria.fr
Note that the regex-dna solution for Haskell tweaks its GC parameters via
the -H command-line parameter:

http://shootout.alioth.debian.org/u64q/program.php?test=regexdna&lang=ghc&id
=2

> -----Original Message-----
> From: caml-lis...@yquem.inria.fr [mailto:caml-list-
> bou...@yquem.inria.fr] On Behalf Of Christophe TROESTLER
> Sent: 23 November 2010 10:38
> To: igo...@yahoo.com
> Cc: caml...@inria.fr
> Subject: Re: [Caml-list] Re: Is OCaml fast?
>
> On Tue, 23 Nov 2010 02:03:48 +0000, Isaac Gouy wrote:
> >

> > > C version : 12.11 secs
> > > OCaml version : 47.22 secs
> > > OCaml version with GC parameters tuned ("interesting alternative"
> > section) : 12.67 secs
> >
> > And of course you know because that GC tuned OCaml program is shown
> > on the
> > benchmarks game website ;-)

> > http://shootout.alioth.debian.org/u32/program.php?test=binarytrees&
> > lang=ocaml&id=1
>
> Since you are here, please explain why C can use memory pools and vec
> tor instructions but tuning the GC of OCaml ― although it is part of
> the standard library ― is considered an "alternative".
>
> Best,
> C.
>

Isaac Gouy

unread,
Nov 23, 2010, 12:53:33 PM11/23/10
to caml...@inria.fr
Christophe TROESTLER <Christophe.Troestler+ocaml <at> umh.ac.be> writes:

>
> On Tue, 23 Nov 2010 02:03:48 +0000, Isaac Gouy wrote:
> >

> > > C version : 12.11 secs
> > > OCaml version : 47.22 secs
> > > OCaml version with GC parameters tuned ("interesting alternative"
> > section) : 12.67 secs
> >
> > And of course you know because that GC tuned OCaml program is shown
> > on the
> > benchmarks game website

> > http://shootout.alioth.debian.org/u32/program.php?test=binarytrees&
> > lang=ocaml&id=1
>
> Since you are here, please explain why C can use memory pools and vec
> tor instructions but tuning the GC of OCaml ― although it is part of
> the standard library ― is considered an “alternative”.


You seem to be suggesting that "tuning the GC" is considered "alternative" only
for OCaml programs.

You seem to be suggesting that "tuning the GC" is considered "alternative" for
every task.

Neither is true.

You seem to be suggesting some kind of equivalence between vector instructions
and "tuning the GC".
You haven't said why they should be considered equivalent.

Nor have you said why you think C should not be allowed to use memory pools.

Isaac Gouy

unread,
Nov 23, 2010, 1:00:12 PM11/23/10
to caml...@inria.fr
Is that book the source for the quotation 'Lies--damned lies--and statistics'?


Fabrice Le Fessant <fabrice.le_fessant <at> inria.fr> writes:
> Maybe you should read "Tainted Truth: The Manipulation of Fact In
> America" by Cynthia Crossen ?

Isaac Gouy

unread,
Nov 23, 2010, 1:05:10 PM11/23/10
to caml...@inria.fr
Jon Harrop <jonathandeanharrop <at> googlemail.com> writes:

>
> Note that the regex-dna solution for Haskell tweaks its GC parameters via
> the -H command-line parameter:


Note that there is no restriction on "tuning the GC" for regex-dna.

Note that there is no restriction on "tuning the GC" for any task except
binary-trees.

Isaac Gouy

unread,
Nov 23, 2010, 1:07:29 PM11/23/10
to caml...@inria.fr
Jon Harrop <jonathandeanharrop <at> googlemail.com> writes:

> Ketil Malde crafted
> a much better solution but noted:
>
> "This is considered cheating, since it is the easy and natural way to do
> it." -
> http://www.haskell.org/haskellwiki/Shootout/Knucleotide


Not even cheating - just an answer to a different question.

Török Edwin

unread,
Nov 23, 2010, 2:14:44 PM11/23/10
to Isaac Gouy, caml...@inria.fr
On Tue, 23 Nov 2010 18:03:10 +0000 (UTC)
Isaac Gouy <igo...@yahoo.com> wrote:

> Jon Harrop <jonathandeanharrop <at> googlemail.com> writes:
>
> >
> > Note that the regex-dna solution for Haskell tweaks its GC
> > parameters via the -H command-line parameter:
>
>
> Note that there is no restriction on "tuning the GC" for regex-dna.
>
> Note that there is no restriction on "tuning the GC" for any task
> except binary-trees.

Sounds good. Then Ocaml could still win if it performs well on the
other benchmarks.

Best regards,
--Edwin

Gerd Stolpmann

unread,
Nov 23, 2010, 2:25:05 PM11/23/10
to Isaac Gouy, caml...@inria.fr
Am Dienstag, den 23.11.2010, 17:53 +0000 schrieb Isaac Gouy:
> Christophe TROESTLER <Christophe.Troestler+ocaml <at> umh.ac.be> writes:
>
> >
> > On Tue, 23 Nov 2010 02:03:48 +0000, Isaac Gouy wrote:
> > >
> > > > C version : 12.11 secs
> > > > OCaml version : 47.22 secs
> > > > OCaml version with GC parameters tuned ("interesting alternative"
> > > section) : 12.67 secs
> > >
> > > And of course you know because that GC tuned OCaml program is shown
> > > on the
> > > benchmarks game website
> > > http://shootout.alioth.debian.org/u32/program.php?test=binarytrees&
> > > lang=ocaml&id=1
> >
> > Since you are here, please explain why C can use memory pools and vec
> > tor instructions but tuning the GC of OCaml ― although it is part of
> > the standard library ― is considered an “alternative”.
>
>
> You seem to be suggesting that "tuning the GC" is considered "alternative" only
> for OCaml programs.
>
> You seem to be suggesting that "tuning the GC" is considered "alternative" for
> every task.
>
> Neither is true.
>
> You seem to be suggesting some kind of equivalence between vector instructions
> and "tuning the GC".
> You haven't said why they should be considered equivalent.
>
> Nor have you said why you think C should not be allowed to use memory pools.

Quite easy: because you are comparing results that cannot be compared.
The reader of this benchmark (binary trees) might have the impression
that C is generally that fast - however, this would be no longer true if
these binary trees were used as library in a bigger program where using
memory pools is inappropriate, e.g. because the data managed by the
binary trees has an unpredictable lifetime.

I do not say that it is complete nonsense to do this comparison, but
only that it is more specific than a reader would assume. The innocent
reader expects a comparison of binary tree performance, not of methods
of managing memory (and this is it finally). The true name of this test
should be "manage_many_small_memory_cells_where_pools_are_allowed". (It
would be actually interesting to compare various versions of this test
with different memory management methods.)

Gerd

>
>
>
>
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>


--
------------------------------------------------------------
Gerd Stolpmann, Bad Nauheimer Str.3, 64289 Darmstadt,Germany
ge...@gerd-stolpmann.de http://www.gerd-stolpmann.de
Phone: +49-6151-153855 Fax: +49-6151-997714
------------------------------------------------------------

Mark Diekhans

unread,
Nov 23, 2010, 2:54:26 PM11/23/10
to Isaac Gouy, caml...@inria.fr

The source of this pharse is unknown, however Mark Twain is credited with
making it well known:

"There are three kinds of lies: lies, damned lies and statistics."

Isaac Gouy

unread,
Nov 23, 2010, 3:05:20 PM11/23/10
to caml...@inria.fr
Mark Diekhans <markd <at> kermodei.com> writes:

>
>
> The source of this pharse is unknown, however Mark Twain is credited with
> making it well known:
>
> "There are three kinds of lies: lies, damned lies and statistics."


http://www.york.ac.uk/depts/maths/histstat/lies.htm

Isaac Gouy

unread,
Nov 23, 2010, 3:18:11 PM11/23/10
to caml...@inria.fr
Török Edwin <edwintorok <at> gmail.com> writes:

-snip-

> Sounds good. Then Ocaml could still win if it performs well on the
> other benchmarks.


The main benchmarks game summary is median and quartiles so one measurement
doesn't have much influence.

There are several tasks which have worse performing OCaml programs than
binary-trees.

Isaac Gouy

unread,
Nov 23, 2010, 3:30:24 PM11/23/10
to caml...@inria.fr
Gerd Stolpmann <info <at> gerd-stolpmann.de> writes:

-snip-


> I do not say that it is complete nonsense to do this comparison, but
> only that it is more specific than a reader would assume.

A reader's wrong assumptions are their own responsibility:

http://shootou.alioth.debian.org/flawed-benchmarks.php


> The innocent reader expects a comparison of binary tree performance,
> not of methods of managing memory (and this is it finally).

Perhaps rather than "innocent reader" you mean careless reader who didn't bother
to read what the programs should do?

http://shootout.alioth.debian.org/u32q/benchmark.php?test=binarytrees&lang=all#about


> The true name of this test should be
> "manage_many_small_memory_cells_where_pools_are_allowed".

"binary-trees benchmark : Allocate and deallocate many many binary trees"

> (It would be actually interesting to compare various versions of this test
> with different memory management methods.)

So do that comparison and publish the results.

Gerd Stolpmann

unread,
Nov 23, 2010, 3:56:07 PM11/23/10
to Isaac Gouy, caml...@inria.fr
Am Dienstag, den 23.11.2010, 20:28 +0000 schrieb Isaac Gouy:

> > (It would be actually interesting to compare various versions of this test
> > with different memory management methods.)
>
> So do that comparison and publish the results.

Please don't tell me what I am supposed to do. I'm not a troll like
others.

Gerd


--
------------------------------------------------------------
Gerd Stolpmann, Bad Nauheimer Str.3, 64289 Darmstadt,Germany
ge...@gerd-stolpmann.de http://www.gerd-stolpmann.de
Phone: +49-6151-153855 Fax: +49-6151-997714
------------------------------------------------------------

_______________________________________________

Christophe TROESTLER

unread,
Nov 23, 2010, 4:14:32 PM11/23/10
to igo...@yahoo.com, caml...@inria.fr
On Tue, 23 Nov 2010 17:53:14 +0000, Isaac Gouy wrote:

>
> Christophe TROESTLER writes:
>
> > Since you are here, please explain why C can use memory pools and vec
> > tor instructions but tuning the GC of OCaml ― although it is part of
> > the standard library ― is considered an “alternative”.
>
> [...] You seem to be suggesting some kind of equivalence between vector

> instructions and "tuning the GC". You haven't said why they should
> be considered equivalent.

I did not say they are equivalent. It is a matter of what is allowed
(or not) and for what reason. The question is why is C allowed to use
an external library for managing its memory but — for this specific
benchmark which is precisely about managing memory — OCaml is not
authorized to make use of its very own library!

On Tue, 23 Nov 2010 18:03:10 +0000, Isaac Gouy wrote:
>
> Note that there is no restriction on "tuning the GC" for regex-dna.
>
> Note that there is no restriction on "tuning the GC" for any task except
> binary-trees.

Note that you did not answer my question.

Isaac Gouy

unread,
Nov 23, 2010, 4:32:37 PM11/23/10
to caml...@inria.fr
Gerd Stolpmann <info <at> gerd-stolpmann.de> writes:

> > > (It would be actually interesting to compare various versions of this test
> > > with different memory management methods.)
> >
> > So do that comparison and publish the results.
>
> Please don't tell me what I am supposed to do. I'm not a troll like
> others.


The tone of my comment was not as harsh as you heard.

Jon Harrop

unread,
Nov 23, 2010, 4:35:04 PM11/23/10
to caml...@inria.fr
Yes, an answer to a better question.

> -----Original Message-----
> From: caml-lis...@yquem.inria.fr [mailto:caml-list-

Isaac Gouy

unread,
Nov 23, 2010, 4:55:44 PM11/23/10
to caml...@inria.fr
Christophe TROESTLER <Christophe.Troestler+ocaml <at> umh.ac.be> writes:

-snip-


> The question is why is C allowed to use
> an external library for managing its memory

I asked why you think C should not be allowed to use memory pools - you haven't
tried to answer that question.

If you think that C should not be allowed to use memory pools because they are
provided in an external library then I disagree - I think it's fine to show use
of popular external libraries.

Of course, a C program that just uses malloc is also shown.


> but — for this specific
> benchmark which is precisely about managing memory — OCaml is not
> authorized to make use of its very own library!

OCaml is authorized to make use of its very own library.

Just like all the other programming language implementations, for binary-trees,
OCaml is not authorized to tune the GC.

oli...@first.in-berlin.de

unread,
Nov 23, 2010, 5:56:22 PM11/23/10
to caml...@yquem.inria.fr
On Mon, Nov 22, 2010 at 05:55:02AM -0800, Dario Teixeira wrote:
> Hi,
>
> > I am on the fence about whether to learn OCaml or not, and while
> > reading an article called "Why OCaml"
> > (http://www.cs.ubc.ca/~murphyk/Software/Ocaml/why_ocaml.html),
> > I saw that OCaml was praised for the speed of the executables it
> > generates - and was referred to, speed-wise, as "second to none",
> > except C and C++.
>
> Yes, Ocaml is fast (more on that later), but you should consider speed
> as just the icing in the cake.
[...]

But if it were not performant enough, I (and many others too, I think)
would consider it being a nice toy language.

AFAIK in the past, functional langauges were not adapted, because they were
very unperformant - at least this is one reason.
Another reason might be, that the available functional languages in the past
were overloaded with parenthess ;)

Ciao,
Oliver

oli...@first.in-berlin.de

unread,
Nov 23, 2010, 6:27:53 PM11/23/10
to caml...@inria.fr
On Tue, Nov 23, 2010 at 02:01:33AM +0000, Isaac Gouy wrote:
> David Rajchenbach-Teller <David.Teller <at> univ-orleans.fr> writes:
>
>
> > I can confirm that old code-snippets were removed (and that both faster
> solutions and environment
> > variable tweaks were rejected).
>
>
> Even back in 2001, Doug Bagley had noted all the things that were
> wrong with the tasks on his "The Great Computer Language Shootout".


And what was wrong in his eyes?


>
> During autumn 2004, on Brent Fulgham's website, new tasks were added
> and old Doug Bagley tasks removed.
>
> By 2005 only 2 tasks remained from that old Doug Bagley website.
>
> By August 2008 none of those old Doug Bagley tasks were measured for
> the current benchmarks game.

So, now the comparisions are perfect?
What problems were removed?


Ciao,
Oliver

Jon Harrop

unread,
Nov 23, 2010, 6:54:38 PM11/23/10
to oli...@first.in-berlin.de, caml...@inria.fr
Oliver wrote:
> AFAIK in the past, functional langauges were not adapted, because they
> were
> very unperformant - at least this is one reason.
> Another reason might be, that the available functional languages in the
> past
> were overloaded with parenthess ;)

That was also true of early ML implementations. When I was first taught ML at university we used the Cambridge ML interpreter and you actually had to sit there and wait for it to solve the 8-queens problem. At the time, I thought ML was a complete joke and could see no use for it outside its very specific domain of theorem proving and actually really resented being taught it on a general CS course. I know better now though. ;-)

Later language implementations inherited many of these inefficiencies though. Many of the things that can make OCaml and Java slow were inherited from Lisp. They are, in effect, designs that still bear the burden of dynamic typing despite being statically typed.

Cheers,
Jon.

Isaac Gouy

unread,
Nov 23, 2010, 7:12:30 PM11/23/10
to caml...@inria.fr
Richard Jones <rich <at> annexia.org> writes:

> > Note that there is no restriction on "tuning the GC" for regex-dna.
> >
> > Note that there is no restriction on "tuning the GC" for any task except
> > binary-trees.
>

> Don't you think this is a pretty ludicrous restriction?
>
> Tuning the GC / adjusting the size of pools or mallocs is essential
> for optimizing for modern processors. I bet the C programs are doing
> this, except that it won't obviously be called "tuning the GC"
> although it amounts to precisely the same thing.


What you say would be interesting if you'd taken the trouble to look at the C
programs.

Isaac Gouy

unread,
Nov 23, 2010, 7:23:33 PM11/23/10
to caml...@inria.fr
<oliver <at> first.in-berlin.de> writes:

> > Even back in 2001, Doug Bagley had noted all the things that were
> > wrong with the tasks on his "The Great Computer Language Shootout".
>
> And what was wrong in his eyes?

Find out for yourself:

http://web.archive.org/web/20010617014807/www.bagley.org/~doug/shootout/


> So, now the comparisions are perfect?

Has anyone said so?


> What problems were removed?

All of them.

Erik de Castro Lopo

unread,
Nov 23, 2010, 8:24:28 PM11/23/10
to caml...@inria.fr
Jon Harrop wrote:

> Many of the things that can make OCaml and Java slow were inherited
> from Lisp. They are, in effect, designs that still bear the burden
> of dynamic typing despite being statically typed.

I'm curious, what in your opinion are those things?

Erik
--
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

Eray Ozkural

unread,
Nov 23, 2010, 8:36:35 PM11/23/10
to Isaac Gouy, caml...@inria.fr
Hello,

I think that this benchmark is lacking in the algorithms department. Where
is a dynamic programming problem? A graph algorithm? Anything with
non-trivial time/space complexity? Anything a little more complex than
matrix product?

Also, it's not uncommon to disallow low-level optimizations such as writing
memory allocators and async file access when comparing implementations of an
algorithm, but such restrictions should be carried out uniformly. In such a
benchmark I would expect each entry to stick to their guns, i.e. use only
the standard libraries and programming styles for instance. Linking in
foreign libraries must most definitely be disallowed. So, if in Java, it's
necessary to call the garbage collector explicitly from time to time, and we
had to do that for a long time, so be it. Or again, if in Java, performance
will suffer unless you only use arrays of integral types, the implementer
may wish to implement as much as is possible with arrays, though I wonder if
it is not better to choose the most natural implementation style for the
particular language. In the case of Java, the claim was that object-oriented
was some kind of a programming-aid that can replace talented programmers :)
It's unfortunate of course that some kinds of optimizations always have to
be made by hand, for instance in functional languages many compilers do not
have deforestation.

Otherwise, of course, any implementation may include a compiler for the
fastest language and present a program in that language, which is not the
objective.

An alternative objective could be to compare the shortest and most
comprehensible, if possible line-to-line compatible implementation of a
given pseudocode in different languages. That would be extremely informative
for serious algorithm researchers! If a computer scientist isn't sure of the
performance of the primitives, he cannot make sure his implementation will
comply with the time-complexity of the given algorithm.

Best Regards,

On Wed, Nov 24, 2010 at 2:23 AM, Isaac Gouy <igo...@yahoo.com> wrote:

> <oliver <at> first.in-berlin.de> writes:
>
> > > Even back in 2001, Doug Bagley had noted all the things that were
> > > wrong with the tasks on his "The Great Computer Language Shootout".
> >
> > And what was wrong in his eyes?
>
> Find out for yourself:
>

> http://web.archive.org/web/20010617014807/www.bagley.org/~doug/shootout/<http://web.archive.org/web/20010617014807/www.bagley.org/%7Edoug/shootout/>


>
>
> > So, now the comparisions are perfect?
>
> Has anyone said so?
>
>
> > What problems were removed?
>
> All of them.
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

--
Eray Ozkural, PhD candidate. Comp. Sci. Dept., Bilkent University, Ankara
http://groups.yahoo.com/group/ai-philosophy
http://myspace.com/arizanesil http://myspace.com/malfunct

Jeff Meister

unread,
Nov 23, 2010, 11:39:29 PM11/23/10
to Isaac Gouy, caml...@inria.fr
Everyone in this thread is capable of reading your site and has
probably already done so. We know what your rules are for
binary-trees; repeating them does not help. Richard's objection, which
you dismissed out of hand, was that your no-GC-tuning rule is silly in
the light of actual uses of garbage collected programming languages on
modern processors. It makes your results unrealistic, and an
unrealistic benchmark is misleading, or at best merely useless. You
are free to tersely reject our constructive criticism, but the only
meaningful consequence will be that OCaml users consider the shootout
untrustworthy and completely ignore its results... what good are the
"language comparisons" your project makes if the communities behind
those languages don't support your benchmarking methods?

On Tue, Nov 23, 2010 at 6:13 PM, Isaac Gouy <igo...@yahoo.com> wrote:
> Eray Ozkural <examachine <at> gmail.com> writes:
>
>
>> Hello, I think that this benchmark is lacking ...
>
> http://shootout.alioth.debian.org/help.php#why
>
>
> Please make the kind of comparison you think should be done and publish it.

Andrew

unread,
Nov 24, 2010, 1:22:30 AM11/24/10
to Jeff Meister, Isaac Gouy, caml...@inria.fr
> ::Jeff Meister

>Everyone in this thread is capable of reading your site and has
>probably already done so. We know what your rules are for
>binary-trees; repeating them does not help. Richard's objection, which
>you dismissed out of hand, was that your no-GC-tuning rule is silly in
>the light of actual uses of garbage collected programming languages on
>modern processors. It makes your results unrealistic, and an
>unrealistic benchmark is misleading, or at best merely useless. You
>are free to tersely reject our constructive criticism, but the only
>meaningful consequence will be that OCaml users consider the shootout
>untrustworthy and completely ignore its results... what good are the
>"language comparisons" your project makes if the communities behind
>those languages don't support your benchmarking methods?

+1. Seriously, Isaac, try to calm down, everything is fine. You might want
to read what others write, I have the feeling that many people were making
valid points, whereas you have mostly been turning down any objection by
pointing people to the same webpage and again.

Andrew.

Isaac Gouy

unread,
Nov 24, 2010, 1:50:49 AM11/24/10
to caml...@inria.fr
Jeff Meister <nanaki <at> gmail.com> writes:


> We know what your rules are for
> binary-trees; repeating them does not help.

When Christophe TROESTLER wrongly states - "OCaml is not authorized to make use
of its very own library!" - he shows that those rules are not known.


> Richard's objection, which you dismissed out of hand, was that your
> no-GC-tuning rule is silly in the light of actual uses of garbage collected
> programming languages on modern processors.

When said Richard opines about programs he apparently hasn't bothered to read, I
take that as a sign his opinions might not be based on anything solid.


> It makes your results unrealistic, and an
> unrealistic benchmark is misleading, or at best merely useless.

You should think that benchmarks (not just these) are unrealistic - your
application is the ultimate benchmark.

Useless? Wouldn't that depend on the objectives? The post you replied to
linked-to a 3 line statement of objectives - did you read it?


> You are free to tersely reject our constructive criticism

Do you really think saying something is "ludicrous" or "silly" is constructive
criticism? :-)

David Rajchenbach-Teller

unread,
Nov 24, 2010, 1:54:32 AM11/24/10
to evil sloot, caml...@inria.fr
Submitted this a few years ago (just with streams, not Lwt), and the solution was rejected.


On Nov 24, 2010, at 12:21 AM, evil sloot wrote:

> Hi,
>
> Regarding alternative solutions to the shootout problems:
> The thread ring problem can be easily implemented using Lwt
> yielding (pun intended ;)) performance
> comparable to the haskell solution
>
> have fun,
>
> Romain.

David Rajchenbach-Teller

unread,
Nov 24, 2010, 1:55:55 AM11/24/10
to Eray Ozkural, Isaac Gouy, caml...@inria.fr
Maybe the solution is to get one of our numbers among the maintainers of the shootout. This would guarantee, if not objectivity, then at least informed choices wrt OCaml.