Clojure 1.2 and the Computer Language Benchmarks Game

[Isaac Gouy]

Now Clojure 1.2 has been released, Clojure programs will be included
in the Computer Language Benchmarks Game.

If you'd like to contribute Clojure programs, please follow the step-
by-step

http://shootout.alioth.debian.org/help.php#contribute

[ataggart]

It's never been clear to me exactly what the code is supposed to be
do. For example, the "spec" for the binary-tree test is so wholly
lacking in any details that I'm left to infer that one is supposed to
copy an implementation used previously, though without any indication
as to which is the canonical version. Do I really need to perform the
itemCheck math ops in the binary-tree test which is ostensibly about
allocating/deallocating memory? Who knows?

[Robert McIntyre]

I hear you --- I got excited about this too, and implemented the fannuchredux
algorithm, only to be thwarted by an undocumented "checksum" each
program is also
supposed to calculate. This checksum depends heavily on the exact
order in which
a set of permutations are traversed. And of course, they aren't
traversed in lexical order,
but whatever order the original implementation used.

:(

--Robert McIntyre

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

[Isaac Gouy]

On Aug 23, 7:07 pm, ataggart <alex.tagg...@gmail.com> wrote:
> It's never been clear to me exactly what the code is supposed to be
> do. For example, the "spec" for the binary-tree test is so wholly
> lacking in any details that I'm left to infer that one is supposed to
> copy an implementation used previously, though without any indication
> as to which is the canonical version. Do I really need to perform the
> itemCheck math ops in the binary-tree test which is ostensibly about
> allocating/deallocating memory?  Who knows?

Some people complain - underspecified - and some people complain -
overspecified - and some people just contribute programs.

Some people complain - forced to write code that isn't idiomatic - as
if there was a canonical version. (Did you find any programs that
didn't perform itemCheck?)

In Clojure does one integer addition and one integer subtraction per
node take a significant amount of time? Who knows? (I guess you could
measure with/without.)

[Isaac Gouy]

On Aug 23, 7:35 pm, Robert McIntyre <r...@mit.edu> wrote:
> I hear you --- I got excited about this too, and implemented the fannuchredux
> algorithm, only to be thwarted by an undocumented "checksum" each
> program is also
> supposed to calculate.  This checksum depends heavily on the exact
> order in which
> a set of permutations are traversed. And of course, they aren't
> traversed in lexical order,
> but whatever order the original implementation used.

By design - "This checksum depends heavily on the exact order in which
a set of permutations are traversed." - people kept finding ways to
not generate some of the permutations or used faster algorithms to
generate the permutations - so fannkuch-redux now allows little
flexibility.

[Stuart Halloway]

Clojure 1.3's performance improvements will significantly impact perf on some of the benchmarks. If you are trying these out, please try them on both 1.2 and 1.3.

Also: the benchmarks are totally a numbers game: throw idioms and readability out the window. Clojure 1.3 should be able to match Java performance if you basically write Java-in-Clojure. On Clojure 1.2 you will have to do stranger things to get there.

Stu

[Isaac Gouy]

On Aug 24, 6:44 am, Stuart Halloway <stuart.hallo...@gmail.com> wrote:
> Clojure 1.3's performance improvements will significantly impact perf on some of the benchmarks. If you are trying these out, please try them on both 1.2 and 1.3.

Has Clojure 1.3 been released?

> Also: the benchmarks are totally a numbers game: throw idioms and readability out the window. Clojure 1.3 should be able to match Java performance if you basically write Java-in-Clojure.  On Clojure 1.2 you will have to do stranger things to get there.

If you choose to throw idioms and readability out the window then
don't be surprised at the comments that will be made about Clojure.

If you have to "do stranger things to get there" with Clojure 1.2 then
doesn't that simply suggest Clojure 1.2 performance doesn't match Java
performance?

[David Nolen]

On Tue, Aug 24, 2010 at 10:30 AM, Isaac Gouy <igo...@yahoo.com> wrote:

Has Clojure 1.3 been released?

Nope.

 

If you choose to throw idioms and readability out the window then

don't be surprised at the comments that will be made about Clojure.

Clojure doesn't encourage mutable state. Most of the benchmarks do. Mutable Clojure will probably always look fairly unidiomatic.

 

If you have to "do stranger things to get there" with Clojure 1.2 then
doesn't that simply suggest Clojure 1.2 performance doesn't match Java
performance?

No.

David 

[Stuart Halloway]

> On Aug 24, 6:44 am, Stuart Halloway <stuart.hallo...@gmail.com> wrote:
>> Clojure 1.3's performance improvements will significantly impact perf on some of the benchmarks. If you are trying these out, please try them on both 1.2 and 1.3.
>
>
> Has Clojure 1.3 been released?
>

No, but since the num/prim/equiv work specifically targets performance, we want to collect people's experiences comparing 1.2 and 1.3. This is totally separate from the benchmark submission process and for our own information.

>> Also: the benchmarks are totally a numbers game: throw idioms and readability out the window. Clojure 1.3 should be able to match Java performance if you basically write Java-in-Clojure. On Clojure 1.2 you will have to do stranger things to get there.
>
> If you choose to throw idioms and readability out the window then
> don't be surprised at the comments that will be made about Clojure.

Let me reduce the stridency of my previous statement: "throwing things out the window" is too strong. Benchmarks are a numbers contest, not a beauty contest. But, there's no saying the resulting programs will be ugly (or even non-idiomatic). Try things. Measure. It is simply the case that some idiomatic code (i.e. numeric code with no hints in the body) is faster in master/1.3 than in 1.2, and that some benchmark-useful things (fns taking/returning primitives) are available only post 1.2.

> If you have to "do stranger things to get there" with Clojure 1.2 then
> doesn't that simply suggest Clojure 1.2 performance doesn't match Java
> performance?

If I had wanted to suggest that, I would have said "you can't get there". With Clojure 1.2, you can get fast programs easily, or screaming-fast programs with effort. Some of the work in Clojure 1.3 reduces that effort.

Stu

[Isaac Gouy]

On Aug 24, 8:48 am, Stuart Halloway <stuart.hallo...@gmail.com> wrote:
> > On Aug 24, 6:44 am, Stuart Halloway <stuart.hallo...@gmail.com> wrote:
> >> Clojure 1.3's performance improvements will significantly impact perf on some of the benchmarks. If you are trying these out, please try them on both 1.2 and 1.3.
>
> > Has Clojure 1.3 been released?
>
> No, but since the num/prim/equiv work specifically targets performance, we want to collect people's experiences comparing 1.2 and 1.3. This is totally separate from the benchmark submission process and for our own information.
>
> >> Also: the benchmarks are totally a numbers game: throw idioms and readability out the window. Clojure 1.3 should be able to match Java performance if you basically write Java-in-Clojure.  On Clojure 1.2 you will have to do stranger things to get there.
>
> > If you choose to throw idioms and readability out the window then
> > don't be surprised at the comments that will be made about Clojure.
>
> Let me reduce the stridency of my previous statement: "throwing things out the window" is too strong. Benchmarks are a numbers contest, not a beauty contest. But, there's no saying the resulting programs will be ugly (or even non-idiomatic). Try things. Measure. It is simply the case that some idiomatic code (i.e. numeric code with no hints in the body) is faster in master/1.3 than in 1.2, and that some benchmark-useful things (fns taking/returning primitives) are available only post 1.2.

Well when Clojure 1.3 is released...

The phrase "idiomatic code" often seems to be used to mean - code
written in a natural way for that language and as if performance
doesn't matter - whereas I seem to have the strange notion that both
code written as if performance matters and code written as if
performance doesn't matter can be "idiomatic code".


http://shootout.alioth.debian.org/u32q/program.php?test=spectralnorm&lang=ghc&id=2


http://shootout.alioth.debian.org/u32q/program.php?test=spectralnorm&lang=ghc&id=4

[David Nolen]

On Tue, Aug 24, 2010 at 12:33 PM, Isaac Gouy <igo...@yahoo.com> wrote:

The phrase "idiomatic code" often seems to be used to mean - code

written in a natural way for that language and as if performance
doesn't matter - whereas I seem to have the strange notion that both
code written as if performance matters and code written as if
performance doesn't matter can be  "idiomatic code".

Idiomatic Clojure is plenty fast, faster than most of the other dynamic langs. But Clojure can get identical to pure Java perf and that may or may not look idiomatic depending on what you are doing.

David

[ataggart]

It would have been more useful to answer the question (particularly
with regards to a canonical implementation) than getting all passive-
aggressive.

[Nicolas Oury]

>>> Clojure 1.3's performance improvements will significantly impact perf on some of the benchmarks. If you are trying these out, please try them on both 1.2 and 1.3.
>>
>>
>> Has Clojure 1.3 been released?
>>
>
> No, but since the num/prim/equiv work specifically targets performance, we want to collect people's experiences comparing 1.2 and 1.3. This is totally separate from the benchmark submission process and for our own information.
>

I have a project that is development and will still be for some time.
I don't mind if it does not work perfectly before 1.3 and I will be
very happy to have it very fast when 1.3 is ready.

I am quite happy to make a branch to check improvement vs 1.2, but I
wouldn't know how to change my code to benefit from the features of
1.3.

Is there some links about that?
Are the prim/equiv work in master yet? And the :static defn with
primitives arg? And the protocols with primitive?
Are there other improvements?

I have a lot of primitives that are spread in a big part of the code,
so I think I would benefit from that?

Best,

Nicolas.

[Nicolas Oury]

On Tue, Aug 24, 2010 at 5:33 PM, Isaac Gouy <igo...@yahoo.com> wrote:
>
> Well when Clojure 1.3 is released...
>
> The phrase "idiomatic code" often seems to be used to mean - code
> written in a natural way for that language and as if performance
> doesn't matter - whereas I seem to have the strange notion that both
> code written as if performance matters and code written as if
> performance doesn't matter can be  "idiomatic code".

For most of the code, being slower (in a reasonable limit) is not
important if you are correct and easily maintable, and reusable.

[Isaac Gouy]

When it doesn't matter it doesn't matter but when it matters it
matters.

[Isaac Gouy]

On Aug 24, 9:50 am, ataggart <alex.tagg...@gmail.com> wrote:
> It would have been more useful to answer the question (particularly
> with regards to a canonical implementation) than getting all passive-
> aggressive.

Did you find any programs that didn't perform itemCheck?

[ataggart]

Thanks for focusing solely on one example, and still not providing any
useful, specific information.

There may be a number of possible implementations for a given design
criterion. The binary-tree "memory allocation/deallocation" test (for
example) includes not only that, but also math ops, in a particular
fashion, a particular style of looping, with particular string
concatenation, and particular places for printing to stdout, etc.
From what I can infer, the criterion being tested is how well can
clojure code perform when it's written just like the imperative
version in, say, C++. I had been assuming this was a serious
comparison various languages performance in achieving a particular
design goal.

Time to move on to something productive.

[Isaac Gouy]

On Aug 24, 7:48 pm, ataggart <alex.tagg...@gmail.com> wrote:
> Thanks for focusing solely on one example, and still not providing any
> useful, specific information.

You asked - "Do I really need to perform the itemCheck math ops in
the binary-tree test" - and if you can't see the answer from simply
looking at the other programs and the program output, then you need
more hand-holding than I'm going to provide.

Will it really make a big difference when I add the indefinite article
to what it already says in the description?

"Each program should
...
- walk the tree nodes, checksum node items (and maybe deallocate the
node)"

> There may be a number of possible implementations for a given design
> criterion. The binary-tree "memory allocation/deallocation" test (for
> example) includes not only that, but also math ops, in a particular
> fashion, a particular style of looping, with particular string
> concatenation, and particular places for printing to stdout, etc.

Yes.

> From what I can infer, the criterion being tested is how well can
> clojure code perform when it's written just like the imperative
> version in, say, C++.

Nonsense.

You could as easily say - the criterion being tested is how well can
clojure code perform when it's written just like the functional
version in, say Clean or Haskell?

>  I had been assuming this was a serious comparison various languages
> performance in achieving a particular design goal.

After so many people have contributed programs for binary-trees in so
many languages, I had been assuming it was kind-of easy.

[John Fingerhut]

I will try submitting one or a few of my benchmark programs created 1 year ago.

For anyone that wants to look at some timing results and/or my source code used to achieve them before then, they are available on github here:

http://github.com/jafingerhut/clojure-benchmarks

I just pushed a few changes required for Clojure 1.2 about 10 minutes ago.  There aren't many changes to the results from Clojure 1.1 alpha I tested against a year ago, except for the following -- note that besides updating my Clojure version since last time, I have also upgraded the version of OS X on my Mac, and the version of Java from Apple.  I have not attempted to test these changes independently to see which one or ones caused the change in run time:

k-nucleotide benchmark completed in about 2/3 the time of earlier runs (good)
mandelbrot also runs in about 2/3 the time of before
n-body takes about 5x longer than before.  I do not know why.

As always, improvements to my versions are welcome.  I am pretty sure some people have posted programs they have worked on for these problems during the most recent 1 year to the group, but I'm sorry to say I haven't kept up with all of them to see if they are faster than my fastest versions.

Andy

[Nicolas Oury]

You can probably boost n-body on 1.2 by replacing arrays with deftypes.

(definterface BodyIsh
(^double getMass [])
(setMass [^double x])
(^double getPosX [])
.....)

(deftype Body [^double ^{:unsynchronized-mutable true} mass ^double
^{:unsynchronized-mutable true} posX .....]
BodyIsh
(getMass [x] mass)
(setMass [x m] (set! mass m)
.....)

And then rewriting the program using Body (and a type annotation on
bodies, of course)
I used interfaces and not protocols here because I am not sure 1.2
supports protocols with primitives.

Best,

Nicolas.

[Isaac Gouy]

On Aug 25, 6:17 am, John Fingerhut <andy.finger...@gmail.com> wrote:
> I will try submitting one or a few of my benchmark programs created 1 year
> ago.
>
> For anyone that wants to look at some timing results and/or my source code
> used to achieve them before then, they are available on github here:
>
> http://github.com/jafingerhut/clojure-benchmarks
>
> I just pushed a few changes required for Clojure 1.2 about 10 minutes ago.
> There aren't many changes to the results from Clojure 1.1 alpha I tested
> against a year ago, except for the following -- note that besides updating
> my Clojure version since last time, I have also upgraded the version of OS X
> on my Mac, and the version of Java from Apple.  I have not attempted to test
> these changes independently to see which one or ones caused the change in
> run time:
>
> k-nucleotide benchmark completed in about 2/3 the time of earlier runs
> (good)
> mandelbrot also runs in about 2/3 the time of before
> n-body takes about 5x longer than before.  I do not know why.
>
> As always, improvements to my versions are welcome.  I am pretty sure some
> people have posted programs they have worked on for these problems during
> the most recent 1 year to the group, but I'm sorry to say I haven't kept up
> with all of them to see if they are faster than my fastest versions.
>
> Andy

1) The command line requested for these first programs doesn't AOT
compile so the measured time includes compiling the program.

Perhaps AOT compilation is an usual way to use Clojure - or perhaps it
would be better to show programs that used AOT compilation?

http://groups.google.com/group/clojure/browse_thread/thread/e9902e30afbb805e


2) mandelbrot - There seems to be some problem with the mandelbrot
program - the program doesn't seem to exit for perhaps a minute after
the output file is created - see the difference between "CPU secs" and
"Elapsed secs".

http://shootout.alioth.debian.org/u32q/program.php?test=mandelbrot&lang=clojure&id=1


3) regexdna

http://shootout.alioth.debian.org/u32q/program.php?test=regexdna&lang=clojure&id=1

[Meikel Brandmeyer]

Hi,

On 26 Aug., 05:37, Isaac Gouy <igo...@yahoo.com> wrote:

> 1) The command line requested for these first programs doesn't AOT
> compile so the measured time includes compiling the program.

Which makes the comparison of languages with this benchmark even more
uninteresting.

> Perhaps AOT compilation is an usual way to use Clojure - or perhaps it
> would be better to show programs that used AOT compilation?

If startup time matters AOT compilation is a valid step to take. AOT
is not necessary but helps in such a case.

> 2) mandelbrot - There seems to be some problem with the mandelbrot
> program - the program doesn't seem to exit for perhaps a minute after
> the output file is created - see the difference between "CPU secs" and
> "Elapsed secs".

(shutdown-agents)

Sincerely
Meikel

[Isaac Gouy]

On Aug 25, 10:31 pm, Meikel Brandmeyer <m...@kotka.de> wrote:
> Hi,
>
> On 26 Aug., 05:37, Isaac Gouy <igo...@yahoo.com> wrote:
>
> > 1) The command line requested for these first programs doesn't AOT
> > compile so the measured time includes compiling the program.
>
> Which makes the comparison of languages with this benchmark even more
> uninteresting.

Have you actually measured the time difference?

[Meikel Brandmeyer]

Hi,

On 26 Aug., 07:58, Isaac Gouy <igo...@yahoo.com> wrote:

> Have you actually measured the time difference?

Compare the mandelbrot numbers for Haskell, Java and Scala. The ranges
are (0.07s 0.86s 13s), (0.19s 0.86s 12s), (0.22s 0.97s 15s). So Java
and Scala are not slower than Haskell, but the low iteration numbers
say something different. So you don't measure what you claim you
measure. And the fact that things vanish asymptotically doesn't fix a
broken methodology.

Sincerely
Meikel

[Isaac Gouy]

On Aug 26, 12:26 am, Meikel Brandmeyer <m...@kotka.de> wrote:
> Hi,
>
> On 26 Aug., 07:58, Isaac Gouy <igo...@yahoo.com> wrote:
>
> > Have you actually measured the time difference?

If you have measured the time difference with/without AOT compilation
then apparently you don't wish to share those measurements. Oh well.


Without Clojure AOT compilation

http://shootout.alioth.debian.org/u32q/program.php?test=mandelbrot&lang=clojure&id=1

With AOT compilation

http://shootout.alioth.debian.org/u32q/program.php?test=mandelbrot&lang=clojure&id=2

> Compare the mandelbrot numbers for Haskell, Java and Scala. The ranges
> are (0.07s 0.86s 13s), (0.19s 0.86s 12s), (0.22s 0.97s 15s). So Java
> and Scala are not slower than Haskell, but the low iteration numbers
> say something different. So you don't measure what you claim you
> measure. And the fact that things vanish asymptotically doesn't fix a
> broken methodology.

If I didn't measure what I claimed to measure then where did you get
those numbers from?


>
> Sincerely
> Meikel

[John Fingerhut]

I have now submitted small modifications that permit AOT compilation.  The compile time was small -- on the order of 1 to 2 sec of the total CPU time, which is often a small percentage of the long runs that are reported on the shootout web site.

But of course it is better if it is not included in the reported times, which it soon will not be.

Andy

Sincerely
Meikel

[Isaac Gouy]

On Aug 26, 8:37 am, John Fingerhut <andy.finger...@gmail.com> wrote:
> I have now submitted small modifications that permit AOT compilation.  The
> compile time was small -- on the order of 1 to 2 sec of the total CPU time,
> which is often a small percentage of the long runs that are reported on the
> shootout web site.
>
> But of course it is better if it is not included in the reported times,
> which it soon will not be.

AOT compiled

http://shootout.alioth.debian.org/u32q/program.php?test=mandelbrot&lang=clojure&id=2

http://shootout.alioth.debian.org/u32q/program.php?test=regexdna&lang=clojure&id=2

http://shootout.alioth.debian.org/u32q/program.php?test=knucleotide&lang=clojure&id=1

>
> Andy
>
> On Thu, Aug 26, 2010 at 12:26 AM, Meikel Brandmeyer <m...@kotka.de> wrote:
> > Hi,
>
> > On 26 Aug., 07:58, Isaac Gouy <igo...@yahoo.com> wrote:
>
> > > Have you actually measured the time difference?
>
> > Compare the mandelbrot numbers for Haskell, Java and Scala. The ranges
> > are (0.07s 0.86s 13s), (0.19s 0.86s 12s), (0.22s 0.97s 15s). So Java
> > and Scala are not slower than Haskell, but the low iteration numbers
> > say something different. So you don't measure what you claim you
> > measure. And the fact that things vanish asymptotically doesn't fix a
> > broken methodology.
>
> > Sincerely
> > Meikel
>
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Clojure" group.
> > To post to this group, send email to clo...@googlegroups.com
> > Note that posts from new members are moderated - please be patient with
> > your first post.
> > To unsubscribe from this group, send email to

> > clojure+u...@googlegroups.com<clojure%2Bunsu...@googlegroups.com>

[John Fingerhut]

Thanks to many people on this list in Aug 2009 who helped improve my code, to Johannes Friestad for writing a nice fast Clojure program using deftype for the n-body problem, to Isaac Gouy for setting up the shootout web site to accept Clojure submissions, and to my having more time than good sense to work on this, there are now Clojure programs for 5 of the 10 shootout web site benchmark problems.  You can see a brief summary of results comparing run time, memory, and code size against "Java 6 -server" here:

http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=clojure&lang2=java

Note: All Clojure programs are AOT compiled before any time measurements are made when running the program.  The programs are short enough that I believe the longest one was around 1.5 or 2 seconds on my MacBook Pro, but that time isn't included on the web site results.

Are there improvements that could be made to the Clojure programs to make them faster?  Most likely so.  I doubt very much these are the fastest ones possible.  I have not written any Java code for these solutions.  I am sure that when push comes to shove in improving the run time of parts of a real-world program that were performance-critical, most programmers would optimize using Java or even C/C++.  The goal here was to see what could be done without using those techniques.

If people want to beat each other up using the shootout web site's results, I would probably recognize in them some of the "my language is best for every purpose" attitude I once had as a teenager.  Now I just see it as one of several ways to evaluate a programming language, and often not the most important one.

Thanks,

Andy

[Isaac Gouy]

On Sep 1, 9:46 pm, John Fingerhut <andy.finger...@gmail.com> wrote:
> Thanks to many people on this list in Aug 2009 who helped improve my code,
> to Johannes Friestad for writing a nice fast Clojure program using deftype
> for the n-body problem, to Isaac Gouy for setting up the shootout web site
> to accept Clojure submissions, and to my having more time than good sense to
> work on this, there are now Clojure programs for 5 of the 10 shootout web
> site benchmark problems.  

iirc Clojure won't show up on the summary pages until there are 7
programs implemented.

Nontheless I've put Clojure on the home page.

> You can see a brief summary of results comparing
> run time, memory, and code size against "Java 6 -server" here:
>

> http://shootout.alioth.debian.org/u32/benchmark.php?test=all〈=clo...

> >http://shootout.alioth.debian.org/u32q/program.php?test=mandelbrot&la...
>
> >http://shootout.alioth.debian.org/u32q/program.php?test=regexdna&lang...
>
> >http://shootout.alioth.debian.org/u32q/program.php?test=knucleotide&l...

>
> > > Andy
>
> > > On Thu, Aug 26, 2010 at 12:26 AM, Meikel Brandmeyer <m...@kotka.de>
> > wrote:
> > > > Hi,
>
> > > > On 26 Aug., 07:58, Isaac Gouy <igo...@yahoo.com> wrote:
>
> > > > > Have you actually measured the time difference?
>
> > > > Compare the mandelbrot numbers for Haskell, Java and Scala. The ranges
> > > > are (0.07s 0.86s 13s), (0.19s 0.86s 12s), (0.22s 0.97s 15s). So Java
> > > > and Scala are not slower than Haskell, but the low iteration numbers
> > > > say something different. So you don't measure what you claim you
> > > > measure. And the fact that things vanish asymptotically doesn't fix a
> > > > broken methodology.
>
> > > > Sincerely
> > > > Meikel
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "Clojure" group.
> > > > To post to this group, send email to clo...@googlegroups.com
> > > > Note that posts from new members are moderated - please be patient with
> > > > your first post.
> > > > To unsubscribe from this group, send email to
> > > > clojure+u...@googlegroups.com<clojure%2Bunsu...@googlegroups.com>

> > <clojure%2Bunsu...@googlegroups.com<clojure%252Buns...@googlegroups.com>

[Sean Corfield]

On Wed, Sep 1, 2010 at 9:46 PM, John Fingerhut <andy.fi...@gmail.com> wrote:
> You can see a brief summary of results comparing
> run time, memory, and code size against "Java 6 -server" here:
> http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=clojure&lang2=java

Very interesting. Clojure is faster than several languages
(surprisingly so in some cases - at least for me) but seems to use
more code to express solutions - which really surprised me!
--
Sean A Corfield -- (904) 302-SEAN
Railo Technologies, Inc. -- http://getrailo.com/
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

[Isaac Gouy]

On Sep 2, 5:28 pm, Sean Corfield <seancorfi...@gmail.com> wrote:

> On Wed, Sep 1, 2010 at 9:46 PM, John Fingerhut <andy.finger...@gmail.com> wrote:
> > You can see a brief summary of results comparing
> > run time, memory, and code size against "Java 6 -server" here:

> >http://shootout.alioth.debian.org/u32/benchmark.php?test=all〈=clo...

>
> Very interesting. Clojure is faster than several languages
> (surprisingly so in some cases - at least for me) but seems to use
> more code to express solutions - which really surprised me!

Perhaps some of those Clojure programs include code that isn't
actually required, for example, code to display a usage message?

Perhaps some of those Clojure programs are written to use all the
cores on the quad-core machine and the programs you have compared them
with are only written to use one core?

Memory use was what I noticed.

[John Fingerhut]

I've got about 5 to 10 Clojure programs I've written for each of the 5 benchmark programs (many of which are only minor variations of each other, looking for ways to make it faster).  If you care to see any of the others, they are on github here:

http://github.com/jafingerhut/clojure-benchmarks

There are definitely shorter Clojure programs that can solve these problems.  In many cases (not necessarily all), those programs would also be slower, and/or use significantly more memory, than the ones on the web site.  Some of the problems involve keeping large strings (tens or hundreds of megabytes, when using one byte for each ASCII character) in memory all at once, and in those cases, it is important not to represent them with the wrong data structure, or else your program won't be able to store it all in memory (e.g. a Clojure lazy seq of characters uses over 50 bytes to represent each element of the seq, at least on a machine with 64-bit pointers).

Some of the Haskell submissions are quite long for what they do.  The k-nucleotide one, for example, implements a mutable hash table using features in Haskell that I had never seen before looking at that program.  Did they need to write all of that code to solve the problem?  No.  Did they choose to, in order to get a much faster program that would be more competitive in run time versus other languages.  Definitely.

Andy

--

[Isaac Gouy]

On Sep 2, 4:51 pm, Isaac Gouy <igo...@yahoo.com> wrote:
> On Sep 1, 9:46 pm, John Fingerhut <andy.finger...@gmail.com> wrote:
>
> > Thanks to many people on this list in Aug 2009 who helped improve my code,
> > to Johannes Friestad for writing a nice fast Clojure program using deftype
> > for the n-body problem, to Isaac Gouy for setting up the shootout web site
> > to accept Clojure submissions, and to my having more time than good sense to
> > work on this, there are now Clojure programs for 5 of the 10 shootout web
> > site benchmark problems.  
>
> iirc Clojure won't show up on the summary pages until there are 7
> programs implemented.
>
> Nontheless I've put Clojure on the home page.
>
> > You can see a brief summary of results comparing
> > run time, memory, and code size against "Java 6 -server" here:
>
> >http://shootout.alioth.debian.org/u32/benchmark.php?test=all〈=clo...

Clojure now shows up on the summary pages

http://shootout.alioth.debian.org/u32q/which-programming-languages-are-fastest.php?calc=chart&java=on&scala=on&sbcl=on&clojure=on&python3=on&jruby=on

[gary ng]

On Thu, Sep 2, 2010 at 6:07 PM, John Fingerhut <andy.fi...@gmail.com> wrote:
> Some of the Haskell submissions are quite long for what they do.  The
> k-nucleotide one, for example, implements a mutable hash table using
> features in Haskell that I had never seen before looking at that program.
> Did they need to write all of that code to solve the problem?  No.  Did they
> choose to, in order to get a much faster program that would be more
> competitive in run time versus other languages.  Definitely.

Is it the requirement for that particular program to handle hash table
(to test in place update) ? That seems to be the message I get from
the description of the program.

If that is the case, I doubt there is any choice for Haskell.

[Isaac Gouy]

On Sep 9, 6:06 pm, gary ng <garyng2...@gmail.com> wrote:

> On Thu, Sep 2, 2010 at 6:07 PM, John Fingerhut <andy.finger...@gmail.com> wrote:
> > Some of the Haskell submissions are quite long for what they do.  The
> > k-nucleotide one, for example, implements a mutable hash table using
> > features in Haskell that I had never seen before looking at that program.
> > Did they need to write all of that code to solve the problem?  No.  Did they
> > choose to, in order to get a much faster program that would be more
> > competitive in run time versus other languages.  Definitely.
>
> Is it the requirement for that particular program to handle hash table
> (to test in place update) ? That seems to be the message I get from
> the description of the program.

Yes.

> If that is the case, I doubt there is any choice for Haskell.

iirc the Haskell programs, and the Clean programs, and the Pascal
programs, and ... use translations of the simple hash table used by
the C programs.

If I ever knew, I don't recall why the Haskell program does not use
Data.HashTable

[gary ng]

On Thu, Sep 9, 2010 at 7:02 PM, Isaac Gouy <igo...@yahoo.com> wrote:
> iirc the Haskell programs, and the Clean programs, and the Pascal
> programs, and ... use translations of the simple hash table used by
> the C programs.
>
> If I ever knew, I don't recall why the Haskell program does not use
> Data.HashTable
>

Could be that it wasn't there at the time it was submitted. And
Data.HashTable also use some mutable array thing so it is still not
the idiomatic Haskell of everything is immutable.

[Isaac Gouy]

On Sep 9, 7:19 pm, gary ng <garyng2...@gmail.com> wrote:
> On Thu, Sep 9, 2010 at 7:02 PM, Isaac Gouy <igo...@yahoo.com> wrote:
> > iirc the Haskell programs, and the Clean programs, and the Pascal
> > programs, and ... use translations of the simple hash table used by
> > the C programs.
>
> > If I ever knew, I don't recall why the Haskell program does not use
> > Data.HashTable


> Could be that it wasn't there at the time it was submitted.

Is there any point speculating about this as outsiders?

It was available - Data.HashTable seems to be copyright 2003.

http://ogi.altocumulus.org/~hallgren/Programatica/tools/pfe.cgi?Data.HashTable

> And Data.HashTable also use some mutable array thing so it is still not
> the idiomatic Haskell of everything is immutable.

afaict idiomatic is everything referentially transparent not
everything immutable, as in - "I needed a halfway decent mutable
collection type – having a properly tested mutable collection that
actually performs well would be a godsend."

And maybe the description for this bug report says it all -

http://hackage.haskell.org/trac/ghc/ticket/3149

[gary ng]

On Thu, Sep 9, 2010 at 10:04 PM, Isaac Gouy <igo...@yahoo.com> wrote:
> Is there any point speculating about this as outsiders?
>
> It was available - Data.HashTable seems to be copyright 2003.
>
> http://ogi.altocumulus.org/~hallgren/Programatica/tools/pfe.cgi?Data.HashTable

Huh ? point ? it was just a casual comment, no point was intended. And
I have read some comments by Don that what is in the shoutout is way
faster than Data.HashTable

>
>
>> And Data.HashTable also use some mutable array thing so it is still not
>> the idiomatic Haskell of everything is immutable.
>
> afaict idiomatic is everything referentially transparent not
> everything immutable, as in - "I needed a halfway decent mutable
> collection type – having a properly tested mutable collection that
> actually performs well would be a godsend."
>

how can you assure that(let's put aside the under the hood things
where it is done in C) if you break the assumption that value can be
changed.

BTW, it seems that to verteran Haskellers, the speed of Data.HashTable
is not a concern to them as they said they don't see the need of
that(mutable hasktable) in their usage. There seems to be a bug(as in
it is slow) filed for this package for quite a while but no one even
bother to attempt to improve it.

[Isaac Gouy]

On Sep 9, 10:15 pm, gary ng <garyng2...@gmail.com> wrote:
> On Thu, Sep 9, 2010 at 10:04 PM, Isaac Gouy <igo...@yahoo.com> wrote:
> > Is there any point speculating about this as outsiders?
>
> > It was available - Data.HashTable seems to be copyright 2003.
>

> >http://ogi.altocumulus.org/~hallgren/Programatica/tools/pfe.cgi?Data....

>
> Huh ? point ? it was just a casual comment, no point was intended. And
> I have read some comments by Don that what is in the shoutout is way
> faster than Data.HashTable

If you knew there was another option why write "I doubt there is any
choice for Haskell" ?

> >> And Data.HashTable also use some mutable array thing so it is still not
> >> the idiomatic Haskell of everything is immutable.
>
> > afaict idiomatic is everything referentially transparent not
> > everything immutable, as in - "I needed a halfway decent mutable
> > collection type – having a properly tested mutable collection that
> > actually performs well would be a godsend."
>
> how can you assure that(let's put aside the under the hood things
> where it is done in C) if you break the assumption that value can be
> changed.

Reading the first paragraph "Introduction to Uniqueness Typing" of
this paper should help you with that question -

http://www.st.cs.ru.nl/papers/2008/vrie08-IFL07-UniquenessTypingSimplified.pdf

> BTW, it seems that to verteran Haskellers, the speed of Data.HashTable
> is not a concern to them as they said they don't see the need of
> that(mutable hasktable) in their usage. There seems to be a bug(as in
> it is slow) filed for this package for quite a while but no one even
> bother to attempt to improve it.

I'm not going to speculate about what unnamed "verteran Haskellers"
have or have not said.

Read the bug report and you'll see that the root problem was GC - do
you think changing GC is a quick fix?

[gary ng]

On Fri, Sep 10, 2010 at 8:49 AM, Isaac Gouy <igo...@yahoo.com> wrote:
>> Huh ? point ? it was just a casual comment, no point was intended. And
>> I have read some comments by Don that what is in the shoutout is way
>> faster than Data.HashTable
>
>
> If you knew there was another option why write "I doubt there is any
> choice for Haskell" ?

That was referring to the original comment about 'making a mutable
implementation' meaning the program itself ASKED for that particular
functionality. The one in the shoutout is one implementation of it,
the one in Data.HashTable is another. When I said no choice, I was
saying no choice but to implement a mutable hash table, in respond to
the underlying message of 'why did they implement a mutable hashtable'
as if making it mutable is only for performance purpose.

>
>
>> >> And Data.HashTable also use some mutable array thing so it is still not
>> >> the idiomatic Haskell of everything is immutable.
>>
>> > afaict idiomatic is everything referentially transparent not
>> > everything immutable, as in - "I needed a halfway decent mutable
>> > collection type – having a properly tested mutable collection that
>> > actually performs well would be a godsend."
>>
>> how can you assure that(let's put aside the under the hood things
>> where it is done in C) if you break the assumption that value can be
>> changed.
>
>
> Reading the first paragraph "Introduction to Uniqueness Typing" of
> this paper should help you with that question -
>
> http://www.st.cs.ru.nl/papers/2008/vrie08-IFL07-UniquenessTypingSimplified.pdf
>
>

Would try to digest that.

> I'm not going to speculate about what unnamed "verteran Haskellers"
> have or have not said.
>
> Read the bug report and you'll see that the root problem was GC - do
> you think changing GC is a quick fix?
>

Don suggested on reddit for some fixes that tackle the GC issue(not
changing the GC, but the way to get around the performance caused by
GC), using similar approach as in the shoutout.

[Isaac Gouy]

On Sep 10, 10:35 am, gary ng <garyng2...@gmail.com> wrote:
> On Fri, Sep 10, 2010 at 8:49 AM, Isaac Gouy <igo...@yahoo.com> wrote:
> >> Huh ? point ? it was just a casual comment, no point was intended. And
> >> I have read some comments by Don that what is in the shoutout is way
> >> faster than Data.HashTable
>
> > If you knew there was another option why write "I doubt there is any
> > choice for Haskell" ?
>
> That was referring to the original comment about 'making a mutable
> implementation' meaning the program itself ASKED for that particular
> functionality. The one in the shoutout is one implementation of it,
> the one in Data.HashTable is another. When I said no choice, I was
> saying no choice but to implement a mutable hash table, in respond to
> the underlying message of 'why did they implement a mutable hashtable'
> as if making it mutable is only for performance purpose.

Clearly, they did choose "to write all that code" "in order to get a
much faster program" - I can't tell you if Andy had noticed the
benchmark was about "Hashtable update and k-nucleotide strings" or
whether he knew about Data.HashTable.

> >> >> And Data.HashTable also use some mutable array thing so it is still not
> >> >> the idiomatic Haskell of everything is immutable.
>
> >> > afaict idiomatic is everything referentially transparent not
> >> > everything immutable, as in - "I needed a halfway decent mutable
> >> > collection type – having a properly tested mutable collection that
> >> > actually performs well would be a godsend."
>
> >> how can you assure that(let's put aside the under the hood things
> >> where it is done in C) if you break the assumption that value can be
> >> changed.
>
> > Reading the first paragraph "Introduction to Uniqueness Typing" of
> > this paper should help you with that question -
>

> >http://www.st.cs.ru.nl/papers/2008/vrie08-IFL07-UniquenessTypingSimpl...

>
> Would try to digest that.

Bite-sized:

"An important property of pure functional programming languages is
referential transparency: the same expression used twice must have the
same value twice."

"A side effect on a variable (file) is okay as long as that variable
is never used again: it is okay for a function to modify its input if
the input is not shared. Referential transparency then trivially holds
because the same expression never occurs more than once."

"Rather than just returning the read character, fgetc returns a pair
consisting of the read character and a new file, file1. Although file0
and file1 point to the same file on disk, they are conceptually and
syntactically different, and thus it is clear that a and b may have
different values."

[gary ng]

On Fri, Sep 10, 2010 at 11:13 AM, Isaac Gouy <igo...@yahoo.com> wrote:
> Clearly, they did choose "to write all that code" "in order to get a
> much faster program" - I can't tell you if Andy had noticed the
> benchmark was about "Hashtable update and k-nucleotide strings" or
> whether he knew about Data.HashTable.

I know they did it to get a faster program, but under the requirement of :

IT MUST BE MUTABLE HASHTABLE

If you change the requirement to something else that acheives the end
result without this specific requirement, they may not use hashtable
at all. Which is also the message I get from various place like reddit
or stackoverlow about this same issue. That is 'use your language's
strength to solve the problem, not to mimic other language, i.e. the
what not the how'.

And this sentiment is not unique to Haskell. I recently saw similar
questions on J language of 'how can I write a Haskell style
filter(HOF)', the overall answers there are 'you don't need haskell
style filter and can achieve the same result in a J way'. Similarly
for Haskell which is 'you don't need to use mutable hash table and can
very likely get the same result using other immutable data structure'

But in this case the problem is 'implement a mutable hashtable' which
is what my 'no choice' was referring to. Unlike language say Python or
F# where hash table is in general considered to be part of the
language, Data.HashTable is not. It is just one 'sample' of
implementation of mutable hash table which happens to be slow.

I read this part, trying to fit that into this mutable hashtable
context. Below this paragraph, it mentioned some
requirement/assumption that file0, file1 ... cannot be 're-used' which
I don't know how to apply to hashtable.

[Isaac Gouy]

On Sep 10, 11:54 am, gary ng <garyng2...@gmail.com> wrote:
> On Fri, Sep 10, 2010 at 11:13 AM, Isaac Gouy <igo...@yahoo.com> wrote:
> > Clearly, they did choose "to write all that code" "in order to get a
> > much faster program" - I can't tell you if Andy had noticed the
> > benchmark was about "Hashtable update and k-nucleotide strings" or
> > whether he knew about Data.HashTable.
>
> I know they did it to get a faster program, but under the requirement of :
>
> IT MUST BE MUTABLE HASHTABLE
>
> If you change the requirement to something else that acheives the end
> result without this specific requirement, they may not use hashtable
> at all. Which is also the message I get from various place like reddit
> or stackoverlow about this same issue. That is 'use your language's
> strength to solve the problem, not to mimic other language, i.e. the
> what not the how'.

It's starting to look like actually there was a point you wanted to
make ;-)

If you change the requirement to something else you'd simply be
missing the point - which was to look at hashtable updates and
strings.

Does it actually say that the hashtable updates have to be
destructive?

> And this sentiment is not unique to Haskell. I recently saw similar
> questions on J language of 'how can I write a Haskell style
> filter(HOF)', the overall answers there are 'you don't need haskell
> style filter and can achieve the same result in a J way'. Similarly
> for Haskell which is 'you don't need to use mutable hash table and can
> very likely get the same result using other immutable data structure'
>
> But in this case the problem is 'implement a mutable hashtable'  which
> is what my 'no choice' was referring to. Unlike language say Python or
> F# where hash table is in general considered to be part of the
> language, Data.HashTable is not. It is just one 'sample' of
> implementation of mutable hash table which happens to be slow.

There's no choice in binary-trees either - you must use binary-trees
and you must allocate all the memory.

There's no choice in pi-digits either - you must use that particular
step-by-step spigot algorithm taken from that Haskell paper.

There's no choice in fannkuch-redux either - you must use that
particular ordering of permutations taken from that Lisp paper.

etc etc

Forgive me but you do seem to be belabouring the obvious.


-snip-

> > Bite-sized:
>
> > "An important property of pure functional programming languages is
> > referential transparency: the same expression used twice must have the
> > same value twice."
>
> > "A side effect on a variable (file) is okay as long as that variable
> > is never used again: it is okay for a function to modify its input if
> > the input is not shared. Referential transparency then trivially holds
> > because the same expression never occurs more than once."
>
> > "Rather than just returning the read character, fgetc returns a pair
> > consisting of the read character and a new file, file1. Although file0
> > and file1 point to the same file on disk, they are conceptually and
> > syntactically different, and thus it is clear that a and b may have
> > different values."
>
> I read this part, trying to fit that into this mutable hashtable
> context. Below this paragraph, it mentioned some
> requirement/assumption that file0, file1 ... cannot be 're-used' which
> I don't know how to apply to hashtable.

Do you know if we re-write that as - Rather than just returning the
updated value, htAddOrUpdate returns a pair consisting of the updated
value and a new hashtable, hashtable1. Although hashtable0 and
hashtable1 point to the same structures in memory, they are

[gary ng]

On Fri, Sep 10, 2010 at 2:04 PM, Isaac Gouy <igo...@yahoo.com> wrote:
> It's starting to look like actually there was a point you wanted to
> make ;-)

You mean the 'no chice' part ? yes.

You mean the why not Data.Hashtable comment ? I don't think so.

>
> If you change the requirement to something else you'd simply be
> missing the point - which was to look at hashtable updates and
> strings.

I know perfectly well what you want to measure and I was not implying
if those measure is valid or not, I am totally neutral to the whole
thing.

My initial comment was all about 'it seems that Haskell submission is
not the typical elegant form' and to me because of the specific you
want to measure, there is no acceptable elegant Haskell form.

>
> Does it actually say that the hashtable updates have to be
> destructive?

If want to go that route, be my guest.

> There's no choice in binary-trees either - you must use binary-trees
> and you must allocate all the memory.
>
> There's no choice in pi-digits either - you must use that particular
> step-by-step spigot algorithm taken from that Haskell paper.
>
> There's no choice in fannkuch-redux either - you must use that
> particular ordering of permutations taken from that Lisp paper.
>
> etc etc

Indeed. Your benchmark suite are very specific about how certain
things must be done in order to measure a specific aspect you want to
measure.

>
> Forgive me but you do seem to be belabouring the obvious.
>

I think you are too sensitive

[Isaac Gouy]

On Sep 10, 2:22 pm, gary ng <garyng2...@gmail.com> wrote:
-snip-

> My initial comment was all about 'it seems that Haskell submission is
> not the typical elegant form' and to me because of the specific you
> want to measure, there is no acceptable elegant Haskell form.

So what are we to do when there's a problem that has "no acceptable
elegant Haskell form"?

> > Does it actually say that the hashtable updates have to be
> > destructive?
>
> If want to go that route, be my guest.

I don't understand what you mean, didn't you say - it must be a
mutable hashtable?

Why couldn't it be a hashtable that copied values?

> Indeed. Your benchmark suite are very specific about how certain
> things must be done in order to measure a specific aspect you want to
> measure.

The other complaint is that it's not specific enough and allows too
much play, with programs in one language being too different to
programs in other languages.

[gary ng]

On Fri, Sep 10, 2010 at 3:06 PM, Isaac Gouy <igo...@yahoo.com> wrote:
> So what are we to do when there's a problem that has "no acceptable
> elegant Haskell form"?

Depending on the intend, for you benchmark program, write something
like what it is now.

For real life cases, call an exnternal library, use another language,
transform the solution into something that would not hit the weak
point of the language(I recently saw an example of this on J about a
simple but interesting quiz).

>
>
>> > Does it actually say that the hashtable updates have to be
>> > destructive?
>>
>> If want to go that route, be my guest.
>
> I don't understand what you mean, didn't you say - it must be a
> mutable hashtable?
>
> Why couldn't it be a hashtable that copied values?

As I said, be my guest.

>
>
>> Indeed. Your benchmark suite are very specific about how certain
>> things must be done in order to measure a specific aspect you want to
>> measure.
>
> The other complaint is that it's not specific enough and allows too
> much play, with programs in one language being too different to
> programs in other languages.
>

not sure what you want to say.