Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Benchmark Results for CL's onWin32

14 views
Skip to first unread message

John Watton

unread,
Dec 1, 1999, 3:00:00 AM12/1/99
to

A while ago I mentioned in a posting that I had done some benchmarking
of Common Lisp's for Win32. I received a few emails requesting the
results. So....here they are.

CL-TYPE Allegro CL LispWorks Poplog CL CLISP CormanLisp
CL-VER 5.0.1 4.1.18 2.0 1999-07-22 1.3
BOYER 1.000 10.050 1.033 4.040 1.708
BROWSE 1.000 4.487 2.184 5.142 3.355
CTAK 1.000 1.839 5.903 4.258 3.757
DDERIV 1.288 3.220 2.190 3.220 1.000
DERIV 1.483 3.758 1.928 3.791 1.000
DESTRU 1.000 2.639 2.222 5.361 1.500
DIV2 1.279 4.256 1.790 5.864 1.000
FFT 1.000 41.050 67.000 46.050 53.998
FPRINT 1.630 1.935 4.261 1.000 6.168
FREAD 1.458 6.958 1.250 1.000 8.534
FRPOLY 1.084 1.737 1.000 3.106 1.299
PUZZLE 1.441 1.816 1.000 4.850 5.409
STAK 1.000 1.333 2.310 4.167 3.910
TAK 1.000 3.500 3.500 21.025 1.863
TAKL 1.000 2.321 2.321 16.357 1.608
TAKR 1.000 4.273 2.000 9.182 1.450
TRAVERSE 1.000 5.156 3.370 13.167 5.829
TRIANG 1.287 1.000 2.779 15.688 8.254
RHYME 1.000 2.178 1.831 2.324 95.729
PRIMECOUNTFIXNUM 1.000 1.740 2.122 3.878 6.463
PRIMECOUNTBIGNUM 2.296 1.000 1.063 1.489 26.100
PNPOLY 1.000 1.290 32.031 42.536 42.301
GENERICFUNCTIONS 1.000 1.111 5.926 3.951 37.946
SLOTVALUES 1.074 1.000 1.407 1.639 5.027
TOTAL 1.000 3.872 5.382 7.877 11.483

Notes:

1) All benchmarks run on a Pentium II at 366Mhz with Windows NT.

2) The benchmark values are normalized to the best performer. The total
is computed
as the sum of the individual normalized values and then is normalized
itself. This
means that each benchmark is weighted equally in the total.

3) Allegro CL http://www.franz.com (I used the commercial version)

4) Lispworks http://www.harlequin.com (I used the commercial version)

5) Poplog CL http://www.cs.bham.ac.uk/research/poplog/freepoplog.html
(Free with source)

7) Clisp http://clisp.cons.org (Free with source)

8) CormanLisp http://corman.net (Free with source) Cormanlisp is the
least complete and robust. Write-line and asin do not work. Loop macro
doesn't understand minimizing or of-type. Defstruct cannot do (:type
list). with-open-file :if-exists :append appears to :supersede
instead. Environment information functions are not implemented.

9) Gabriel benchmarks downloaded from
ftp://ftp.cs.umass.edu/pub/eksl/gabriel/. Benchmarks run with
(optimize (speed 3) (safety 1) (space 0)).

10) Primecount benchmarks also make use of a Lispworks nonANSI
declaration
(optimize (fixnum 0)).

11) Rhyme exercises read and write of ascii files along with string sort
and list reversal.

12) Primecountfixnum is a fixnum integer intensive computation.

13) Primecountbignum is a bignum integer intensive computation.

14) Pnpoly is a floating point intensive computation. Pnpoly is
optimized
for floating point using declarations and optimized for (speed 3) and
(safety 0). I believe that only Allegro CL and Lispworks pay any
attention. Unfortunately Lispworks also requires a (declare (optimize
(float 0))) to make any real gains as well. This is an nonANSI
feature. It is also not a very robust one. FFT is floating point
intensive but the use of (float 3) only came up with Lispworks
compiler errors.

15) Genericfunctions and slotvalues test generic function calling and
slot
value accessing. They were obtained from the CormanLisp distribution.

16) My source:

(defun rhyme (from to &aux dict)
(declare (optimize (speed 3) (safety 0)))
(with-open-file (in from :direction :input)
(setq dict (loop for w = (read-line in nil nil)
until (not w) collect w)))
(setq dict (mapc #'nreverse dict))
(setq dict (sort dict #'string<))
(setq dict (mapc #'nreverse dict))
(with-open-file (out to :direction :output :if-exists :supersede)
(dolist (x dict)
#-cormanlisp (write-line x out)
#+cormanlisp (progn (write-string x out) (terpri out))
)))

(defun prime-count (s e)
(declare (optimize (speed 3) (safety 0))
#+Lispworks (optimize (fixnum-safety 3))
(integer s e))
(macrolet ((search-template (typeint)
`(loop for n of-type ,typeint from (if (oddp x) x (1+
x)) to y by 2
for stop of-type ,typeint = (isqrt n)
with count of-type ,typeint = (if (<= x 2 y) 1 0) ;
to count 2
when (= n (loop for test-div of-type ,typeint from
3 to stop by 2
when (zerop (mod n test-div)) do
(return test-div)
finally (return n)))
do (incf count)
finally (print count))))
(flet ((search-fixnum-range (x y) (search-template fixnum))
(search-bignum-range (x y) (search-template integer)))
(if (> e most-positive-fixnum)
(search-bignum-range s e)
(search-fixnum-range s e)))))


(defun pnpoly (npol xp yp x y)
(declare (optimize (speed 3) (safety 0))
#+Lispworks (optimize (float 0))
(fixnum npol)
(double-float x y)
(type (simple-array double-float (*)) xp yp))
(let* ((c nil)
(j (1- npol)))
(declare (fixnum j))
(dotimes (i npol c)
(declare (fixnum i))
(if (and (or (and (<= (aref yp i) y) (< y (aref yp j)))
(and (<= (aref yp j) y) (< y (aref yp i))))
(< x (+ (aref xp i) (/ (* (- (aref xp j) (aref xp i))
(- y (aref yp i)))
(- (aref yp j) (aref yp i))))))
(setq c (not c)))
(setq j i))))

(defun pnpolymain (npol div)
(declare (optimize (speed 3) (safety 0))
#+Lispworks (optimize (float 0))
(fixnum npol div))
(let* ((xp (make-array npol :element-type 'double-float))
(yp (make-array npol :element-type 'double-float))
(theta 0.0d0)
(a 10.0d0)
#+clisp (pi (float pi 0.0d0))
(fdiv (/ (* 2 a) div))
(count 0))
(declare (double-float fdiv a theta)
(fixnum count)
(type (simple-array double-float (*)) xp yp))
(dotimes (i npol)
(declare (fixnum i))
(setq theta (/ (* 2 i pi) npol))
(setf (aref xp i) (+ a (* a (expt (cos theta) 3)))
(aref yp i) (+ a (* a (expt (sin theta) 3)))))
(dotimes (u (1+ div))
(declare (fixnum u))
(dotimes (v (1+ div))
(declare (fixnum v))
(if (pnpoly npol xp yp
#+Lispworks (* (coerce u 'double-float) fdiv)
#-Lispworks (* u fdiv)
#+Lispworks (* (coerce v 'double-float) fdiv)
#-Lispworks (* v fdiv)
)
(incf count))))
(format t "Area: ~a" (/ (* count 4 a a) (* (1+ div) (1+ div))))))

(defclass foo () ())

(defclass bar () (a b c))

(defclass baz () (x y))

(defgeneric gf-1-1 (a))
(defmethod gf-1-1 ((a foo)))

(defgeneric gf-1-2 (a))
(defmethod gf-1-2 ((a foo)))
(defmethod gf-1-2 ((a symbol)))

(defgeneric gf-1-3 (a))
(defmethod gf-1-3 ((a integer)))
(defmethod gf-1-3 ((a bar)))
(defmethod gf-1-3 ((a t)))

(defgeneric gf-2-3 (a b))
(defmethod gf-2-3 ((a integer) (b bar)))
(defmethod gf-2-3 ((a foo) (b t)))
(defmethod gf-2-3 ((a t) (b integer)))

(defun bench-gfs ()
(let ((foo (make-instance 'foo))
(bar (make-instance 'bar))
(baz (make-instance 'baz)))
(dotimes (i 10000)
(gf-1-1 foo)
(gf-1-1 foo)
(gf-1-2 foo)
(gf-1-2 'quux)
(gf-1-3 5)
(gf-1-3 foo)
(gf-2-3 5 bar)
(gf-2-3 foo 'quux)
(gf-2-3 baz 7))))

(defun bench-slot-value ()
(let ((bar (make-instance 'bar))
(baz (make-instance 'baz)))
(dotimes (i 100000)
(setf (slot-value bar 'a) 10)
(setf (slot-value baz 'y) 20)
(slot-value bar 'a)
(setf (slot-value bar 'c) 777)
(slot-value bar 'c)
(slot-value baz 'y))))

(defmacro ttime (&rest body)
`(progn ,@body))

(defun testrhyme ()
(print (ttime (rhyme "c:/personal/wattojd/benchmarks/gabriel/dict.txt"

"c:/personal/wattojd/benchmarks/gabriel/rhyme.txt"))))

(defun testprimecountfixnum ()
(print (ttime (prime-count 0 200000))))

(defun testprimecountbignum ()
(print (ttime (prime-count 1000000000050 1000000000100))))

(defun testpnpoly ()
(print (ttime (pnpolymain 200 400))))

(defun testgenericfunctions ()
(print (ttime (bench-gfs))))

(defun testslotvalues ()
(print (ttime (bench-slot-value))))

--
John Watton
Alcoa Inc.


Sent via Deja.com http://www.deja.com/
Before you buy.

Fernando

unread,
Dec 2, 1999, 3:00:00 AM12/2/99
to
On Wed, 01 Dec 1999 20:14:29 GMT, John Watton <john....@alcoa.com>
wrote:

>
>
>A while ago I mentioned in a posting that I had done some benchmarking
>of Common Lisp's for Win32. I received a few emails requesting the
>results. So....here they are.

<snip>

Thanks for posting it. :-) BTW, is there a benchmark for some unix
flavour? I'd like to see how CMUCL performs. :-)

PS Is someone working on a win32 of cmucl? O:-)


//-----------------------------------------------
// Fernando Rodriguez Romero
//
// frr at mindless dot com
//------------------------------------------------

Pekka P. Pirinen

unread,
Dec 2, 1999, 3:00:00 AM12/2/99
to
John Watton <john....@alcoa.com> writes:
> 9) Gabriel benchmarks downloaded from
> ftp://ftp.cs.umass.edu/pub/eksl/gabriel/. Benchmarks run with
> (optimize (speed 3) (safety 1) (space 0)).

We don't think these short benchmarks are a good indicator of system
performance, but if you're going to play around with them, you ought
try them with (SAFETY 0).

> FFT is floating point intensive but the use of (float 3) only came
> up with Lispworks compiler errors.

Do you mean (FLOAT 0)? Either way, we've not seen such errors.
Evidently, the version of FFT that you used didn't have declarations
that LW could use to avoid boxing; your version of PNPOLY seems to
have the right ones, so you could try something like that in FFT.
--
Pekka P. Pirinen, Adaptive Memory Management Group, Harlequin Limited
"A man never speaks of himself without losing something. What he says
in his disfavor is always believed but when he commends himself, he arouses
mistrust." - Michel Montaigne

Martin Simmons

unread,
Dec 2, 1999, 3:00:00 AM12/2/99
to

John Watton <john....@alcoa.com> wrote in article
<823vj1$ska$1...@nnrp1.deja.com>...

> 14) Pnpoly is a floating point intensive computation. Pnpoly is
> optimized
> for floating point using declarations and optimized for (speed 3) and
> (safety 0). I believe that only Allegro CL and Lispworks pay any
> attention. Unfortunately Lispworks also requires a (declare (optimize
> (float 0))) to make any real gains as well. This is an nonANSI
> feature. It is also not a very robust one. FFT is floating point
> intensive but the use of (float 3) only came up with Lispworks
> compiler errors.

I've just had a look at the problem with FFT: it is a bug that only
manifests itself with float 0 & safety > 0, whch is an unusual set of
options.

For best performance, you should also turn off all debugging support. In
LispWorks, thie means doing:

(proclaim '(optimize (debug 0)))
(toggle-source-debugging nil)

--
Martin Simmons, Lisp Group, Harlequin Limited
mar...@harlequin.co.uk


Erik Naggum

unread,
Dec 2, 1999, 3:00:00 AM12/2/99
to
* "Martin Simmons" <mar...@harlequin.co.uk>

| For best performance, you should also turn off all debugging support.

all of these benchmarks seem to be testing fairly unusual conditions.
the arguments we see here for "tuning" seem to me to imply that we have
to chuck all the Lispy features we can think of to get a comparison. I
think this is completely bogus. I'm not going to trust _myself_ enough
to run my code with the kinds of optimization settings people advocate
for these benchmarks. I _want_ a lot of dynamism, debugging support,
etc, and I'm not usually ready to run my code in production without these
things, except in particular functions that I have determined to need it,
and which have been specially written to be blindingly fast. however, in
the latter case, I'd be pretty damn stupid if I thought I could have 100%
portable ANSI Common Lisp code _and_ super-duper maximal performance at
the same time. _sheesh_, guys. if I had such needs as you imply with
these declarations for "best performance", I would even hack the compiler
output and look for instruction scheduling opportunities myself. using a
special function to turn off debugging, for instance, is just _one_ such
measure to squeeze the bejeezus out of the system, and we're already
outside of the portable ANSI Common Lisp domain.

so in my view, you either want super-fast code and profile it and get the
optimal results with hand-tweaked code that may not work anywhere else,
or you use fully portable code under _normal_ condtions. anything else
is very close to cheating, rendering the benchmark results evidence of
how good you are at cheating at random in each implementation with no way
of comparing your skills and then _nothing_ else. if you let instead
consciously and openly do your extreme best to write fast code for each
implementation, and _then_ compare the results, you at the very least get
comparable results for comparable _people_ (i.e., experts).

I suggest such benchmarks limit themselves to the narrowest possible type
declarations, the highest SPEED and the lowest DEBUG and SAFETY, and
either full local tweaking or no local tweaking. if a declaration isn't
enough, that's a factor that you need to take into account when and if
you engage in _full_ local tweaking, and not at all until then.

#:Erik

John Watton

unread,
Dec 3, 1999, 3:00:00 AM12/3/99
to
Okay, I reran things incorporating the suggestions of the two Harlequin
folks so that we are compiling with (optimize (speed 3) (safety 0)
(space 0) (debug 0)). Also specifically for Lispworks I set (toggle-
source-debugging nil) and I got the (declare (optimize (float 0))) to
work in FFT. I also incorporate a suggestion into PRIMECOUNTFIXNUM
which benefits ACL (the fixnum range is declared to be positive only).

I agree with Erik to some degree and it does feel like cheating when
going outside the ANSI standard to use declarations such as Lispworks
float and fixnum-safety and their toggle-source-debugging. Otherwise
sticking with the standard I don't feel like I am cheating at all and
in fact most of my (real life) applications are sensitive to floating
and fixnum arithmetic and as a consequence they are highly optimized
with declarations and I feel that I am quite good at making the most of
this approach with the Franz ACL product.

CL-TYPE Allegro CL LispWorks Poplog CL CLISP CormanLisp
CL-VER 5.0.1 4.1.18 2.0 1999-07-22 1.3

BOYER 1.177 3.889 1.399 5.017 1.000
BROWSE 1.000 4.791 2.512 5.791 4.077
CTAK 1.000 1.077 7.000 5.077 4.462
DDERIV 1.000 4.517 2.667 4.167 1.188
DERIV 1.000 4.736 2.170 4.321 1.131
DESTRU 1.000 5.214 5.643 13.643 3.831
DIV2 1.000 5.091 2.205 7.273 1.239
FFT 1.000 1.500 67.000 45.550 53.126
FPRINT 1.622 1.422 4.311 1.000 6.297
FREAD 1.478 5.870 1.217 1.000 8.897
FRPOLY 1.000 1.779 1.119 3.655 1.542
PUZZLE 1.452 1.452 1.000 5.006 5.577
STAK 1.444 1.000 3.556 6.481 6.057
TAK 1.000 1.500 3.500 21.025 1.853
TAKL 1.000 1.370 2.407 16.889 1.664
TAKR 1.000 2.300 2.100 10.000 1.592
TRAVERSE 1.000 2.824 3.926 15.336 6.767
TRIANG 1.531 1.000 3.210 18.219 9.504
RHYME 1.000 2.033 1.910 2.451 87.079
PRIMECOUNTFIXNUM 1.000 4.958 6.334 11.537 19.227
PRIMECOUNTBIGNUM 2.220 1.000 1.093 1.536 26.246
PNPOLY 1.000 1.902 32.779 43.533 41.890
GENERICFUNCTIONS 1.333 1.000 7.167 4.667 51.133
SLOTVALUES 1.156 1.000 1.682 1.966 6.105
TOTAL 1.000 2.225 5.909 8.979 12.370

Fernando

unread,
Dec 3, 1999, 3:00:00 AM12/3/99
to
On 02 Dec 1999 17:38:42 +0000, pe...@harlequin.co.uk (Pekka P.
Pirinen) wrote:

>John Watton <john....@alcoa.com> writes:
>> 9) Gabriel benchmarks downloaded from
>> ftp://ftp.cs.umass.edu/pub/eksl/gabriel/. Benchmarks run with
>> (optimize (speed 3) (safety 1) (space 0)).
>

>We don't think these short benchmarks are a good indicator of system
>performance, but if you're going to play around with them, you ought
>try them with (SAFETY 0).

What would you consider a good indicator of system performance
then? What benchmarking would you use?

Fernando

unread,
Dec 3, 1999, 3:00:00 AM12/3/99
to
On 2 Dec 1999 20:34:48 GMT, "Martin Simmons" <mar...@harlequin.co.uk>
wrote:


>I've just had a look at the problem with FFT: it is a bug that only
>manifests itself with float 0 & safety > 0, whch is an unusual set of
>options.

What about boyer?

Martin Simmons

unread,
Dec 3, 1999, 3:00:00 AM12/3/99
to

Erik Naggum <er...@naggum.no> wrote in article
<31531662...@naggum.no>...

> I suggest such benchmarks limit themselves to the narrowest possible
type
> declarations, the highest SPEED and the lowest DEBUG and SAFETY, and
> either full local tweaking or no local tweaking. if a declaration
isn't
> enough, that's a factor that you need to take into account when and if
> you engage in _full_ local tweaking, and not at all until then.

In fact, given the insane number of combinations for OPTIMIZE decls, this
is a good strategy for all code, not just benchmarks.

Moreover, with the luxury of hindsight, some of the extra optimization
settings for LW should probably be controlled by the existing quality
names.

Jason Trenouth

unread,
Dec 6, 1999, 3:00:00 AM12/6/99
to
On Fri, 03 Dec 1999 16:43:53 GMT, John Watton <john....@alcoa.com> wrote:

> Okay, I reran things incorporating the suggestions of the two Harlequin
> folks so that we are compiling with (optimize (speed 3) (safety 0)
> (space 0) (debug 0)). Also specifically for Lispworks I set (toggle-
> source-debugging nil) and I got the (declare (optimize (float 0))) to
> work in FFT. I also incorporate a suggestion into PRIMECOUNTFIXNUM
> which benefits ACL (the fixnum range is declared to be positive only).
>

You might also try (COMPILATION-SPEED 0) to make the LispWorks compiler work
harder on larger bodies of code. The Gabriels probably sufficiently small for
this not to make much difference.

__Jason

0 new messages