Fritz 4 and the SSDF
Moritz Berger
rating list - which is ELO 2285 on a P90, rank 28! - I want
to share my personal experiences with you - just in case you
wondered how this could happen, since Fritz 3 was ranked by
the SSDF more than 80 points better and Fritz 3 had an ELO
performance of 2508 over 30 games in FIDE tournaments it
participated in (ChessBase even claims this on the outside of
the box).
Hardware performance
====================
There are many performance parameters like different cache
architectures, EDO RAMs and mainboard chipsets (e.g. Intel
Triton etc.). Performance can be up to factor 2 faster/slower
with the same processor, i.e. if SSDF writes P90, this can
(also depending on the chess program and the frequency it
uses the computers RAM and things like if some data fits into
the 2nd level cache or not) translate to a performance
anywhere between a P120 or P66 relative to a P90 in an
hypothetical "average" mainboard. In the SSDF list you just
don't know how fast their machines really are, I don't think
that their P90 are all of the same speed (since that would
have more or less implied that all their testers bought the
same machines - somewhat difficult for a non-commerical
organisation).
Hashtable issues
================
But it gets even worse: Fritz 4 is a good example of a
(Windows-)program that is a real memory hog. If you have less
than 16MB RAM with Windows 95, you can't increase the default
size of the 512 KB hash tables. Fritz is one of the fastest
programs, in my experience it plays better (i.e. there's a
small, but measurable advantage) with 32 MB hash tables than
with 16MB HT even at 30 minutes/game time controls. Compared
with the default hash table size of 512 KB you can easily
observe a speedup of factor 2 or even more with bigger hash
tables at longer time controls (e.g. 32MB at tournament
level, SSDF rating are supposed to express playing strength
at tournament level). Since I don't assume that all SSDF
testers have more than 32 MB RAM in all their machines, this
could mean that if you have a more generous RAM capacity and
a very fast mainboard, the program under discussion will play
in an entirely different class (several times faster). I
guess that Fritz 4 is well above 2400 FIDE ELO (if computers
could be rated by FIDE) on my P5/166 with 64 MB EDO RAM,
Intel Triton chipset and synchronous PB cache.
In some cases (e.g. 8 MB with Win95) Fritz 4 could even be
forced to permanently access the swap file due to
insufficient RAM. This is the worst thing of all and can slow
down the program almost infinitely.
Of course most of us don't have the most powerful machines
around for chess playing purposes. For the average, "slow"
486/100 computer and 8 MB RAM running Fritz 4 under Win95 the
SSDF results indicate that in terms of playing strength there
might be better programs around. But considering recent RAM
prices you might have wondered what it did to your Fritz 4
program if you added more RAM.
SSDF testing methods
====================
The SSDF people don't eliminate duplicate games, i.e. if a
program doesn't have a learning function, it can always lose
the same game which is then rated as if 20 games had been
played. Or if it does have a learning function, who
guarantees that the same program runs on the same machine?
Nobody does, so you could imagine 10 testers playing 20 and
producing 2 different games (except if we had major
deviations due to different hardware, see above).
The strength of a programs opponents is not predictable.
Fritz 4 was only played against dedicated chess computers,
not against other PC programs (except Genius 4). Although the
average rating of the opponents is given and in the longer
SSDF list individual scores are mentioned, some phenomena
like "incest" among programs (programs playing against older
versions of themselves) can lead to skewed results (Genius 4
P90 played against 7 or 8 of its predecessors, some games
even against itself on slower hardware, 60 games alone
against Lyon (also a Lang program)). Also, the maximum number
of games between 2 opponents varies widely. If I can score
19.5-0.5 against a very weak opponent (Mchess 5 vs.
Sapphire), this surely will give me some of the ELO points I
need to be at the top of the list (under the assumption that
some of my rivals will only play 4 games (Rebel 6) or 8
games (Rebel 7) against this weak opponent) - you get the
picture.
Killer books
============
So called "killer" opening books tend to obscure the true
strength of programs - at least for some time. This is one of
the reasons, that for a given program, playing strength
usually seems to decline over time: The program cannot
outplay more recent programs with its killer lines, on the
other hand it becomes a victim of newer programs which are
prepared for it with still more recent "killer" openings.
Just look at the SSDF ("P90") results of Mchess 4 and 5 vs.
Hiarcs 3 and 4:
Mchess 4 vs. Hiarcs 3 6.5 - 13.5
Mchess 5 vs. Hiarcs 3 16.5 - 3.5
Mchess 5 vs. Hiarcs 4 6.5 - 13.5
You got the idea: Mchess 4 killed by Hiarcs 3 killed by
Mchess 5 killed by Hiarcs 4.
Bugfixes
========
In the special case of Fritz 4, there's a new version called
4.01 which plays better in some positions than the initial
4.0 release. I verified this with the apparently losing moves
that Fritz alter ego Quest made in the games it lost at the
Micro WM 1995 in Paderborn. Almost all questionable moves
that 4.0 would still make are replaced by better ones with
the 4.01 version. This might be some "learning function" like
in Rebel 6, on the other hand it could indicate a better
chess engine. Like with Rebel 7 and "7.01" (actually, there
never was a version called 7.01, just an inline fix still
called 7.0), you can't tell by the SSDF list which program
was used for the rating.
I don't want to defend Chessbase (the company behind Fritz 4)
beyond reason. There are some things about Fritz 4 that I
would have liked to be different, e.g. they could have easily
produced an engine that would run under the old DOS version
of Fritz 3, eliminating some HT size problems on machines
with 8 or 16 MB RAM (IMHO, this would have been very easy for
them since they do have a Fritz 4 engine (that comes with
the program for free) for the Chessbase database). The copy
protection just makes me scream. Still, under optimal
conditions in terms of hardware, Fritz 4.01 is one of the
stronger programs and the disappointing SSDF rating could be
misleading.
Conclusions
===========
The rating deviations in the SSDF list could be even larger
than the statistical error margin they quote with their
individual ratings. The only thing that makes their results
not useless is the law of the big number (is this the English
technical term?) which basically means that if you play
enough games with enough programs on enough computers, you
get more and more "mainstreamed" results and errors tend to
chancel each other out. One problem is the copy protection
(and copyright reasons) of almost all programs that also
makes it for the testers harder to test all programs on all
computers - a program either gets a "good" or a "bad" machine
and then is likely to stay with it. Fritz 4 has the worst
copy protection of all, being unable to run on machines
without CD-ROM.
I don't blame the SSDF people for their results and I don't
accuse them of cheating. They certainly try hard and I
respect their efforts. I only think that they face some
obstacles which they can overcome (sounds like I'm talking
about chess programming - no I don't want to contribute to
the rgcc Deep Blue topic).
Comments welcome, flames > NULL.
Moritz
(ber...@zeus.informatik.uni-bonn.de)
Jouni Uski
Chessbase has never wanted to help SSDF testing - they haven't send Fritz3 and
Fritz4 to Sweden (as other programmers)! Consequence: ratings are delayed with
many months.
And now without SSDF we get (from CSS) conclusion that Fritz4 is a miracle, when
the truth is that it is CRAP. Let's hope that SSDF will never stop their excellent
work.
Jouni
Chris Whittington
Danger: can of worms being opened ....
Jouni is quite correct, but: it is not right to say it is crap.
Frans Morsch is a very good programmer, and Fritz is a strong program.
To read the highly influential CSS, however, you would believe Fritz
was a mega-poweful, mega-beast with zero opposition.
Fritz does appear on SSDF quite low down in comparison with
this level of hype it receives.
Fritz, as a result of this hype and the control of various individuals
was the initial program chosen for the Intel challenge against Kasparov.
Many people reacted, saying this was not right.
Accusations of mafia and corruption abound. Many German programmers
and supporters are rooting for Fritz to lose at tournaments - which
is a shame, since Frans Morsch is a good guy; although I won't
comment about the Chessbase company.
On the other side of the argument CSS publishes the SSDF list which
shows Fritz down in the rating, so ........
Chris Whittington
Robert Hyatt
: One important thing to know: as we can read from SSDF:s club magazine "PLY":
: Chessbase has never wanted to help SSDF testing - they haven't send Fritz3 and
: Fritz4 to Sweden (as other programmers)! Consequence: ratings are delayed with
: many months.
: And now without SSDF we get (from CSS) conclusion that Fritz4 is a miracle, when
: the truth is that it is CRAP. Let's hope that SSDF will never stop their excellent
: work.
:
: Jouni
I wouldn't call it CRAP. From my perspective, it's one hell of a tough
computer opponent. I know. I play it on ICC all the time, thanks to Lonnie.
:)
Moritz Berger
wrote:
> One important thing to know: as we can read from SSDF:s club magazine "PLY":
> Chessbase has never wanted to help SSDF testing - they haven't send Fritz3 and
> Fritz4 to Sweden (as other programmers)! Consequence: ratings are delayed with
> many months.
> And now without SSDF we get (from CSS) conclusion that Fritz4 is a miracle,
> when
> the truth is that it is CRAP. Let's hope that SSDF will never stop their
> excellent
> work.
>
> Jouni
I just wondered if you have read my article. You made your own statement,
but there's no comment concerning my arguments. Maybe you should start
a different thread - how about "Why I hate ChessBase". It's just an issue
of netiquette and I think that we're talking about different
issues that have only the title in common.
Let's split this thread - ok?
Moritz
Hal Bogner
One thing which is not mentioned explicitly in this post, and should be, is
that CSS is controlled in whole or in part by vested parties who derive income
from sale of Fritz and Chessbase, and not from competing products.
In article <83597876...@cpsoft.demon.co.uk> Chris Whittington <chr...@cpsoft.demon.co.uk> writes:
>Jouni Uski <Jouni...@nce.nokia.com> wrote:
>>
>> One important thing to know: as we can read from SSDF:s club magazine "PLY":
>> Chessbase has never wanted to help SSDF testing - they haven't send Fritz3 and
>> Fritz4 to Sweden (as other programmers)! Consequence: ratings are delayed with
>> many months.
>> And now without SSDF we get (from CSS) conclusion that Fritz4 is a miracle, when
>> the truth is that it is CRAP. Let's hope that SSDF will never stop their excellent
>> work.
>>
>> Jouni
>
>
Robert Hyatt
Hal Bogner (h...@netcom.com) wrote:
: One thing which is not mentioned explicitly in this post, and should be, is
: from sale of Fritz and Chessbase, and not from competing products.
Also, something from personal experience, is that Fritz is *not* crap. It's
a dangerous program (see Destroyer on ICC for one instance of it that I see
all the time.) Crafty out-plays it positionally quite often, but Fritz is
always ready to demonstrate it's tactical ability. Lonnie can give a better
feel for how Crafty does against it, but in my book (which has changed after
playing a lot of games vs fritz and genius) fritz is the better opponent for
Crafty. I wouldn't presume to say that Fritz is better against humans, because
I have no data to go on. But in comparing Crafty vs Fritz (both on P6) and
Crafty vs Genius (ditto) fritz is clearly better. Lonnie has reached the
same conclusion as I recall from comments. Better not hang anything when
Fritz is lookin'....
: In article <83597876...@cpsoft.demon.co.uk> Chris Whittington <chr...@cpsoft.demon.co.uk> writes:
: >Jouni Uski <Jouni...@nce.nokia.com> wrote:
: >>
: >> One important thing to know: as we can read from SSDF:s club magazine "PLY":
: >> Chessbase has never wanted to help SSDF testing - they haven't send Fritz3 and
: >> Fritz4 to Sweden (as other programmers)! Consequence: ratings are delayed with
: >> many months.
: >> And now without SSDF we get (from CSS) conclusion that Fritz4 is a miracle, when
: >> the truth is that it is CRAP. Let's hope that SSDF will never stop their excellent
: >> work.
: >>
: >> Jouni
: >
: >
: >Danger: can of worms being opened ....
: >
: >Jouni is quite correct, but: it is not right to say it is crap.
: >Frans Morsch is a very good programmer, and Fritz is a strong program.
:
: