Snowie 4 vs. GNU - 100 matches
Albert Silver
I just finished a series of 100 7-point matches between Snowie 4 and
GNU 0.13 in order to see how they did. The series was played by Tony
Lezard's program Dueller 2.1
(http://www.jobstream.com/~tony/backgammon/) and finished 56-44 in
GNU's favor. Though it is a very nice victory for GNU, analysis (see
below) shows that the two programs are actually of exactly the same
strength.
The conditions were Snowie 4 playing at 3-ply Precise using the larger
bearin DB from the CD, whereas GNU was simply playing at Supremo (its
equivalent of Snowie's 3-ply as GNU starts counting at 0-ply) and
using the Snowie match equity table. The reason for this last detail
is so that any disagreement on doubling decisions is due to different
evaluations and not the table used. Snowie 4 rolled the dice in all
the matches.
Any wishing to receive the matches (in .mat format) need only ask,
though I expect they will no doubt be available from Tony's site very
soon. The zipped file is a mere 200k BTW.
Albert Silver
Joern Thyssen, one of the principal authors of GNU, and certainly the
most active contributor toward the GUI development, as well as cube
and match formulae, analyzed the series of 100 7-point matches
factoring in the luck factor. The result was a very interesting tie:
On Mon, Jun 09, 2003 at 01:06:40PM -0300, Albert Silver wrote
>
> > The average of the luck adjustment values (for all 100 matches) is
> > 0.500, i.e., the bots are equally good.
>
> Is that official or are you just saying this out of argument?
It's official.
> If it IS offical could you post some numbers (such as how this was
> reached) so that I might share this with GOL and RGB readers?
For each match I analysed the luck at 0-ply:
set priority idle
import mat ...
set analysis cube off
set analysis moves off
analyse match
show stat match
I extracted the total luck rate from the match statistics
Player Snowie4 gnubg
Luck rate (total) -1.501 (-49.717%) -0.341 (
-1.415%)
In the example above gnubg won, so the luck adjusted result is
100% - 1.415% - ( -49.717% ) = 51.698%
I averaged all the luck adjusted results: 49.9947% with standard
deviation 13.5%, hence the 95% confidence interval is 50.0% +/-2.7%
(13.5% * 1.96/sqrt(100) = 2.7%).
I've attached a file which has the following entries:
game number, actual result, gnubg luck, snowie luck, luck adjusted
result
For example, for game 100 (my example above):
100 1 -.01415 -.49717 .51698
Some of the games could be very interesting to inspect carefully. For
bots of similar strength we expect luck adjusted results around 50%.
However, this is not always true in the 100 match sample you've sent
me:
Examples:
8 0 .19603 .19853 .00250
39 1 .07856 .00087 .92231
Either gnubg's luck analysis is totally wrong or snowie (gnubg) played
very bad in game 39 (game 8).
Match 39 re-analysed:
0-ply: 1 .07856 .00087 .92231
1-ply: 1 .1150 -.0151 .87449
2-ply: painfully slow; I gave up
The result is changed by 5%, but we're still far from a luck adjusted
result of 50%. I can't explain this...
Jørn
_______________________________________________
Bug-gnubg mailing list
Bug-...@gnu.org http://mail.gnu.org/mailman/listinfo/bug-gnubg
Mogath3
>
>I just finished a series of 100 7-point matches between Snowie 4 and
>GNU 0.13 in order to see how they did. The series was played by Tony
>Lezard's program Dueller 2.1
>(http://www.jobstream.com/~tony/backgammon/) and finished 56-44 in
>GNU's favor. Though it is a very nice victory for GNU, analysis (see
>below) shows that the two programs are actually of exactly the same
>strength.
This is ESPECIALLY wonderful for me. Now this proves that you don't have to buy
the RIDICULOUSLY over priced Snowie (wouldn't anyway) when GNU is just as good
(as I have been saying all along.)
Regards,
Jeff
Kees van den Doel
Mogath3 <mog...@aol.com> wrote:
He he he, and those victims that have put down $400 for it must feel
especially sheepish now.
Kees (I try how they religiously clain superiority of laat harde hand
who spilled it possibly because Israel voor Engels?)
Jørn Thyssen
> In article <20030616123138...@mb-m12.aol.com>,
> Mogath3 <mog...@aol.com> wrote:
>
>>This is ESPECIALLY wonderful for me. Now this proves that you don't
>>have to buy the RIDICULOUSLY over priced Snowie (wouldn't anyway) when
>>GNU is just as good (as I have been saying all along.)
>
>
> He he he, and those victims that have put down $400 for it must feel
> especially sheepish now.
Not necesarily. One thing is the strength of the program; another thing
is the usability, where Snowie might be ahead of gnubg.
Also, there may still be positions where Snowie is stronger than gnubg
and vice versa.
Jørn
Mogath3
>is the usability, where Snowie might be ahead of gnubg.
>
>Also, there may still be positions where Snowie is stronger than gnubg
>and vice versa.
>
I would take GNU over Snowie ANY day.
Regards,
Jeff
Albert Silver
> Kees van den Doel wrote:
> > In article <20030616123138...@mb-m12.aol.com>,
> > Mogath3 <mog...@aol.com> wrote:
> >
> [snip]
> >>This is ESPECIALLY wonderful for me. Now this proves that you don't
> >>have to buy the RIDICULOUSLY over priced Snowie (wouldn't anyway) when
> >>GNU is just as good (as I have been saying all along.)
> >
> >
> > He he he, and those victims that have put down $400 for it must feel
> > especially sheepish now.
>
> Not necesarily. One thing is the strength of the program; another thing
> is the usability, where Snowie might be ahead of gnubg.
Yes, there are a few things I prefer about Snowie, I'll gladly admit,
and thus constitute part of a wishlist for GNU:
- A unified GUI. This is by far the biggest issue IMHO. I multitask a
lot, and never only have GNU running. This means that everytime I want
to switch from GNU while it is analyzing a move deeper (such as on
3-ply or 4-ply), running a rollout, or simply comparing its choice
with Snowie's analysis, I have to individually click on 3-4 windows
every time. If I'm posting some analyisis here or somewhere else such
as the GOL forum, it's the same. And finally, even using the most
efficient scheme possible in a 1024x768 resolution (the highest
standard resolution), they always overlap some. I never see the text
annotations of course since they are permanently hidden below, and the
score and pip-count are covered by the annotation window. Don't get me
wrong, I love GNU as is well-known, but that much is a major pain in
the neck. That said, it isn't worth Snowie's $380 pricetag to my eyes.
I actually thought of trying my own hand at learning GTK to try to
implement this, however crudely, but found the initial documentation
dismaying. I don't have previous programming experience (unless you
can call some light 15-year-old knowledge of Pascal for the Apple II
as such) so its initial statement that it presumes a working knowledge
of C++ on the reader was a turn-off.
- Its graphs of the stats. I initially thought this was nothing much,
but after using it must agree it is a cool feature. It helps quickly
highlight what games were the source of major errors by each player
and gives one a better idea of one's error rate. For example, if it
said I had a 5.0 error rate in a 6-game match I could see whether I
played that consistently or whether I did better with a 2.0 error rate
spoiled by a huge blunder in one game.
- The ability to simply modify the position from a match I'm looking
at and analyze it. Thus I could see an error highlighted and move a
checker or two to see whether this would make a difference. This is a
very important learning tool. Snowie's way of handling this is to
simply open a new board window with the same position and edit buttons
activated at the top. One can then set the match length or money,
score, etc, and hit the analyze buttons. When done, either save it, or
close it and return to the initial window with the match itself. Two
things that really suck (IMHO) in Snowie about this function is the
need to go to this mode to save a position from a match, and the need
to go to the top and click on the Window menu to switch from one to
the next. There is a huge amount of hidden and wasted real-estate in
Snowie, and here they could simply have added some tabs to switch back
and forth (like Excel), but whatever.
- I like its export to RTF quite a bit. It is very elegant, but of
course they have a proprietary font that allows this.
- And finally, the ability to run batch analyses of matches or batch
rollouts.
-----------------------------------------------------
GNU however has MANY other functions and possibilities that outshine
Snowie, so let's not think it's a one-sided story. Hell no!
- GNU's rollouts are second to NONE! There is simply no comparison.
a) First of all, GNU's 0-ply can only be compared to Snowie 4's 2-ply
since the 1-ply would be far too outclassed. That means it is far
faster to achieve meaningful results.
b) It can also play according to score in the rollouts, which Snowie 4
cannot (I'll never get over that).
c) It has tons more features to control the rollout conditions, such
as move filters, speed in checker play AND cube, different strengths
for each, truncated LEVELS such as 2-ply for the first 6 plies, and
then 0-ply for the rest, etc.
- The newly implemented Temperature Map is just great. It helps
understand the dangers of moves and the volatility of cube decisions
in a striking visual way. A huge kudos.
- Imports the known game formats (including TrueMoneygames), AND keeps
the text annotations from Gamesgrid matches, AND *exports* to more
than one format.
- It has customizable graphics for the HTML exports, not to mention
other formats.
- It works on more than 1 OS!!! It runs on Linux, MacOS, and a bunch
of others as well.
- It has direct copy to clipboard functions such as the Gammonline
HTML, but also text and ASCII for users such as RGB readers and others
sending positions via e-mail. To do this in Snowie, you need to save
it as a .txt file then open it, then copy it and then paste. Not so in
GNU.
- Use any Match Equity Table you want and even create and use your
own. I can use Woolsey's, Jacob's and Trice's, or even Snowie's for
example.
- A Tutor mode to warn you while you play of a mistake about to be
made and let you reconsider.
Not to mention super responsive support, a great tutorial (cough!
cough!), and a monster of an engine. I just can't get over how strong
GNU has become. This thread started by stating that GNU 0.13 and
Snowie 4 are of equal strength. While absolutely true, what wasn't
mentioned is that a couple of days later, GNU 0.14 was released that
is 1.18% *stronger* than GNU 0.13.
Albert Silver
kiwi
been luckier than snowie if not the result could not be 56/44. So am i
right if i postulate that gnu's luckrating must be 600% higher than
snowie's over this 100 matches.
silver...@hotmail.com (Albert Silver) wrote in message news:<f9846eb9.03060...@posting.google.com>...
GeoffArnold
Hello There..........
I have both snowie and gnu.. but only ver 0.13 of gnu. My question is
how does one just update to ver .14 of gnu without downloading the
whole setup.exe file again?? I am clueless when it comes to this type
of thing .. Any help would be appreciated.
Thanks
Geoff
Christian Tagsold
the graphics are much better. it is much more educational to view a
match with snowie as you can highlight all types of errors in the way
you choose to. sounds simple and naive but having an error orangend
and in one window with the actual postion works much better for me
than the click back and forth gnu-gui.
i prefer to play gnu as an opponent lately (for no specific reason
though) but analyze the matches with snowie.
as the engine itself is now not worse than snowie and maybe even a bit
better it might be time for gnu to concentrate on the looks a bit.
with the gnu team having done such a fantastic job up to now i am
confident snowie 4 won't be my first analze-tooll choice for long
anymore :))
cht
Albert Silver
> > Not to mention super responsive support, a great tutorial (cough!
> > cough!), and a monster of an engine. I just can't get over how strong
> > GNU has become. This thread started by stating that GNU 0.13 and
> > Snowie 4 are of equal strength. While absolutely true, what wasn't
> > mentioned is that a couple of days later, GNU 0.14 was released that
> > is 1.18% *stronger* than GNU 0.13.
> >
> > Albert Silver
>
> >
> >
> > >
> > > Also, there may still be positions where Snowie is stronger than gnubg
> > > and vice versa.
> > >
> > > Jørn
>
>
>
>
> Hello There..........
>
> I have both snowie and gnu.. but only ver 0.13 of gnu. My question is
> how does one just update to ver .14 of gnu without downloading the
> whole setup.exe file again?? I am clueless when it comes to this type
> of thing .. Any help would be appreciated.
>
> Thanks
> Geoff
http://users.skynet.be/bk228456/GNUBgW.htm
You need to install the LibArt DLL and the weights. You must also
download the latest build and install it. In order to use it, go to
the gnubg directory, delete the gnubg.exe file, and rename the
gnubg-030614.exe to gnubg.exe. The numbers represent the date of the
build so it changes.
Albert
CuriousFellow
When you say that these files have to be installed, do you mean simply that
they have to be downloaded and the self-extractor run to put them in the
gnubg directory, or once there do I have to do something like use the menu
option File/New/Weights? Is there some way from within gnu to tell what
external files (weights, etc) that the program is using?
"Albert Silver" <silver...@hotmail.com> wrote in message
news:f9846eb9.03061...@posting.google.com...
Albert Silver
> Albert,
>
> When you say that these files have to be installed, do you mean simply that
> they have to be downloaded and the self-extractor run to put them in the
> gnubg directory,
Yes, just run the self-extractor for the three small downloads.
or once there do I have to do something like use the menu
> option File/New/Weights?
No, In order to use the new self-extracted build, go to the gnubg
directory, delete the gnubg.exe file, and rename the gnubg-030614.exe
to gnubg.exe. The numbers represent the date of the build so it
changes.
Is there some way from within gnu to tell what
> external files (weights, etc) that the program is using?
Once you have done the above, you can check in the Help menu by
selecting "About gnubg"
Albert Silver
Back4U2 BBL
"CuriousFellow" wrote
> Is there some way from within gnu to tell what
> external files (weights, etc) that the program is using?
Analyse / Evaluation Engine.
That will list what gnubg is using.
Nardy
