Leela-Stockfish "no opening book" match results.

[jimgl...@gmail.com]

For all those who want to see what a "no opening book" match between Leela and Stockfish would have looked like, at least as to the opening selections, here's the result of the first 80 games in the 100-game match now in progress at TCEC.  (Much faster time control than in the SuFi.)

https://tcec.chessdom.com/live.html

Leela as white played 1.e4 all 40 times. Stockfish answered with 27 Sicilians and 13 French defenses.

The resulting opening, and result for Leela.

12 x French defense, Steinitz; +3 =9 

 1 x French defense, Winawer; +1

24 x Sicilian, Najdorf; +3 -2 =19

 2 x Sicilian, Nimzovich-Rossolimo attack; =2

 1 x Sicilian, (unspecified); =1

Stockfish as white played 1 e4 a total of 32 times. Leela responded all 32 times with 1 ... e5.

Fish opened the other eight games with 1 d4. Leela always answered with 1 ... d5.

The resulting opening, and result for Leela:

24 x Ruy Lopez, Berlin defence; =24

 7 x Giuoco Pianissimo; +1 -1 =5

 1 x Giuoco Piano; +1

 6 x QGD, 4.Nf3;   0+ -1 =5

 2 x Queen's pawn game  =2

Totals...

Leela as white: +7 -2 =31

Leela as black: +2 -2 =36

Leela overall +9 -4 =67

[jimgl...@gmail.com]

That was all "just the facts". My subjective opinions...

Compared to a "no book"  match,  the openings used in the SuFi certainly appear to have helped Fish and handicapped Leela..

OTOH, in a no-book match Leela's internalized book of openings learned in 60 million training games gives her a real advantage over a bookless Stockfish.

Of Leela's 5 point lead, 4 of the points came from her going 4-0 in the mere 13 French defense games -- and we know how she loves to play the French.  Suppose a bookless Fish had randomly played it 15 more times? 

This supports the Fish fans' argument that in the no-book match against Alpha0 the latter had a significant advantage in the openings.  See all those QI games.

So book helps one, no-book helps the other, have fun solving that conundrum for TCEC 15.

Personally, it doesn't bother me.   Stockfish and the Leela Zero project have very different purposes, the respective programs are relatively best at very different things, and TCEC's purpose is not to name the "best" chess program.   It's a fun and enlightening event.  We all know the "best" competitor doesn't always always Wimbledon, the Super Bowl, the World Series, or Wrestlemania, but the competition is great to watch.

Also personally, I'd like to see TCEC and Chess.com run fewer of these same tourneys that they repeat over and over, just at different time controls, and more special event tourneys where openings and other parameters are *really* selected.  

How about having programs with different styles replay famous past world championship games, and past 'immortal' games, and see what new annotations they might come up with?   They could be run in gauntlet style as a competition to create competitive rooting interest.

Or, how about theme tournaments, as were popular in olden days?   A "wild gambits" tourney, or special tourneys on the Evans Gambit, QP counter gambits, Latvian gambit, whatever -- the possibilities are endless.   These could be very entertaining -- and valuably add to opening theory too.

I mean, CCCC 6 is fine and all, but didn't we just finish CCCC 5?  How about something fun and different?

One person's opinion, FWIW. 

[NuclearPawn]

Utilizing opening books doesn't favor SF. It give both engines the same opportunity to show which is superior in different situations. And SuFi14 featured SF as marginably better than Leela out of 50 openings.

[Dustin Irwin]

Thanks for the summary. It’s very interesting to contrast the results in the different scenarios.

Caution against conflating “no opening book” and long forced opening play a la SUFI that sometimes featured 15 or 20 ply exits. We can give SF an opening book while letting engines play from the standard position.

Lc0 would likely still win.

> On Mar 3, 2019, at 3:31 PM, NuclearPawn <marku.b...@gmail.com> wrote:
>
> Utilizing opening books doesn't favor SF. It give both engines the same opportunity to show which is superior in different situations. And SuFi14 featured SF as marginably better than Leela out of 50 openings.
>

> --
> You received this message because you are subscribed to the Google Groups "LCZero" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/b8320ca7-3d62-4c9e-a791-683806bc33b8%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Jeff Wads]

Just what I and others with experience knew...it would be a bloodbath.  Carry on.  Opening books are a farce.

[Tyler Bischel]

This is very different than the opening choices of AlphaZero, who strongly preferred 1. d4 openings.

[Maciej Żenczykowski]

There's another reason why long openings favour stockfish.

The openings from sufi were sufficiently long that in a sense they get
us out of the opening and into the middle game.
At which point SF's very deep analysis starts reliably hitting TB
endgame positions, while Lc0 is nowhere close.
As soon as you hit TBs your actual positional analysis stops mattering
- you don't need to judge whether a position is good or not - you
simply know.
This happens far far less 10 moves earlier in the game (or with
shorter TC or weaker HW).

You only need 50 different openings to have a 100 game match. There's
20 different opening moves, and 20 different responses... that's
already 400 possible 2 move openings, so you only need to keep 1 in 8.
I'm sure it's possible to come up with 50 reasonable very *short*
openings.

Basically what I'm trying to say is the openings should be *far*
shorter. 3-4 plies tops.
Alternatively HW needs to be downgraded or TC massively decreased.

To a large extent with powerful HW, huge TC, and many forced opening
plies, SF is no longer playing chess, it is simply brute force solving
a mating puzzle - something it happens to be *very* good at, precisely
because of how many nps it can pull off, which is directly related to
how cheap and thus inaccurate it's board analysis is.

Similarly Lc0 shouldn't be trained purely out of start position - to
give it a more varied training experience.

Another thing to consider is whether the sufi 100 games being on the
same hardware and at the same tc even makes sense... You could have
some games faster, some slower, etc. That would presumably add more
variability... etc...

> --
> You received this message because you are subscribed to the Google Groups "LCZero" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/09f7639e-72ca-4aa7-ac06-829aecbec5de%40googlegroups.com.

[Gokhan M]

Good job, Leela! Very impressive so far (+12 -4 =75 w/ LC0 leading).

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/CAHo-OoyNZD_A%2BTKPuaRf_jjkkrkbykOnJeO8K8JZE3%3DG17cuWA%40mail.gmail.com.

[Stephen Timothy McHenry]

Yes, Yes, why not do that. Let them play from standard position, at least some of the games, but give SF all the book it wants, you will see, it won't make a difference - Leela still win that

[jimgl...@gmail.com]

Final result:

Leela 56 Stockfish 44.

Leela +16 -4 =80

[Markus Kohler]

I have the suspicion as well that TB are a significant advantage for SF.one should check the sufi games whether there's correlation.
Also Leelas TB handling has it's problems, there is a ticket (742?).
Also Leelas trolling maybe has an impact after all, because if she would prefer shorter wins, maybe she would hit TB earlier.
And yes training on Monday starting positions makes sense to me. The difficult question to answer is which other positions to use.

[jimgl...@gmail.com]

Write up about the match results and Leela's victory:

http://mytcecexperience.blogspot.com/2019/03/s14-bonus-match-leela-stockfish.html

[Kevin Parent]

Leela is much stronger with 1. d4  vs AB engine, it's sad.

[Jesse Jordache]

Other generations of lc0 also preferred 1.d4, eventually.  Actually the French was the first stop on the evolution of openings for Leela's first network series, eventually moving to e5/c5, finally settling on c4/d4 with the king's knight developed first.

Which is also A0's preference.

[Rajen Gupta]

Great write up. Will follow your site

Raj

[Dennis Tillotson]

I post this as the devils advocate as I am a huge Leela fan. These results represent a huge bias in the match in favor of Lc0. So, I think they are only an indication of what happens when you tilt things in favor of Leela by removing the opening book. The last time I looked at SF code the devs had removed code supporting an engine opening book. (That could have changed.) Which means SF only has heuristic eval code for the middlegame and endgame.  So my opinion is; it's an unfair test. Now if heuristic and/or an engine opening book were added to SF, then you would get a battle worth measuring.

On Sunday, March 3, 2019 at 2:40:42 PM UTC-5, jimgl...@gmail.com wrote:

[Jesse Jordache]

I thought this WAS "stockfish has book, Leela does not".

[Todd Freitag]

Neither had a book.

[mike2...@gmail.com]

Leela essentially has her book built in to her 50MB net file.  It's learned during her millions of training games.... so yeah, she has book.  And SF playing without a book is disadvantage to it - it's actually pretty simple.

[Uri Blass]

It is not so simple.

The question is what is the definition of opening book.

If it is a list of positions and moves to play then Leela does not have a book and the 50MB net file is not a list of positions and moves.

I agree that it is better if stockfish can also use a file with the same size like Leela(I am not sure that opening book is the best way to use it)

Maybe the file can include information in the hash tables of move and score and depth that the engine can use in the search for pruning(for example no need to search 1.e4 for depth 30 if you already memorize a score for it at depth 40 in the hash tables)

[Shah]

It would be equally correct to say that sf also has a book, hard coded to it in its algorithms, which encapulate all human chess insight and knowledge, part of this knowledge is related to book openings.
But, it was also suggested before to make a match where sf has the ordinary book, while leela has none. (Or if you insist, leela has her book encoded into her network)
It was also suggested that leela can prepare her own small 4-ply book before the match and bring it with her...
Which to me sounds fair.
Anyhow, the issue here to me is fundamental. how, if at all possible, could these two engines be compared in any significant manner?

[jimgl...@gmail.com]

Here's my proposal to try to resolve this conundrum to produce a "fair fight" - to the extent possible.

* Leela effectively has a limited book in her learning from millions of self-play games. Stockfish has no book. So when the opening is one Leela has "studied" she has a real advantage. And if allowed she will play to these lines.

See her domination of Fish in French defense games in the no-book match, and also her big +16 -4 victory there overall.

* Leela lacks opening knowledge in lines she "doesn't like" and so hasn't "studied deeply" via self play. So in random openings that she hasn't studied the advantage goes to Stockfish as the superior power calculation machine. See the SuFi games and final result.

Proposal: Give Stockfish a full opening book, use it with some random variation so the same lines aren't repeated endlessly. Let Leela play only the lines she prefers and knows well.

This would be like a real tourney among living people. The other guy plays a book opening, you respond with your favorite prepared defense. You open with your favorite studied line, the other guy uses a book response. That's fair, and how real people play.

In contrast, the SuFi was run as if a tourney arbiter imposed (sometimes crazy) random starting positions on each human player at the start of each game. What human would want to play like that? And the way the programs function, this clearly favors Stockfish as the better raw calculation machine. Just as 'no book at all' favors Leela with her superior opening positional judgement in her favored lines she will play to.

[Friday9i]

Other proposition:
1) train LC with all classical openings (forcing it to play them regularly in selfplay training games)
2) in tournaments, let non-NN engines use an opening book (after the forced openings decided for the tournament) and don't let LC use one
That should be fair, as LC is trained for openings and other engines have an opening book 😁

[Pin Yi]

I cannot agree with you. There are only 3 type positions in chess white win/draw/ white lose. For engine, stronger engine recognize the exact evaluation on those positions more often. Any misevaluation will lead to lose at least half point. TCEC give both engines 50 positions to find out the right evaluation.the engine evaluate more accurately win the match. You can only argue about 50 samples is not large enough. The first +10 with unlimited rounds will be more sound. But opening itself shouldn't be the problem.

[jimgl...@gmail.com]

"But opening itself shouldn't be the problem."

If one thinks the opening doesn't matter just look at the different results in the 100-game book and no-book matches.

[Dennis Tillotson]

I would like to expand on what I posted earlier. One of the reasons I started following TCEC was the fact that there was a tournament opening book and no engine opening books. Which introduced fairness. The battle was shifted from who had the best opening book to the middle game. This made it so more engines could compete, because they didn't have to have the development resources for an opening book. In addition, it solves the problem of repetitive opening play, so we don't see the same games repeated over, and over, and over with the same result. Having the tournament opening book has more pluses than minuses. And to overcome the downside of an engine advantage in a position both engines get to play from the same position as white. 

 

The only reason for wanting to change and remove the book is if you think your engine has an algorithmic advantage in the opening. And it makes me wonder if in the match A0 vs SF 8 a tournament opening book was used, or if it was a biased tournament without it.

My prediction if TCEC removes the tournament opening book is, engines developers will want to be configured with an opening book, or they will have an internal opening book and improved opening heuristic code. The AB engine internal opening book will steer the game into the positions Leela hates, and into tactical positions she can't handle. In the long run, removing the tournament opening book just expands the area of battle back up into the opening, but whether or not Leela is really better in the opening is still questionable.

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/0c0f21ea-dd38-420a-b6c1-8320db4e751b%40googlegroups.com.

[Shah]

Hi Dennis, I think you are referring actually to two different things.
one thing is making tournaments interesting for the audience, the other is being able to answer the question which engine is better.
both are important in my view.
for the first, using a book might be vital.
for the second aim, my understanding is that using a book creates a bias in favor of the ab engine.
you are predicting that “AB engine internal opening book will steer the game into the positions Leela hates, and into tactical positions she can't handle.”
that is possible, but why don’t you think leela will be able to steer the game into her comfort zone?
I am talking about tournament rules where the ab has access to book, whether internally or externally, and where leela doesn’t have any book.

[Sam Jukes]

I agree. I would love to see Lc0 vs SF10 where Lc0 has no book and SF has its own opening book. Let both engines play to their best. 

[mike2...@gmail.com]

Well said, jimgl...@gmail.com.

[Markus Kohler]

Not sure that would work, because A/b engines would steer into positions, where material must be exchanged and then steering towards TB as fast as possible. Would not be interesting I guess.

[smashu2]

You want to compete in tcec you play tcec rule and stfu.

[Shah]

It is actually the other way around.
If you want to attract spectators, you (tcec) change the rules to make the match as interesting as it can be... and interesting if fair...
But anyhow, this is not about tcec imo.
This is a general theoretical discussion about how to compare those engines...