TCEC is... unfair!!

[Shah]

Yeh.. I know.. stop whining you say...
But really:
When you put leela in an already developed position, taken from the book, as they are doing on tcec, you are actually depriving leela from effectivly using her best asset, which is quick and deep strategic moves in the begining of a game. That is where her most strength is.
And thus, thats where her best chance is to gain an advantage that will increase her chances to finally win the game.
I know, the book is used so that the games are more versatile and interesting.
But, the book does create a bias, and it nog in favour of leela.
The chances that a certain opening is exactly in line of what *leela understands* to be a strong positional opening, I beleive, are very small.
Ask this question, how many of the book moves would have leela played?

[Jeff Wads]

Meh, the joke I see is TB wins being draws. Drawn match in my opinion.

[Fahim Saharaiar]

"The chances that a certain opening is exactly in line of what *leela understands* to be a strong positional opening, I beleive, are very small." how do you know that?

And the alternative is to start from the original position, which would make all the 100 games similar. 

[Dustin Irwin]

Sent from my iPhone

On Feb 23, 2019, at 4:10 PM, Vencislav Georgiev <venc...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/df01c941-414f-4351-909e-f2347d79be33%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Volodymyr Kleban]

Opening books are customary for the AB engines, which are bad in openings. Stockfish was further married to advanced opening Cerebellum library to get Brainfish. Yet these modifications are banned from tournaments. I think this is unfair to AB engines, as Leela has its opening book essentially embedded inside, and its sucky endgame play is greatly enchanced by the tablebases.

I think either tablebases have to be banned from the tournaments as well, or opening books (or libraries like Cerebellum) should be allowed. Then it's going to be more level playing field.

[Dustin Irwin]

The alternative to 15 move exits is the standard starting position? Oh my what a bizarre notion.

But not to worry. I don’t think many view the TCEC as the authority on the best chess program. It only reflects the best engine for a set of prepared openings.

On Feb 23, 2019, at 4:06 PM, Fahim Saharaiar <fah...@gmail.com> wrote:

"The chances that a certain opening is exactly in line of what *leela understands* to be a strong positional opening, I beleive, are very small." how do you know that?

And the alternative is to start from the original position, which would make all the 100 games similar. 

--

You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/e91752f1-740d-428d-83b4-61446f6075d8%40googlegroups.com.

[Savickis Matas]

It might be that leela is better at the beginning but both engines had the same conditions. But I somewhat agree with your opinion. I would have been nice to have like 10 game from starting position

[NuclearPawn]

I'm sure TCEC would have been fair had Leela won.
Accepting a loss is a mark of sportsmanship and it doesn't devalue Leela's tremendous development in one year. SF is a project running for 11 years. Your sense if fairness might be strongly biased.

[Savickis Matas]

Yeah I agree about 1 year thing. Give leela few more year and we can organise leela vs alpha zero

[Daniel Rocha]

Well, it's not the case that Leela is even aware that she's playing chess... for now.

Em sáb, 23 de fev de 2019 às 19:18, NuclearPawn <marku.b...@gmail.com> escreveu:

I'm sure TCEC would have been fair had Leela won.
Accepting a loss is a mark of sportsmanship and it doesn't devalue Leela's tremendous development in one year. SF is a project running for 11 years. Your sense if fairness might be strongly biased.

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/f42d7cac-e03d-491e-9ee1-955f23f6bf86%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Daniel Rocha - RJ

danie...@gmail.com

[Shah]

Hey, I didn't say I know that...
And I also suggested a way to verify that.
Simply check how many of the book moves leela would play, even at a very long TC.
Again, I do not know the answer. So this is a question. I only beleive not many moves.

[Shah]

You are saying:

I'm sure TCEC would have been fair had Leela won.

I must be honest and agree with you about that... (-:
And I am also fully aware of the unprecedented acheivement.
Nevertheless, you are not referring to the point...

[jimgl...@gmail.com]

A bigger "imbalance" IMHO is the calculating power of CPU vs GPU. But how to figure what the equivalents would be? And who really had the edge?

Bottom Line: A tremendous achievement for the Leela project and devs, in under a year - remember dead last in Division 4 in TCEC 12?

Kudos to the devs and contributors!

Leela wasn't expected to even be in the final, much less win it. But now she is dead even with Fish, and there are four(!) Leela variants just qualified for the title run at CCCC 6.

Test 40 is maturing and 'monster' Test 60 is on the drawing board.

Looking forward to TCEC 15 and TCEC 16!

[Shah]

A tremendous achievement indeed.
No one can question that!!!

Regarding cpu vs gpu, I once saw a suggestion to force an equal energy consumption... (sounds like a fair deal to me) same quota of electrons for each participant...

[Dustin Irwin]

Maybe same principle for humans and we institute weight classes. Maybe Fabs is the P4P champ!

> --
> You received this message because you are subscribed to the Google Groups "LCZero" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/24935b52-faa0-4bce-83aa-da39cba7572a%40googlegroups.com.

[Michael Elkin]

The only thing that may be unfair is game 66. Because the moves played by Leela after the restart were not as good as before. Game 66 was going into an easy draw before the network outage. Why it was performing differently is unknown, but at least they should have spent more time to have moves consistent with the first game.

Also, I don't know why Leela blundered so bad in 66, or failed to win in 65. My home PC is much slower and without TB and SF 10 solves both games easily. Was Leela performing sub optimally after the restart, but how did it fail so badly in game 65 before the restart?

Either way, this SuFi is by far the best I have ever seen with every game counting, and that is what everybody wanted. Hopefully next season will be even better.

[Stephen Timothy McHenry]

Yes, YES!, why not just a few pure games from the start position without faulty human books?

[jimgl...@gmail.com]

I believe the TCEC people say they are going to do that in SuFi 15, for a few games.

But "no book" is impossible for 100 games.

Remember the Alpha0-Stockfish no-book match in which A0 with its internal trained book beat Fish over and over and over with the same Queen's Indian defense.

Fish's advocates protested that "no book" was unfair because A0 had a full opening book from its millions of training games while Fish had none. That had a fair point - and they'd have same against Leela.

There has to be some way to vary the openings.

[Dustin Irwin]

Giving SF and opening book isn’t the same as forcing 10 or more move openings for each side.

Give SF an opening book and let them at it from the standard position. If that’s not enough for you, you can get plenty of game variation from selective openings of no more than 4 ply. Though really I don’t understand the need for “entertainment” in watching computers compete.

> --
> You received this message because you are subscribed to the Google Groups "LCZero" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/c2fefd97-7b6e-4fb9-a9be-7c2f98118fdb%40googlegroups.com.

[OmenhoteppIV]

Yeah I really love the way Leela invented her own openings, for example her discovery of new variation in Kings Gambit or in Nimzo Indian.. Leela is really good in converting opening to middlegame..
Btw there's nothing we can do.. that's the rules

[Pin Yi]

the most powerful engine should play the best move at any given position. I have been follow the match closely.Clearly stockfish did better in this 100 match. Not only stockfish won little bit more game, but also fish given out more objective evaluation more often. Leela represent herself by accurately evaluated in some type position like close pawn chain French. But her overall accuracy isn't that great.

[Stephen Timothy McHenry]

They are clearly about equal right now. As to the opening book: sure, if SF devs want it to have the faulty, human books then fine, give it all they want when you start at opening position, that's not going to change anything in playing against Leela. It would probably make a difference if SF were allowed to have like a 10 billion full games db at hand to consult so it could avoid all known bad moves, that would make the games be at a very high level indeed.

[jimgl...@gmail.com]

Leela is not a conventional chess engine. In fact, due to the "zero" nature of the project she's not even optimized for chess - lack of endgame training, no broad opening book, etc.

Her tactics are quite weak (compared to the other top engines) as measured by tactical suites.

Her understanding of positions she's trained is clearly superior to that of other engines, which makes up for that very impressively. (Fire's never going to play another French against her.)

But in unfamiliar situations she flounders: her farcial multiple Queen and underpromotion endings, etc.

Also, she is handicapped this way in rare tactical openings she's not trained on: that KGA, and that opening giving that rare Queen versus three pieces middle game, no wonder she lost those.

Remember, she's still very much in "Test" mode - which makes this achievement very much more impressive!

She also looks very much in the mode of Alpha0. Stockfish, the calculation monster, smacked A0 all over the board in their few games where raw tactics came to the fore. But for the most part A0 positionally strangled then executed the Fish.

On to Test 40. And Test 60!

When she gets the full opening and endgame training she needs, all will cower before her.

[Michel VAN DEN BERGH]

On Saturday, February 23, 2019 at 9:03:39 PM UTC+1, Shah wrote:

Yeh.. I know.. stop whining you say...
But really:
When you put leela in an already developed position, taken from the book, as they are doing on tcec, you are actually depriving leela from effectivly using her best asset, which is quick and deep strategic moves in the begining of a game. That is where her most strength is.

Since the only _practical_ use of a chess engine is analysis, it should play good chess in _any_ position. Not just the starting position.

[Shah]

That sounds like a good idea...

1. Give SF an entire opening book.

2. Force only the first 4 plys, to create some diversity still...

3. Nice to have: Let leela create her own opening book (offline) only for those 4-ply. Not sure if this is necessary at all.

[Shah]

You say "the most powerful engine should play the best move at any given position"

That is true, but that is not the question. No one claims that Leela is a perfect chess playing engine.

The question here is I think which engine is "better in playing chess".

This turns out to be a hard question to answer.

But the most accurate and persuading answer I believe, would be given by letting both engines play from the starting position.

Any other attempt might be entertaining, that's for sure, but biased.

Think for example of a chess-puzzle solver engine (e.g. white to move, find a mate in 3...) 

No one doubts SF would outperform Leela in this.

But, does it imply SF is a better chess player? Of course not.

With long forced openings you are doing something like that.

[Arlet]

Best result would be a clear Leela victory in a match where conditions favor SF.  Leela just needs to avoid a couple of silly mistakes to accomplish that. 

[Margus Riimaa]

Sorry about that stupid question, but as I understand Leela gave away a few certain wins in the tournament.
My question is, how big blunders were they - would a human, grandmaster or a hobby player win these positions easily?

[Hope]

It’s hard for me to see Leela’s moves as a built-in opening book as the process for move selection doesn’t change as she evaluates the position and selects what she evaluates as being the best move. With an opening book, she, like SF, would consult a database of predetermined moves and only switch to evaluating positions once out of book. To determine which chess engine is the best, I think you have to compare their ability to evaluate positions on their own (no book or table bases).


On a side note, the hardware used by SF in the superfinals were far more expensive than the hardware used by leela. Makes you wonder what the score would be if they both ran on equipment with similar costs.

[Lothar Jung]

No book, no TB, compareble hardware!

[OmenhoteppIV]

Yeah that's why it's Zero from Leelachesszero because she learned everything from scratch, which means that Leela didn't perform well if we based on human created variations. I agree with you.. I thought of that before but no one is listening 😒

[Yotta Zeta]

It is basically. TCEC can make SF or LC0 win. Give more French openings, then LC0 will win. Give more King-Gambit accepted openings, then SF will win.

The ability to control the openings give too much power to organizers. It should be 1 move or 2 more random openings.

[Sergey Morozov]

Please note that at the time the openings were officially decided the organizers did not expect Leela to play in the Superfinal. This looks unbiased to me.

[Shah]

Looking now at tcec, it seems that they are having leela against sf without an opening book.
It also seems that leela is far ahead after about 40 games. (wins vs 1 win to sf)
I dont know how versatile those games were.
But anyhow, I think that demonstrates well what this post was all about.
Go leela!

[jimgl...@gmail.com]

The problem is that Leela has a sophisticated opening book of her own learned in her millions of training games. If Stockfish has no book that is a major advantage for Leela. Which is what the Fish fans pointed out after its no-book match with Alpha0, and all those Queen's Indian games.

[Hope]

Ya but unlike a real book, Leela’s “book” is just a superior real-time evaluation of the opening much like SF has a superior real-time evaluation of tactics. It sounds more like SF devs decided to rely on a database of predetermined moves rather than developing SF’s ability to evaluate the opening. It would be like in a marathon complaining that one runner is faster than the other at the start of the race, so the slower runner should be allowed to ride in a car for the first mile.

[Shah]

Not sure you can exactly say that leela has her own book. She has her strategy.
But if so, why not make it "formal"?
Let each engine choose half of the openings in a match.
Each opening will be played twice, as in tcec, in each color, so a significant advantage is not a good idea.
Makes sense?

[NuclearPawn]

You Leela fanatics are funny. I love both engines, and I actually want to see them both progress.
The point is, that a chess engine real strength in chess overall is tested through a various range of opening. Naturally the more the better. In TCEC there are 50 openings. Not bad for a 100 sample size. Fair enough.
If you leave it up to engines they'd always follow one path, and yes maybe SF would go for french all the time getting destroyed by Leela. You say that devs should just make it so SF follow book moves and such. But that's not the point.
SF is an engine used by many people especially chess players like me. I don't want an engine that decides it's own opening lines.
Implementation of an opening book in SF might be a piece of cake coding wise, but that's not the point here.
And before you say Leela has better strategy than SF in the opening then you might be right, but the thing is each opening has it's own strategy. So guess what? You need to force engines on different openings to see their real strength overall. Otherwise you'd just some sort of biased games based on engine strengths and weaknesses.
A fair match enforces different openings.
Of course in human games there's no such need for forced openings since players are conscious of their opponent weakness at high levels.

[Hope]

If the purpose is to determine which engine plays better chess then you force both engines to rely solely on their ability generate/evaluate moves from the beginning of the game. Giving access to books and tb helps to hide engine weaknesses and limits the promotion of improvement in areas of weakness.

I want to see improvement in both SF and Leela as competition breeds excellence. To ignore the opening evaluation by forcing a specific opening or using a book doesn't help to improve the engine's ability.

[Dennis Tillotson]

I don't normally comment on anything in the forum, but this issue about opening book gets raised frequently. I believe you are correct that it's unfair to Lc0 to use the tournament opening book, but it's really a bad idea to change the rules. If Leela wins under changed rules SF fans will just be able to claim she won ONLY because the rules changed. It's better to compete and win in the current format to prove dominance. 

[Shah]

Thanks Dennis,

I didn't actually call to change the rules... (-;
I was actually just pointing out this bias issue because it was such a surprising insight for me. (Yes, the title tried to be a little bit provocative... that's all..)

It is also intimately related to an interesting question which also gets raised frequently:
How NN and AB engines could in principle be correctly and accuratly compared?
HW-wise and Tournament-rule-wise.

Btw, personaly I am not sure they can be.
Maybe such a comparison simply has no concrete meaning?
But both engines play chess... even against each other...
That is why these issues are so intersting IMO.

Cheers!

[Dave Whipp]

I think I've noted this before; but the problem here, if that's the right term, is that Leela is being trained to play one game; and then is being measured on a different (but similar) game where pieces have different starting positions. Incorporate a variety of opening-book positions into her training and she'l get better at alternative starting positions (of course, it would need to be a large set of openings ... but across millions of games that's not a problem).

Note that the reason for adding these books is not for "chess 960 improves midgame tactics" reasoning that we sometimes hear; but simply to make the training regime more similar to the evaluation criteria. If she doesn't get to chose the initial move, then it doesn't matter how good she is at choosing it!

--
You received this message because you are subscribed to the Google Groups "LCZero" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lczero+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lczero/ced84814-97e7-4b4d-9f03-713092e8a51d%40googlegroups.com.

[Uri Blass]

I think the most fair tournament is to let both programs to use the same memory in their exe file.

If lc0 can use a big network and Stockfish is not allowed to use a book inside the exe file then it is unfair.

lc0 already learned how to play the opening based on a lot of games against herself  so she already has a file that is equivalent to a file that include evaluations for different positions that she can get in the opening.

[NuclearPawn]

^someone who actually gets it

[michae...@gmail.com]

It's not only a question of whether openings should be used, but also in what way. I can't be sure, but given the almost equal result, I suspect that Leela would be able to win if:

a) Openings were used to provide variety, but for less plies, and not as sharp, or

b) No opening lines were forced but stockfish could use an opening book

[vladimir...@gmail.com]

Same energy consumption, and same RAM and disk memory (sum of executable size and all data files, opening books or network weights). This would be fair.

[Lan DiBart]

Personally ..I think its a bit suspect not in the sense that anything sneaky was happening ONLY in fairness  because leela is a different type of engine for one  but it was agreed to follow those rules .. now heres why I feel those rules are a bit unfair where leela is concerned..She is  self learning, which means to me she avoids games she plays worse , the way a human player would avoid as best a possible games that would not favor them and this starts from the beginning of the game NOT after the opening is done. NOW she is not as good at tactics for sure , so she needs to play away from many types of positions into other  positions that work for her, she will steer her games into them from the start this means from move 1 ..

  many openings are geared toward tactics, so to toss leela into an opening that is more tactical, less positional .. is unfair because she did not go into it willingly as it were ... the same it would be to take a less tactical more positional human player and say defend against this gambit opening  it is equal with great tactical play!  this is not fair .. To me  unless the other program was good enough to be able to steer her there, that program did not earn the position ..

leela she has to be looked at as different because she is.... SO this is where we see her true advantage steering her game to her liking ..its also why when the game become murky and you can shuffle pieces around she seems to me to excel in it .. 

   Changing rules is for leela is not good either, as people would complain .. but she is not the same as all the other engines at the same time so those rules put in place  were because all engines can be tested fairly as possible under those particular rules .. now we have a different engine which rules are less fair for her but she was also put into this competition under the agreement of those rules so there really is nothing to be said about it .. hopefully in the future newer rules can be made that will be fairer .. in any case we have 2 engines stockfish/leela  that if we could ever blend them woow .. hahaha.. 

On Saturday, February 23, 2019 at 1:03:39 PM UTC-7, Shah wrote:

Yeh.. I know.. stop whining you say...
But really:
When you put leela in an already developed position, taken from the book, as they are doing on tcec, you are actually depriving leela from effectivly using her best asset, which is quick and deep strategic moves in the begining of a game. That is where her most strength is.

[Shah]

Hi Dave, 

      

IMHO, the mere fact that the "pieces have different starting position" is not the precise reason that justifies relating to book openings as a "different game"

I think that the starting position (and the first few moves that follows) are indeed inherently different, but in the sense that they are almost purely *strategical* positions.

And in that sense they deserve to be referred to as "a different game", as opposed to midgame, which is not purely strategical.


If I am on the right track here, then I think two consequences could be expected:


1. "Incorporate a variety of opening-book positions into her training", as you said, should not gain Leela decisive advantage in midgame, and hence 
will not significantly change her overall win rate.

This is because of the inherent nature of NN which allows it to excel in understanding a position strategically.

While for analyzing tactical positions you'll still need brute force calculations.

2. If you make a variant to the opening position, but still keep it "strategical", Leela should still maintain the upper hand, as she is now, with no additional training.

For example, handicap, chess 960, and alike. 

(If I'm not mistaken an early 10xxx network with a pawn handicap had once beaten SF in one of the official tournaments bonus rounds)


BTW, for endgame, you might need yet a 3rd different kind of chess playing algorithm, or use a TB,

For not strategy nor even deep brute force will help you win a QR vs R or KB vs K...


I admit that this understanding of mine cannot begin to explain how a 1-node analysis by Leela can possibly play chess effectively, let alone reach mind boggling Elo's in the 2000+ zone, as it is claimed.

This is a puzzle to me.


Cheers!

[Uri Blass]

In case b I do not think that you are right because I read in the past that stockfish can use an opening book that lead to tactical positions when leela does not use book.

http://www.talkchess.com/forum3/viewtopic.php?f=2&t=69175&p=783249#p783249

From the Initial Board position (their dominant opening in tests):

Score of lc0_v191_11261 vs SF10: 179 - 131 - 190 [0.548] 500
Elo difference: 33.46 +/- 24.07
Finished match

SF10 using BookX.bin (it's not an 8-mover, it's a 15MB good quality small, varied polyglot book. Against Lc0, SF usually exits the book in 4-14 moves):

Score of lc0_v191_11261 vs SF10: 98 - 216 - 186 [0.382] 500
Elo difference: -83.57 +/- 24.41
Finished match

note that in all these games lc0 did not use an opening book so it seems using bookX.bin can help stockfish significantly.

[Margus Riimaa]

I read a lot of fair/unfair in this topic, but really there isn't one. It only depends, what you want to measure.
A whole or a part of the system...and what behaviour you want to measure.
It is clear, that the purest measurement of how the engine plays chess is without a book. But it has little to do with, how much it understands it - then you need a big book to try out many different circumstances.
You might want to measure only its endgame capabilities, or problem solving.

Or you might think, lets create a chess entity, like with wrestling 10kg max, or some power consumption max, of course then you would add opening- and endgame tablebases.

Remember that humans do use opening books - masters know many moves from memory.
So a human is a chess entity.

...
So back to the original, it depends, what you want to measure.

But I don't see any of these as fair or not fair.

I personally am interested in a program that understands chess well including my stupid games - to achieve this I imagine one uses a lot of opening books, even weak ones, solve chess puzzles and is able to finish games as well in variety, so NO endgame tablebases. (but i love they exist.)
That would simply be my preference. To have an engine that measures well under those circumstances.

[Patrick Hill]

To me, it makes the most sense to start with the engines playing from the opening and then add openings to avoid boring repetition. TCEC insisting that all superfinal openings need to be at least eight moves is kind of silly. The most fair format that I can come up with is to play a couple of games from the start position, 20 or so games with 1-4ply openings, 20 games with 5-8 ply and so one. That way we would have some diversity of play, but now throw out opening ability entirely.

I rather like watching the games from the start position. On TCEC hardware with long time control, Leela and Stockfish are rewriting and refining opening theory, which I find fascinating.