[Computer-go] CGOS rating has drifted a lot

[Rémi Coulom]

Hi,

I trained a much stronger 19x19 network for Crazy Stone in December, and connected it to CGOS yesterday. It is winning almost 100% of its games against the version I connected 3 years ago, and still has a lower rating. It seems there has been a very big drift of ratings.

Even the Bayeselo list seems to have drifted.

My feeling is that this may be caused by strong programs losing on time against weak programs. It would be interesting to compute a rating list that excludes games lost on time.

I could also reconnect CrazyStone-18.03. I will do it in a few days when the current version will have played enough games to establish its rating.

Rémi

[li...@chinapeek.com]

您好！我是江苏君华特种工程塑料制品有限公司的总经理李军，我的手机号码为13382868677，感谢您发来的邮件！我会尽快处理答复您！
Hello，I am Jolly Li， the general manager of Jiangsu Junhua High Performance Engineering Plastic Products Co.，Ltd. Thanks for your mail ，it is well received，I will get back to you as soon as possible.
_______________________________________________
Computer-go mailing list
Compu...@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[Hiroshi Yamashita]

Hi Remi,

I agree with you losing on time against weak program is bad effect for ratings.
And I think same program matches with different playouts, like 400 playouts vs 800 playouts are worse.
There are too many Leela Zero vs Leela Zero, and recently KataGo vs KataGo.
Maybe for human, current top program rating is not so high.

I will try to calculate BayesElo without losing on time and same program matches before very long.

Thanks,
Hiroshi Yamashita

[Hiroshi Yamashita]

Hi,

Deleting losing on time games, and deleting losing on time games and same program matches are follows.

19x19 BayesElo (deleting losing on time games)
http://www.yss-aya.com/cgos/19x19/bayes_20210109_del_timeout.html
19x19 BayesElo (deleting losing on time games and same program matches)
http://www.yss-aya.com/cgos/19x19/bayes_20210109_del_timeout_same_program_match.html

Top rating and ELFv2 (delete on time)
black_hole_v7 5245
LZ_05db_ELFv2_p800 4005

Top rating and ELFv2 (delete on time and same program matches)
black_hole_v6 4958
LZ_05db_ELFv2_p800 3855

Top rating and ELFv2 (original)
Katago_40b_s37_4V 4762
LZ_05db_ELFv2_p800 3686
http://www.yss-aya.com/cgos/19x19/bayes.html

I had thought top rating of "delete on time and same program matches" was less than original, but it is not.

The matches between these names are deleted.
For example, if black name has "LZ" and white name has "ELF", this game is deleted.

LZ | lz | Leela | ELF
kata | Kata | KG | k40b | k160 | Ka
Zen
CrazyStone | CS
Rnz | rn | Rn
Aya
myCtest
AQ
Dirichlet
CGI
DCNN-
Odin


731369 ... number of games
721122 ... number of games after deleting timeout
592249 ... number of games after deleting timeout and same program matches

[Rémi Coulom]

Hi Hiroshi,

Thanks for your work. In both lists CrazyStone-18.04 is ahead of CrazyStone-81-TitV, which is really completely wrong.

I will reconnect CrazyStone-18.04 soon. It may help to adjust things a little bit.

Rémi

[Rémi Coulom]

Hi Hiroshi,

I tried to compile my old code, but it turns out to be a bit too complicated because it was using tensorflow. I am not using tensorflow any more, and don't want to try to reinstall and recompile it. But I would like to try to take a look at the data. Can I download it anywhere? Can you send the PGN of the 3 lists to me?

Thanks,

Rémi

[Hiroshi Yamashita]

Hi Remi,

This has latest 19x19.pgn and some tools to make bayes.html.

http://www.yss-aya.com/20210112cgos_bin.tar.bz2

I got these tools from Joshua Shriver.

Thanks,
Hiroshi Yamashita

_______________________________________________