True playing strength of programs
Amir Ban
Robert Hyatt wrote:
> Amir Ban (ami...@m-sys.com) wrote:
: Robert Hyatt wrote:
: >
: > I've never claimed any more. :) But I would go much further:
: > I'd claim that *no* program running on a PC is above 2400. *none*
: > Nada. zilch... etc...
: >
: > I'm not counting the DB series, of course... *non-PC*... :)
: Bob,
: I beg to differ with that rather strongly. It's really too far from
what
: I know from first hand experience.
: Junior is certainly above 2400, and since we know something about how
it
: compares to other PC programs, it is not the only one.
: We know this since Junior competes in tournaments fairly regularly,
: though not very frequently, over the past years, and has a rating.
This
: enables us to place ourselves in the ranks with great confidence. This
: opinion is not only our own, but belongs also to our opponents and to
: many chess masters that have watched many tournament games played by
: Junior, and have told us their estimate of the program's strength.
Some
: place it above 2500, which may be too high, but I have not heard an
: estimate below 2400 for a long time.
>One key statement above. "though not very frequently." We found that
>Cray Blitz was a killer in human tournaments. It was the first computer
>to ever win an open state championship event with at least one master
>playing. But it won based on "computer shock." As you have seen, on
>ICC things are different. The humans aren't "shocked." Most are
>"prepared". And after you play a few games, they get over the initial
>shock and confusion and figure out what you can and can't do, and then
>it is over, unless you continually tune, or learn, or something...
: The evidence ? Here:
: - Junior latest rating is 2349 ELO (Israel rating, on a par with
FIDE).
>That is right in the middle of the band where I think programs belong,
>by the way...
: - This rating is certainly deflated, since the program was initially
: ranked in the 1900's in early 1994. It takes a whole lot of games to
: advance from there up to 2400. The rating does not include a single
game
: played with better than P5/133. It does include quite a few games
played
: with 486/33.
: - Junior did not concede any loss or draw to anyone under 2300 in the
: last two years.
>Including on the chess servers? I've seen ban lose to 2100 players.
>And that is 2100 USCF (tim), not to FIDE titled players. A 2400-2500 player
>simply is not going to lose more than 1 out of 100 games to players
>rated that low. "Ban" did it. "Crafty" has done it. "Ferret" has
>done it. And so has every other program I've seen play...
: - As to players above 2300, we played 15 games with them in the last
: year, in 4 different events, all of them dead-serious full
time-control
: competitions played for rating points, money and glory. Our score is 7
: wins, 4 draws, 4 losses. 3 of the losses are to GM's. The annual TPR
is
: above 2500. All these games were played on P5/133.
>I've seen similar results. Crafty finished ahead of 5 GM players in the
>great GM vs computer event on chess.net... Crafty won the Pan Am event
>last year, one computer, many humans. But the truth remains, it is *not*
>a 2500 player. *not* close. We may simply have to agree to disagree,
>but I don't buy the idea of a PC GM just yet. I know too many GM players,
>and the weakest of them knows more about chess than the best computer
>programs, by a *huge* margin. Pop over to chess.net and tackle Roman a
>few times. Longer time controls or shorter, your choice. He can be a
>killer. Pop over to ICC and play IM Orlov. Or "greg1"... then you'll
>see what I mean. Greg1 *was* a 2500 USCF player years ago. He is a
>computer-killer now.
Your arguments are confusing to me. I thought you were making the point
that all ratings based on special events such as Aegon, SSDF, ICC should
not be taken seriously. I was quoting something different: A long
playing record in regular tournaments under normal conditions, rated by
an official body who does not make any special provisions for you,
without the benefit of choosing your opponents or basing your rating on
the results and theories that suit you. If you won't buy that, then what
is it that you are prepared to accept ?
Strange that you refute my official-play results with ICC results, with
its chess cafe atmosphere. I'm sure an ICC rating of over 2700 means
something, but I don't know what that may be. That's a good reason to
quote results that have an unequivocal meaning, and I thought I was
doing that. I never said that I cannot lose to under 2300's, except that
it does not happen for a long time in official play, and this alone
seems to point rather strongly that the winner must be significantly
over 2300 himself.
Your point about the computer having a "human killer" effect that wears
out through practice is something I don't acknowledge as actually
happening. If it were true you would expect our performance to be very
unstable, beating a 2600 player and losing the next round to a
better-prepared 2200. You would also expect our rating to slip badly,
now that we are playing for 4 years and becoming familiar to almost
everyone. We see nothing of the sort. The performance is increasing
gradually and stably, which makes the rating really credible and
meaningful. In the past, we played many friendly games with GM Ronen
Har-Zvi, for example, and he would consistently murder us. Nowadays, he
has real problems against us. What's the conclusion except that we used
to be much weaker than him and this is no longer true ?
: Not implying any parity here, but is Junior's tournament record really
: inferior to DT/DB's ? Remember USCF ratings should be deflated. I
can't
: really claim parity even if I wanted to, because data on DT/DB's real
: playing strength is next to non-existent.
>USCF ratings are actually FIDE+50, roughly, at the upper end. That is
>a very narrow difference. But yes, DB's tournament record is much
>better. First, they have played 12 games against Kasparov, at tournament
>time controls. They have played several other GM players in exhibitions.
>They played enough games to satisfy the Fredkin requirement for GM-level
>rating (2550+ if I remember the requirement for stage II).
So DT's famous USCF rating was only a hypothetical one based on
exhibition events ? Ouch. There goes my last illusion about DB/DT. I
also have played exhibition games and matches against GMs Liss,
J.Polgar, Yudasin, and that's not counting blitz events. Unfortunately,
only rated events get rated.
Was DT introduced at a rating of 1900 ? If it had, it would need to play
hundreds of games to get to a 2550 level. If I wanted, I could make a
strong case for a 2600 rating: Just discard my pre-1996 results, add the
exhibition games, make adjustments for a PP200, and there you have it.
Is the Fredkin committee still convened ? I don't do that. Making
theories about ratings is a pleasant but idle pastime. But at least
concede me the rating I acquired officially.
Amir
Robert Hyatt
Amir Ban (ami...@m-sys.com) wrote:
: Your arguments are confusing to me. I thought you were making the point
: that all ratings based on special events such as Aegon, SSDF, ICC should
: not be taken seriously. I was quoting something different: A long
: playing record in regular tournaments under normal conditions, rated by
: an official body who does not make any special provisions for you,
: without the benefit of choosing your opponents or basing your rating on
: the results and theories that suit you. If you won't buy that, then what
: is it that you are prepared to accept ?
A rating over time, is very good. if and only if it is based on a lot
of games in events where the players get to see it. If you do as we did
with Cray Blitz, and play 5 games this year, 7 games next year, you are
still taking advantage of "computer shock" and will have a grossly
inflated rating. The servers are much better because everyone can
play you, watch you play, and pick up on what you do well and what
you don't... just like Kasparov has to deal with since everyone is
looking at his games under a microscope.
I'm skeptical of ratings that are long term, but based on a few
games per year like Cray Blitz did. That sounds like what you have
done as well. And it obviously doesn't mean you play poorly. But
it likely means you win games that you would not normally...
: Strange that you refute my official-play results with ICC results, with
: its chess cafe atmosphere. I'm sure an ICC rating of over 2700 means
: something, but I don't know what that may be. That's a good reason to
: quote results that have an unequivocal meaning, and I thought I was
: doing that. I never said that I cannot lose to under 2300's, except that
: it does not happen for a long time in official play, and this alone
: seems to point rather strongly that the winner must be significantly
: over 2300 himself.
I don't count ICC ratings as meaning much number-wise... but if you look
at the pool of players, statistically a rating of > 2700 does mean a program
is playing quite well. But it does not come close to meaning a "FIDE
2700" rating. I still think Crafty would be able to support a 2350 rating
playing in every FIDE event that is held. But I don't think it could do
a lot better, although there would obviously be spikes over 2500 (TPR)
and there would be an occasional dip below 2200...
: Your point about the computer having a "human killer" effect that wears
: out through practice is something I don't acknowledge as actually
: happening. If it were true you would expect our performance to be very
: unstable, beating a 2600 player and losing the next round to a
: better-prepared 2200. You would also expect our rating to slip badly,
: now that we are playing for 4 years and becoming familiar to almost
: everyone. We see nothing of the sort. The performance is increasing
: gradually and stably, which makes the rating really credible and
: meaningful. In the past, we played many friendly games with GM Ronen
: Har-Zvi, for example, and he would consistently murder us. Nowadays, he
: has real problems against us. What's the conclusion except that we used
: to be much weaker than him and this is no longer true ?
But when you say "becoming familiar" what does that mean? hundreds of
games a year? 10's of games? A few per year. That makes a *huge*
difference, since every year the program is significantly different in
how it plays...
We were much weaker. And most are very strong now. But *not* GM.
and not even "close" to GM yet. Grab yourself an IM or GM and talk
to him at length, and he can explain exactly why computers don't cut
the mustard just yet, although they are getting *very* dangerous,
even against GM players...
: : Not implying any parity here, but is Junior's tournament record really
: : inferior to DT/DB's ? Remember USCF ratings should be deflated. I
: can't
: : really claim parity even if I wanted to, because data on DT/DB's real
: : playing strength is next to non-existent.
: >USCF ratings are actually FIDE+50, roughly, at the upper end. That is
: >a very narrow difference. But yes, DB's tournament record is much
: >better. First, they have played 12 games against Kasparov, at tournament
: >time controls. They have played several other GM players in exhibitions.
: >They played enough games to satisfy the Fredkin requirement for GM-level
: >rating (2550+ if I remember the requirement for stage II).
: So DT's famous USCF rating was only a hypothetical one based on
: exhibition events ? Ouch. There goes my last illusion about DB/DT. I
: also have played exhibition games and matches against GMs Liss,
: J.Polgar, Yudasin, and that's not counting blitz events. Unfortunately,
: only rated events get rated.
The "2550" was a "FIDE" rating. But since there is no way for a computer
to get a FIDE rating, they used FIDE calculations, in many matches, which
is the only way to pull this off. That met the Fredkin criteria...
: Was DT introduced at a rating of 1900 ? If it had, it would need to play
: hundreds of games to get to a 2550 level. If I wanted, I could make a
: strong case for a 2600 rating: Just discard my pre-1996 results, add the
: exhibition games, make adjustments for a PP200, and there you have it.
: Is the Fredkin committee still convened ? I don't do that. Making
: theories about ratings is a pleasant but idle pastime. But at least
: concede me the rating I acquired officially.
I do, just as surely as I concede Cray Blitz's actual > 2200 rating in
1981 when it was thought to be impossible. But computer shock was a
big issue...
Dan Thies
On 28 Aug 1997 19:05:11 GMT, hy...@crafty.cis.uab.edu (Robert Hyatt)
wrote:
>I'm skeptical of ratings that are long term, but based on a few
>games per year like Cray Blitz did. That sounds like what you have
>done as well. And it obviously doesn't mean you play poorly. But
>it likely means you win games that you would not normally...
In further support of this idea.....
When I first trotted KC out at the club, it beat everyone.
First night, performance rating about 2200 USCF.
One week later, about 2150.
Two weeks later, about 2000.
Last few times, stable at about 1900, and I finally got some useful
losses to learn from.
If I'd brought it out every three months to play in a quad, it would
have a USCF OTB rating about 2100-2200, which would be grossly
overrated. I brought it over twice a week for a month, and the
homeboys learned what was wrong with it.
Familiarity breeds accuracy.
Dan
Robert Hyatt
Dan Thies (rt...@wt.net) wrote:
: On 28 Aug 1997 19:05:11 GMT, hy...@crafty.cis.uab.edu (Robert Hyatt)
: wrote:
: >I'm skeptical of ratings that are long term, but based on a few
: >games per year like Cray Blitz did. That sounds like what you have
: >done as well. And it obviously doesn't mean you play poorly. But
: >it likely means you win games that you would not normally...
: In further support of this idea.....
: When I first trotted KC out at the club, it beat everyone.
: First night, performance rating about 2200 USCF.
: One week later, about 2150.
: Two weeks later, about 2000.
: Last few times, stable at about 1900, and I finally got some useful
: losses to learn from.
: If I'd brought it out every three months to play in a quad, it would
: have a USCF OTB rating about 2100-2200, which would be grossly
: overrated. I brought it over twice a week for a month, and the
: homeboys learned what was wrong with it.
: Familiarity breeds accuracy.
: Dan
That is a typical scenario. This is often called "computer shock" and
happens when a chess program is first seen in a particular location, or
by a particular player or group of players. But over time, the shock
wears off, and the humans learn how to shred the program. This is the
reason we see very high performance ratings for computer programs, but
no real USCF ratings above 2200-2300 for chess programs (excepting DB).
Dan Thies
On 5 Sep 1997 04:27:28 GMT, hy...@crafty.cis.uab.edu (Robert Hyatt)
wrote:
>Dan Thies (rt...@wt.net) wrote:
>: When I first trotted KC out at the club, it beat everyone.
>: First night, performance rating about 2200 USCF.
>: One week later, about 2150.
>: Two weeks later, about 2000.
>: Last few times, stable at about 1900, and I finally got some useful
>: losses to learn from.
>
>: If I'd brought it out every three months to play in a quad, it would
>: have a USCF OTB rating about 2100-2200, which would be grossly
>: overrated. I brought it over twice a week for a month, and the
>: homeboys learned what was wrong with it.
>
>: Familiarity breeds accuracy.
>
>: Dan
>
>That is a typical scenario. This is often called "computer shock" and
>happens when a chess program is first seen in a particular location, or
>by a particular player or group of players. But over time, the shock
>wears off, and the humans learn how to shred the program. This is the
>reason we see very high performance ratings for computer programs, but
>no real USCF ratings above 2200-2300 for chess programs (excepting DB).
The "shock factor" for KC is a lot higher, too. At first, they all
said "it's a computer, don't open the position." Once they realized
that KC was beating them positionally in closed games, they started to
play "normal" chess and beat the tar out of it. This is an unusual
situation, where the program has a hard time keeping up tactically
with the better club players, since they play speed chess all the
time, but they rarely understand something as simple as center
control.
If you could sell a program on the basis of a USCF rating, which only
takes ten games, then playing in 3 quads in 3 cities might be the best
way to do it, as long as you never play it in the same place twice.
This is probably how Fidelity used to do it....
Dan