Loebner Prize 2006 $2000 and a Bronze Medal
loe...@gmail.com
New York City.
(Contest Management reserves the right to change the venue)
$2000 and a Bronze Medal -
In 2006, $25,000 and the Silver Medal will be at risk.
Important Dates:
June 1, 2006: Opening date for submission of entries.
June 30, 2006: Closing date for entries.
July 30, 2006: Semi-Final 8 announced
Sept 1, 2006: Final Four announced
Oct 1, 2006: Loebner Prize 2006 Contest
The rules for 2006 are substantially the same as for 2005 with the
following exceptions:
a. Final Four entrants will each be awarded a USD 250 stipend. The
winner of the Final Four round will receive the Competition Cash prize
of USD 2000, making a total cash award of USD 2250 to the winner
(unless there is only one compliant entry, in which case the cash
award will be USD 2000.
b. Final Four entrants must be present at the competition.
c. Communications programs will be supplied by contest management.
d. Judges will have 20 minutes to review transcripts of their
interactions before scoring.
e. Judges will have 27 minutes to interact with each pair of entities.
Rules:
Entrants should submit their entries on CD, DVD or USB Flash via a
medium (a) requiring a receipt signature and (b) having a time/date
stamp (E.g. Certified, Registered, FedEx, UPS, etc)
To: Loebner Prize Contest
c/o Crown Industries, Inc.
155 North Park St.
East Orange, NJ 07017
If there is no compliant Entry for the 2006 Competition, the $2000
prize will be added to the 2007 Competition prize making the 2007
prize $4000, and the 2007 Competition will be held under these rules.
If there is only one Entry compliant with these rules the submitter(s)
of that Entry will receive $2000 and a Bronze Medal. If there are two
or more Entries compliant with these rules a Competition will be held
and the submitter(s) of the winning Computer Entry
will receive $2000 and a Bronze Medal.
Contest management reserves the right to enter one or more publicly
available open source programs,
The 2006 Competition will be a RESTRICTED CONVERSATION contest.
Each male Confederate will begin his interactions with each Judge by
asserting: "Hello, my name is John and I am a man."
Each female Confederate will begin her interactions with each Judge by
asserting: "Hello, my name is Joan, and I am a woman.
Each Entry MUST make one of those two assertions. The choice is left
to the entrant.
This assertion must be made on the first interaction between the
Entity and every Judge, but not necessarily more than once for each
Judge. If a Judge re-questions an Entity it is not expected or
required that the Entity reassert its name and sex (unless a line of
question dictates this). Following the initial forced assertion each
Entity should respond in such a manner as to attempt to convince the
Judge that the Entity is a human responding truthfully.
There are no other restrictions on conversations or topics.
Confederates and the submitters of Computer Entries must understand
that the transcripts of the interactions between the Entities and the
Judges will be published. Judges, Confederates and submitters of
Computer Entities are responsible for the content of their entries'
utterances.
It is the intent of the rules to provide a framework by which the
Competition may be held in a frugal and simple manner while being
meaningful.
Judging and Selection of finalists will depend on the number of
Entries according to the following Selection Rules.
If there is only one compliant program, the submitter(s) will be
declared the winner and awarded the medal and money.
If there are 2 - 4 compliant Entries, all will be entered in the Final
Four Round. The Final Four Round shall consist of paired comparisons
between the Computer Entries and the Confederates.
If there are 5 - 8 compliant Entries all will be entered into The
Prelim Eight Round. The four entries having the highest scores in that
round will enter the Final Four Round.
The Preliminary Eight Round shall consist of real-time questions and
answers between Judges and Computer Entries only. No human
confederates will be employed.
If there are 9 - 32 entries they will be prescreened down to eight.
These eight will be entered in the Prelim Eight Round. Prescreening
will consist of presenting all entries with a suit of test questions.
The eight entries responding in the most human manner will be selected
to enter the Preliminary Eight round.
If there are more than 32 Entries the first 32 compliant Entries
submitted will be entered in the prescreening round. HOWEVER, I
reserve the right to invite into the prescreening round compliant
Entries of outstanding merit.
Scoring For the "Final Four".
The Final Four Competition will be scored using the Method of Paired
Comparisons.
The task of each Judge will be to apportion 100 points between the two
terminals based upon how "human" they seem. The judgment is relative,
and ties are forbidden; one of the two terminals must be awarded at
least 51 points. No judge may use the same apportionment twice. That
is, if a judge apportions 100 points 40/60 for one Entry/Confederate
pair, then he or she may _not_ use a 40/60 apportionment again. This
will prevent ties if it becomes necessary to convert the judge's
scores into rankings.
We wish (a) each Entry to be compared with every Confederate; (b) each
Judge to evaluate every Entry, (c) each Judge to evaluate every
Confederate.
Label the four Entries E1..E4, four Confederates C1..C4, and four
judges J1..J4
The following matrix has Judges as rows and Entries as columns. The
intersection of each row and column shows which Confederate is
assigned to the combination of Entry and Judge.
....... E1 .... E2 .... E3 .... E4
----------------------------------
J1 .... C1 .... C2 .... C3 .... C4
J2 .... C4 .... C1 .... C2 .... C3
J3 .... C3 .... C4 .... C1 .... C2
J4 .... C2 .... C3 .... C4 .... C1
For example, reading across the row 2 we see that J2 compares E1 with
C4, E2with C1, E3 with C2, and E4 with C3. J2 will have scored every
Entry and every Confederate, but in different combinations than J's 1,
3 and 4.
Reading down the third column, we see in the first row that E3 is
judged by J1against C3. Let us enter a 1 in that cell if E3 scored51
or higher against C3, and a 0 otherwise. We may continue down the
column, entering a 1 in the second row if E3 scored 51 or higher
against C2, etc. The sum of the column will be the number of times E3
was judged as "more human" than a Confederate. We may do this for each
Entry.
SCORING
The Entry with the highest column total will be declared the winner.
**** The Silver Medal and $25,000 will be awarded **** if the score is
"2" or higher.
(I believe that an Entry which has been independently rated by two
different Judges as "more human" than two different Confederates in
direct paired comparisons deserves the Silver Medal.)
If two or more Entries tie for high column totals, each judge's scores
shall be converted to rankings. That is, for each judge, the entity
with the highest score will be scored with '1', the next highest with
a '2', and so forth to the lowest score which will receive a '8.'
These rank orders will assigned as scores. The rank scores that each
Entry received shall be totaled and the Entry having the lowest total
score of ranks will be declared the winner.
If two or more of the tied Entries remain tied based upon rank order
scoring, then victory will be decided by the results of the Semi-final
Judging.
Judging will consist of seven rounds of thirty minutes duration. Not
all Judges and Confederates will participate in every round. In each
round, Judges will have 27 minutes to interact with a pair. After the
27 minute evaluation period there will be a 3 minute break for
reassignment.
At the conclusion of the seven rounds the judges will have 20 minutes
to review the transcripts of their interactions with the four pairs
and to assign points based upon that review.
The following table shows each round. In the first round J1 compares
E1 with C1 and J2 compares E3 with C2. Judges J3 and J4 and
Confederates C3 and C4 are excused from the round. Excused Judges will
be kept separate from excused
Confederates and both will be kept separate from the competition
Round ...... Participating ............Excused
1 .... J1E1C1 J2E3C2 ................ J4 C3 J3 C4
2 .... J4E1C2 J3E3C1 J2E4C3 ......... J1 C4
3 .... J3E1C3 J4E3C4 J1E2C2 ......... J2 C1
4 .... J2E1C4 J3E4C2 ................ J1 C1 J4 C3
5 .... J2E2C1 J1E3C3 ................ J3 J4 C2 C4
6 .... J1E4C4 J4E2C3................. J2 J3 C1 C2
7 .... J4E4C1 J3E2C4................. J1 J2 C2 C3