$5,000 Prize money is guaranteed.
At risk will be $25,000 and the Silver Medal if an entry succeeds in
passing the text-only version of the Turing Test
Good luck with this. Isn't there some anniversary date coming up for
this or Alan Turing?
Scott Jensen
I would suggest a new prize be added and that for the best
conversational AI in a computer game. This new prize might do FAR
more to advance AI than the current prizes presently offered by you.
Once computer game companies start pushing conversational AI, real big
money could finally get put behind such development. It might even
spark a competition between such companies and be a competitive edge
they promote to the public ... which should encourage even more money
to be given to its development. If you want to discuss this new prize
further, drop me an email at: sj (at) rpwsj (dot) com
Scott Jensen
I know this has probably been covered by NIST but I think a prize for
translation would be in order. Humans translate badly because they
don't know Arabic well. A computer may be loaded with all languages
equally and bad translation therefore will imply a lack of
understanding in ALL languages.
I will pose my Turing Loebner question :- "What is the difference
between fighting Israel and pouring concrete at 50C?"
This came from Sadat's speech on the completion of the Aswan Dam.
Google translated "central air battle" when it should have been "the
battle against climate/environment".
Any computer that translates like Google will by definition fail to
answer that question.
- Ian Parker
Yes, good natural language translation is really the same problem in my
view as full human behavior. Not because I think langauge is the key to
intelligence, but simply because I believe all intelligent human behavior
is produced a single generic system of strong behavior learning. Language
behavior is a class of behavior broad enough, and complex enough, to be a
good test of that underlying behavior learning system. If you can't do a
good job at langauge behavior, you also won't be able to do a good job at
any intelligent learned behavior.
However, to translate accurately in all contexts, requires knowing far more
than just the langauge. It requires you know all human knowledge - that
is, the sum total of everything that every human knows. To translate a
message, you have to first fully understand the message. And to fully
understand any message, requires you understand the context the message was
created in. If you are trying to translate a speech by Sadat, you have to
understand all he knows to understand just what his words are talking
about. If he makes a reference to the Aswan Dam, the computer can't
translate the message correctly if all it knows is that a Dam holds water
and "Aswan" if the name of the dam. Is he talking about a pile of mud a 6
year old child made to dam up water running across the playground, or a
huge expensive nationally important construction project? To translate it
correctly, you have to know the full context which the words "Aswan Dam"
brings into the conversation - which means kowing history, and politics,
and engineering, and human nature, etc etc.
All human behavior, including all langauge behavior, is a problem of
producing the right behavior at any instant, based on the current context.
But for the system to correctly identify the current context, means it must
be correctly trained in what context the different stimulus signals
creates. The words "Aswan Dam" are a stimulus signal that activates in a
typical human a large context which then regulates the future behaviors.
All stimulus signals activate different parts of that huge complex context
which in turn, regulates what the human does next.
To "know" what "Aswan Dam" means, is to have internal context signals that
"correctly" activate in response to that stimulus signal, relative to the
context that was activate when the stimulus was received. I put
"correctly" in quotes simply because there is no single correct context or
response to that (or any other) stimulus signal. It will activate a
different context in each of us, based on our own personal experience of
those words - aka what we know about it. There is simply a wide range of
ways to respond to that stimulus which we would see as a typical valid
human response.
To solve the problem of AI, we have to understand how to build a machine,
that can learn from experience, how to create these large and internal
context representations, and use those, to regulate the production of
behavior. What we are waving our hands, or flapping our lips to make
words, it's all the same problem. We have to move the body parts, in an
appropriate way, based on our past experience (conditioning) in response to
the current stimulus context.
To translate langauge as well as any human translator requires the machine
to have a full, and complete, internal context created in response to the
stimulus signals, as the human translators brain creates when it receives a
request to create a translation. And that context that activates in a
human when he hears language, is not just a langauge context - it's a full
body experience context which relates all the sensory domains, and all the
action/movement domains.
On the subject of test format however, I think the traditional Turning
format used, as well as one based on a document translation, are both off
the mark.
Human interaction and human behavior is a REAL TIME process. It doesn't
work by producing an output sentence in response to an input sentence. Nor
does it work by producing an output document, in response to an input
document. It works by producing a continuous stream of output behaviors in
response to a continuous stream of sensory data. In this regard, both
suggested tests for "intelligence" miss the mark by a wide margin, and
encourage programmers to work on the wrong problem. How you write code to
produce a single one time, response to a typed input message, is very
different, from how you have to structure a system to produce a constant
stream of behavior.
At the same time, the key to real intelligence, is learning. How the
system reacts to a given input stream, should never be hard-coded in any
sense if you are testing for real intelligence. It must be learned from
experience. The correct way to respond to the stimulus signal "Aswan Dam"
is not some universal truth. It's a problem of how one correctly learns
the socially appropriate behavior to produce in response to that stimulus.
Document translation, and the traditional Turing test format encourage the
programmers to in effect cheat the real problem, by attempting to hard-code
all their personal knowledge (aka behavior) about correct social behavior
in our cultures into their systems, instead of working on the real problem
of building strong behavior learning systems.
I think langauge based tests - espcially interactive ones like the Turing
test, are a good test for intelligence. However, they are a test that
mostly encourages programmers to work on the wrong problem. They "cheat",
trying to get the best score possible for the specific test, instead of
working on the real problem.
If you want to make people work on the real problem of intelligence, you
shouldn't use langauge tests - they are basically too hard at this point -
which is what encourages the programmers to find ways to do better by
cheating.
To make them work on the real problem, create a real time learning problem
that deals with a constant flow of inputs and requires a constant flow of
outputs. Make it real time in the sense that not only must the good
outputs be generated, but make it time critical - like what you get when
you try to build a robot to catch a flying ball, or drive a moving car - so
that it does things like makes the turn soon enough so it doesn't drive off
the cliff, and not too soon, so it doesn't turn into the wall.
And make it a learning problem so that programmer doesn't even get to know
what domain, or what problem, the system will be solving ahead of time - so
there's no ability for the programmer to solve it by hard coding his own
knowledge into the machine. With contests like that, you will make the
programmers work on the real problem.
When these machines get really good a learning behavior from experience,
you can send them to school, teach them reading, writing, arithmetic, and
world history behaviors, and then when you give them a Turning test by
sending them the stimulus: "Translate this speech by Sadat please", they
will respond showing their true intelligence as they look up from the
computer they were surfing the net with: "Huh? Did you say something"?
Then you know it has passed the Turing test by correctly responding to a
translation request.
--
Curt Welch http://CurtWelch.Com/
cu...@kcwc.com http://NewsReader.Com/
"E by GUM 'n' Fold!" etc's
back in 2 weeks or years SO WHAT?
(only this time wiv de beard, huh! LOL :) ) <BLINKS!>
It is more than not "knowing" a language. The ability to use
language rides on innate as well as learned models of the
world and until these are in place language translation by a
program will always be limited.
JC
You are right. In fact to understand the speech properly you must also
understand the history of the Dam, Nasser's determination to build it.
Nasser complains about the treachery of the West. Not mentioned is the
fact that the West originally agreed to finance the Dam. Nasser
proceeded to recognize Communist China, something which at that time
was a red rag to a bull for the Americans. They then stopped the
financing.
The USSR then agreed to finance the Dam. Sadat in his speech talks
about eternal gratitude for the Soviet Union. I will accept that
translation is easier with that knowledge.
Let me take another example
http://docs.google.com/Doc?docid=0AQIg8QuzTONQZGZxenF2NnNfNzY4ZDRxcnJ0aHI&hl=en_GB
This is a scientific passage, about the different types of stars. Note
that Google has translated the Stefan Boltzmann law wrongly (radiation
is proportional to T^4 NOT 4 times the temperature. The surface area
of a sphere is wrong too.
Science takes us into the area where we have formal systems. In my
prologue to the translations I talk about the main sequence and the
science in general.
There are two questions.
1) Can we use formal data - The SB law is in systems like Mizar in
translation.
2) Can we put in political considerations, such as the relationship
between the US, Communist China, Egypt etc. and the Dam in. The Dam
itself is in some senses trivial. I can talk in pure engineering terms
about the Dam, the flow of water in the Nile, the fact that the White
Nile rises in Lake Victoria and has an essentially constant flow
throughout the year, whereas the Blue Nile rises from Lake Tana in
Ethiopia. Flow is strongly seasonal and the Blue Nile (essentially)
sizes up Aswan.
This essentially gives me my constraints.
This discussion is in fact instructive from the point of view of AGI.
Suppose I want to build another dam, say the 3 gorges in China. I need
to engineer it.
There is formal data, therefore in the form of geographical and
hydrological information about what the dam is. There is also abundant
information in the shape of PERT charts about the building of dams.
I can view politics as a graph with Communist China, the West, Egypt
the USSR as nodes with relationships between them.
You talk later on about "cheating". To me the essential question is
can you encode such information in such a way that the computer can
understand it, and if so how. If I look up Aswan Dam on Wikipaedia I
get out a lot of facts. What do we do?
To me way of thinking the first thing to do is to look out for key
words/expressions. We do this by
1) Looking for certain critical n-grams. This can be done in a hashing
table.
2) Using Latent Semantic Analysis to find out whether we have the
precise meaning we expect.
3) Using facts to guide a translation or to include in conversation in
a Turing Loebner test.
http://sites.google.com/site/aitranslationproject/Home/formalmethods
This gives some indication of how we can look at relevant topics. The
table on the "Big Bang" etc. is from Google. I think that using a Web
3.0 type of methodology one could come close. Just a thought.
> To solve the problem of AI, we have to understand how to build a machine,
> that can learn from experience, how to create these large and internal
> context representations, and use those, to regulate the production of
> behavior. What we are waving our hands, or flapping our lips to make
> words, it's all the same problem. We have to move the body parts, in an
> appropriate way, based on our past experience (conditioning) in response to
> the current stimulus context.
>
> To translate langauge as well as any human translator requires the machine
> to have a full, and complete, internal context created in response to the
> stimulus signals, as the human translators brain creates when it receives a
> request to create a translation. And that context that activates in a
> human when he hears language, is not just a langauge context - it's a full
> body experience context which relates all the sensory domains, and all the
> action/movement domains.
>
Interesting you should say this. I have been saying this for some
time. Suppose you ask me the question in a naive way. "Is the person
intelligent" I do not expect any restrictions in the format. Turing
certainly did not.
If you have a broad range of questions you cannot "cheat". I expect
when you talk about "cheating" you mean a program with a narrow
knowledge domain. In fact if we can encode knowledge about Aswan.
If I take another narrow domain, that of Astronomy, I can cheat in a
similar way. If however I cheat in everything I am not cheating,
because then the system will truly be intelligent. It may well be that
AGI is simply a case of having a large number of AI programs and
covering the field.
> If you want to make people work on the real problem of intelligence, you
> shouldn't use langauge tests - they are basically too hard at this point -
> which is what encourages the programmers to find ways to do better by
> cheating.
>
> To make them work on the real problem, create a real time learning problem
> that deals with a constant flow of inputs and requires a constant flow of
> outputs. Make it real time in the sense that not only must the good
> outputs be generated, but make it time critical - like what you get when
> you try to build a robot to catch a flying ball, or drive a moving car - so
> that it does things like makes the turn soon enough so it doesn't drive off
> the cliff, and not too soon, so it doesn't turn into the wall.
>
> And make it a learning problem so that programmer doesn't even get to know
> what domain, or what problem, the system will be solving ahead of time - so
> there's no ability for the programmer to solve it by hard coding his own
> knowledge into the machine. With contests like that, you will make the
> programmers work on the real problem.
>
> When these machines get really good a learning behavior from experience,
> you can send them to school, teach them reading, writing, arithmetic, and
> world history behaviors, and then when you give them a Turning test by
> sending them the stimulus: "Translate this speech by Sadat please", they
> will respond showing their true intelligence as they look up from the
> computer they were surfing the net with: "Huh? Did you say something"?
> Then you know it has passed the Turing test by correctly responding to a
> translation request.
>
They will never "go to school". What we are basically after is
including an encoding of knowledge which already exists. If I search
for "Stefan Boltzmann" or "Black Body" I get the correct law. Why
can't Google translate properly?
- Ian Parker
Thoughs things ARE what "knowing a langauge" refers to. What else could
the phrase refer to? If you don't have the ability to produce and respond
to the langauge correctly, then YOU DON'T KNOW IT. If you do have the
ability, then YOU KNOW THE LANGUAGE. How the hardware works, in terms of
how much of the hardware is configure by nature and how much of the
hardware is configured by nurture is irrelevant to everything I wrote.
--
Curt Welch http://CurtWelch.Com/
It would also need to include a program to select, manage and
modify those AI programs. It would not require you to encode
all the data manually.
JC
I was hasty in my response and should have read it with
more care. Ian Parker went on to write:
"A computer may be loaded with all languages equally and
bad translation therefore will imply a lack of understanding
in ALL languages."
The point I was making was the "understanding" is in the
model not in the part of the model that is the languages
themselves which seems also to be what Ian was saying.
For example if the model is simply replacing one English
word for an Arabic word I don't think there is any human
level understanding of English or Arabic languages.
> If you don't have the ability to produce and respond
> to the langauge correctly, then YOU DON'T KNOW IT.
How to define "correctly"? If I ask the program, "What
is the color of Mary's hair" and the program using a
data base look up with sentence templates responds "She
has brown hair" can you say the program "understands"
anything about Mary or even what it is talking about?
> If you do have the ability, then YOU KNOW THE LANGUAGE.
But how do you cover all the bases to know that the
program "understands" (responds correctly)? The Dr Eliza
program can give the appearance of understanding a sentence.
> How the hardware works, in terms of how much of the
> hardware is configure by nature and how much of the
> hardware is configured by nurture is irrelevant to
> everything I wrote.
I wasn't responding to what you wrote. The relevance
of nature vs. nurture is a practical issue. You want to
evolve it all and I say we begin with a default model
that reflects the real world.
JC
This is what I consider to be quite a well considered program on how
to do this. Matt is a leading researcher in AI and his thesis was just
this subject.
What you need is a system which has rules, a system which gives you
time stamps and which allows different programs to mix together.
Suppose I have the following in Matt's system.
1) A program to map out hydrological resources. Sizes up the Blue
Nile.
2) A Pert chart for Dam construction.
3) Chemistry of concrete.
4) Lastly politics. It is perhaps ironic that the Soviet Union forgot
itself in 1989. Sadat said that he would be eternally grateful.
We have descriptions of all these programs in NL. Now the NL
representation describes how they are all interrelated. We need to
size up water flow (Program 1).
The understanding of NL then is the understanding of sub domains. I
have a 5th program, one that performs LSA and other forms of
linguistic analysis. I take my PERT chart and then fit programs into
it.
The system has to provide a set of rules, you are quite right in
saying that. You can read about what these rules are from the
reference.
http://mail.google.com/mail/?hl=en&tab=wm#label/AGI/124f578fa3555666
Matt has in fact costed his proposals. I personally believe this to be
a mistake. I have not replied to his thread but I feel it should be
possible to test his proposals on a small scale, and then make the
whole thing self financing.
I have a similar set of programs for the Stefan Boltzmann law. In fact
with Mizar I will start of with Clifford Algebra. This in fact will
define the symmetries that lead to Fermi Dirac and Bose Einstein
statistics. The photon is a boson BTW.
There are a few things Matt has left out.
1) As entities coalesce they will multiply to infinity. A pruning
process will give the look at feel of a genetic algorithm working on
"survival of the fittest".
2) Matt has got rules for executing programs. These rules will have to
include identification protocols based on RSA like algorithms. This
will make the whole system intrinsically secure.
- Ian Parker
>> If you don't have the ability to produce and respond
>> to the langauge correctly, then YOU DON'T KNOW IT.
>
> How to define "correctly"? If I ask the program, "What
> is the color of Mary's hair" and the program using a
> data base look up with sentence templates responds "She
> has brown hair" can you say the program "understands"
> anything about Mary or even what it is talking about?
Lenat's Cyc was evidently much too ambitious. But, hypothetically
assuming that there were a comprehensive data base to answer
all sorts of questions, the issue of whether the program
"really" understands wouldn't practically matter, I suppose.
M. K. Shen
Children produce and respond to language without a large
data base. Language is more than looking up answers to
questions. Language is used to communicate about things
that exist in the brain in another form. You may not
recall the actual sentences used in this post but you
may retain an understanding of what was written. The
understanding resides in some innate abstract mental
framework of space, time, causality and substance that
allows the acquisition and use of language.
JC
Both you and Mok are right in a sense. If you are satisfied with weak
AI it does not matter how this is achieved. I feel everyone should
also remember that a database contains not only responses to Turing/
Loebner questions but also programs.
If I were to ask a question about a complex configuration in General
Relativity. - viz how are jets formed by a black hole? I can call on a
program which will do a finite element calculation.
Tis is not how the brain works, or is it? Let me take another
instance. Suppose I am attempting to understand something. What are my
mental images that you talk about. Answer - they are PROCEDURES that
are refered to. Let me leave you with this thought.
- Ian Parker
That's his old version. The new one is here:
http://www.mattmahoney.net/agi2.html
> The system has to provide a set of rules, you are quite right in
> saying that. You can read about what these rules are from the
> reference.
>
> http://mail.google.com/mail/?hl=en&tab=wm#label/AGI/124f578fa3555666
It appears that link can be read by you and only you from your Gmail
mailbox.
Well, I think in fact the brain is doing nothing more than "looking up the
answers" in a large database. However, the way that database is created
and the way the "answers" are stored, is what is not at all typical of what
we think about when we talk about this database lookup abstraction.
If I ask you a question, I would argue your brain is simply looking up the
answer and making you produce some behavior as the answer. But the
"lookup" is not limited to the question I asked you. It's based on the
brain's entire sensory context which will include far more than just the
question, and will extend much further back in time than just to the
beginning of the question. The brain will produce the "answer" (aka
behavior) that it has been conditioned to produce for the given context as
defined by its recent past sensory inputs.
I think "understanding" is nothing more than having the right answers in
that database - which is why I would also argue that a computer program
that simply looks up the answer to a question like "What is 2 + 2" and
produces the response "4" has some understanding. Having that one bit of
stimuls->resonse coded into the data base means it doesn't understand much
at all about math, but it understands how to answer that one question.
> Well, I think in fact the brain is doing nothing more
> than "looking up the answers" in a large database.
> However, the way that database is created and the way
> the "answers" are stored, is what is not at all typical
> of what we think about when we talk about this database
> lookup abstraction.
Any system can be reduced to a look up table in the sense
you are suggesting but it isn't useful in understanding
in what way it is "not at all typical".
> If I ask you a question, I would argue your brain is
> simply looking up the answer and making you produce
> some behavior as the answer. But the "lookup" is not
> limited to the question I asked you. It's based on
> the brain's entire sensory context which will include
> far more than just the question, and will extend much
> further back in time than just to the beginning of the
> question.
Yes, well, that is the problem isn't it? How do you
use all the past inputs to fill up a look up table
with useful values?
JC
Suppose we type something into Turing/Loebner. Let us suppose I ask
"What is the Stefan Boltzmann Law?" - Factual answer required.
"How is it derived?" - Quantum mechanical answer.
"Could you give me examples of Fermi Dirac?" - Big Bang - Quark soup
I could go on. Our brains are associating things.
- Ian Parker
> Suppose we type something into Turing/Loebner.
I see the Loebner prize as essentially rewarding the
best fake human intelligence of the year.
> Let us suppose I ask
>
>
> "What is the Stefan Boltzmann Law?" - Factual
> answer required.
>
>
> "How is it derived?" - Quantum mechanical answer.
>
>
> "Could you give me examples of Fermi Dirac?"
> - Big Bang - Quark soup
Why start with physic questions? A speech interface
to a data base may be useful but it isn't the same
as human intelligence.
> I could go on. Our brains are associating things.
And more than just associating things ...
I am posting from comp.ai.philosophy and I think
your views are from comp.ai.nat-lang?
JC
I was taking Physics as an example because it is contained in
http://docs.google.com/Doc?docid=0AQIg8QuzTONQZGZxenF2NnNfNzY4ZDRxcnJ0aHI&hl=en_GB
When you have been trained in Physics and you see howlers it gets to
you. When you (or was it someone else) talked about understanding a
language some transhumanistic thoughts crossed my mind. I translated
the text with the aid of hashing algorithms. Together we understand
Arabic, separately neither of us does.
Anyway we could do the same thing for Politics. I could take the
phrase "we will always remember the Soviet Union" ans associate it
with such things as events in Europe and the fall of the Berlin Wall.
You see the USSR forgot itself. However I decided to make my main
examples scientific. Too much politics creeps in.
- Ian Parker
I have just visited your home page. I see your view
on achieving AI is:
"To achieve AI one needs a formal description of our
thought processes."
I won't comment further as I don't have anything useful
to add with regards to interests of those posting to
comp.ai.nat-lang.
JC
Yes, they associate a behavior, with a stimulus. That idea fully explains
everything you are suggesting above.
The stimulus ""Could you give me examples of Fermi Dirac?" (within the rest
of the stimulus context of you writing a Usenet post response to the
messages you had read) produced the behavior "Big Bang - Quark soup" from
you.
Whether that behavior is produced as an internal "thought" (as we like to
talk about it) or an external lip moving behavior is not relevant. They
are both still the response your brain produced in response to the
stimulus.
How the brain manages this large association task between continuous
parallel input streams to produce associated continuous parallel output
streams is the question - the one John just asked as well which I have not
yet, but will try to find the time to answer.
> - Ian Parker
The way one would attempt to solve it is to look at problems which are
associated with it, see if I can get some insights.
As far as formal descriptions are concerned. At one level programs are
formal descriptions - therefore AI must be a formal description. A
formal description (encoded in Mizar) is unambiguous and tells the
computer exactly what it wants done.
The translation errors in the second Arabic text - the one on stars
and the main sequence were formal scientific errors as well.
To me though the $64,000 (or perhaps I should say $64 billion) is
1) Can you represents other forms of knowledge than mathematical in
formal terms?
2) Can you build an interface (like Alcor) that operates in Natural
Language?
- Ian Parker