Thoughts on Watson?
Ian Danforth
Drew
a good start towards an idea I have brewing. I’d like to use my
scratch-built psych model to create an AI (Artificial Insanity) which
can mimick human emotional and psychological states when responding to
conversations. I think I’ve got a good foundation as for how to
represent everything internally and be able to have it build a sense
of “self” and thusly ideas about “self” which will reflect in its
emotional responses. I even have considered how to adapt it to
different cultures, religions, and belief systems. I mainly just need
the language processing and KB component because it is huge and not
really my area of expertise, though I suppose it could become that if
necessary.
Cheers,
Drew
On Jan 14, 4:03 am, Ian Danforth <iandanfo...@gmail.com> wrote:
> After watching the truly amazing performance of Watson today, I wanted to
> know what your groups reaction was! It will be an exciting day when their
> structured dataset is well known as it will compliment NELL nicely.
>
>
> Ian
kyle hobson
Can you not use google's database of language?
Bryan Kisiel
a remarkable demonstration of being able to figure out exactly what is
being talked about given a single sentence. I wonder how much easier it
might be, though, to get that sort of system working only for Jeopardy
questions rather than for arbitrary sentences. I also wonder what all
they decided to store in Watson. NELL's a bit complementary in that we're
working on learning information rather than on using it, but, to my mind,
one may as well drive the other.
I've always imagined finding queries that would realy put NELL's interior
model of the world to the test, and then being able to get some
quantitative measurement of performance, or at least relative quality of
one KB vs. another. Better yet if two NELLs could try to stump or outwit
each-other.
Bryan Kisiel
need in such a KB, do let us know! The primary requirement is that you
provide about a dozen seed examples to go along with each.
(Also, I haven't forgotten about the web service, which I actually did get
a start on just before Christmas. The burden of periodically having
one person review and give feedback on 10 iterations worth of NELL's
learning became to great, and so there was a sudden drive to put in a
whole new feedback infrastructure and build a UI for it. I'll get back
around to the web service "one of these days".)
On Sun, 16 Jan 2011, Drew wrote:
> I have to say that was very interesting and impressive and definitely
> a good start towards an idea I have brewing. I�d like to use my
> scratch-built psych model to create an AI (Artificial Insanity) which
> can mimick human emotional and psychological states when responding to
> conversations. I think I�ve got a good foundation as for how to
> represent everything internally and be able to have it build a sense
> of �self� and thusly ideas about �self� which will reflect in its
Chris Troutner
In regards to putting NELL’s model to the test and/or improving it, wasn’t there a proposed idea to do a question and answer interface via Twitter where NELL could make a statement about something and users could provide feedback as to weather that statement was correct or not?
I can’t remember the details, but there was a web-based project a few years ago that implemented evolutionary algorithms to generate art. Visitors to the website would choose their favorite of two presented pieces of generated art. This input would then be used to seed the generation of the next two pieces of artwork to be voted on by the next visitor. As a result, there were some extremely beautiful and complex designs that were totally self-generated by the computer.
I imagine it wouldn’t be too complicated to implement a similar system utilizing the KB. This would be a good fork in the project as you could start a separate project by using a snapshot of the current knowledge base, crowd source corrective actions as mentioned above, and compare accuracy and depth of knowledge between the forked knowledge bases after a few months.
Cheers!
Chris Troutner
Bryan Kisiel
Yes, we're still planning on feeding the Twitter replies that NELL gets
back into it as a source of human feedback. It's just taking time to get
all the right pieces into place. Actually, it turns out that we're
starting at the question asking end of things -- we want NELL to figure
out on its own what it needs help with since we can't keep up with trying
to keep an eye on everything that it learns.
But we'll get to crowsourcing eventually, and all the different kinds of
things we could do with it. And we're still collecting all the tweets we
get for when that day comes.
Drew Mcpherson
Drew Mcpherson
being ready to start full steam on this anyway. I'm still thinking through
a lot of the concepts.
Cheers,
Drew
-----Original Message-----
From: Bryan Kisiel
Sent: Monday, January 17, 2011 7:18 PM
To: NELL: Never-Ending Language Learner
Subject: Re: [cmunell] Re: Thoughts on Watson?
Chris Troutner
Bryan,
Thanks for responding to this. It’s good to hear that crowdsourcing NELL is still in the plan. I understand it takes time to build and change the system in a responsible manner. I’m sure looking forward to seeing how crowdsourcing will improve NELLs accuracy, but its success will depend largely on how the interface is designed. Keep up the good work!
Bryan Kisiel
this stage of the game, NELL can make good use of a simple "that is right"
or "that is wrong" kind of feedback when it learns category instances.
But this is not the case for relation instances -- most of the benefit
that NELL can get currently requires something more along the lines of why
it is wrong. For instance, NELL gets a lot more out of it if it is told
that animalIsTypeOfAnimal("screwdriver", "pests") is wrong because
"screwdriver" is not an animal. Recently, NELL decided that "rabbits"
were a kind of "pests", which we regard as being correct for our purposes,
but NELL was actually thinking of "pests" in the context of insects, and
so we had to come up with a new way to tell it that it had learned
something that is wrong only in certain contexts (because rabbits are not
insects).
We expect NELL to start paying more attention to synonomy soon as well,
which is going to make it even more difficult for a given belief to be
clearly communicated to a human. Maybe NELL will say that Apple owns the
garbage trucks in New York City. Anybody would say that that is
incorrect, but maybe what NELL was trying to say is that Manhattan owns
the garbage trucks, and it simply went too far after learning that
Manhattan is also known as The Big Apple. How is the human supposed to
know that? And how is NELL supposed to know that the human isn't saying
that Manhattan doesn't own its own garbage trucks? So I have to agree
that interface design will have a lot to do with success.
Chris Troutner
Hey Bryan,
I’m really excited to hear that you and your team are focusing on synonymy and contextual interpretation with NELL. My personal opinion is that this is the biggest stumbling block facing modern internet search and that your results with NELL may hold the possibility of pointing in the right direction.
In both your examples for NELL, it seems that it struggles with the same contextual / synonymous problems that humans frequently struggle with. Say for example the stereotypical ‘hick’ visiting New York City for the first time. That person will experience new slang terms and contextual verbiage that will be confusing. They may be able to make ‘educated guesses’ as to their meaning, but will most likely misinterpret several of them. The way that humans correct their understanding is by asking other people to clarify the meaning of a sentence.
This is really a long winded way of asking: are you really looking for an ‘ideal’ algorithmic approach to get it right (the majority of the time) by working through a conceptual framework? E.g. insects == pets and rabbits are pests, therefore rabbits == insects. It seems to me that a human would easily make this mistake if they knew nothing of rabbits. I envision that the answer to this predicament is much like the approach used in inverse kinematics or null hypothesis testing.
In forward kinematics, you start with the knowledge of the shoulder location and calculate the needed positions of the elbow in order to place the hand at a desired location. Inverse kinematics starts with the location of the hand and computes the possible ranges the elbow and shoulder can take to achieve the known location of the hand. It calculates backwards.
This is sort of like Bayesian logic where you set up a problem to use P(A|B). However, if you already know P(B|A), you may be able to back-calculate to get P(A|B).
These are really just off-the-wall ideas to point out that perhaps the future ‘interface’ for NELL should include a way to identify contradictions and back-calculate to minimize the error or contradiction. If NELL knows that ‘Apple owns the garbage trucks in New York City’ with high confidence and its been told (with high confidence) that this statement is wrong, then it needs a function for minimizing the contradiction/error by looking at other possibilities in the knowledge base. This should lead to a corrective devaluation of the confidence level that Apple == The Big Apple == New York City.
Does that make sense?
Cheers!
Chris Troutner
-----Original Message-----
From: cmu...@googlegroups.com [mailto:cmu...@googlegroups.com] On
Behalf Of Bryan Kisiel
Sent: Monday, January 24, 2011 8:25 AM
To: cmu...@googlegroups.com
Subject: RE: [cmunell] Thoughts on Watson?
Indeed, I think the interface design will be a particular challenge. At this stage of the game, NELL can make good use of a simple "that is right"
Bryan Kisiel
That makes lots of sense to me (and I rather like the forward kinematics
analogy, BTW). NELL has a "knowledge integrator" component responsible
for taking stock of all the candidate beliefs that come in from the
different learning subcomponents, and it's responsible for deciding what
to to believe and what not to believe. Right now, the heart of the
knowledge integrator is still the same clunky heuristic thing that we used
a year ago to demonstrate the viability of our coupled-bootstrapping
approach. Before NELL can do any kind of real "reasoning" about anything,
that needs to be upgraded. And in fact one of the popular ideas here is
to use the sort of minimize-the-error-given-the-evidence arrangement that
you suggest.
Once that's in place, we could definitely go on to try to resolve
contradiction in the same process. Or maybe add a contradiction-assessor
as another subcomponent that would then throw it's agreement or
disagreement into the ring? I don't know if anybody has spent much
thought yet on how to model all that kind of stuff, but we have intentions
of getting NELL to ponder itself down the road, and I think what you
suggest fits right in with that. And I suppose that needing to make use
of vauge feedback could wind up motivating us to progress in that
direction. Interesting parallel -- thanks!
laserblue
The ELIZA effect comes to mind to temper enthusiasm. The expertise of the people that created the extraction patterns creates an illusion as deceptive as answering machine messages that sound like someone live is on the phone or Lenat's AM. Also, the format allowed one to form the heightened impression that speech recognition was occurrimg when it was not.
John Ohno
there's some logical structure. But, clearly automated learning is
done at least partly through modern statistical techniques. It looks
like it's a crossbreed, which makes sense -- ontologies, until they
can be generated automatically in a way that is reliably sensible and
involves minimal human observation, are not scalable to the level of
generality that WATSON seeks (hence CYC being simultaneously very
impressive and quite useless).
I have to say, when I am wrong, I too am completely out to lunch (as
is almost anyone with an extreme lack of domain-specific knowledge --
think of new-earth creationists trying to talk about geophysics, for
instance, and the mistakes they make). One can hardly claim that's
unique to a particular style of knowledge representation (although
shallow statistical techniques like taking a markov model of
space-tokenized strings can yield particularly amusing mistakes). One
is also reminded of the famous cold war era ontological AI project:
the 'soviet simulator' that claimed that the Soviet Union was planning
to throw eggs at the United States.
There are readily available papers about the WATSON internals. They
don't give quite enough detail to be useful for implementing similar
projects, but they make it clear that the system is two-fold, using
both neat and scruffy techniques. (I suspect what is being modified
for the purposes of using it for cancer research is the ontological
part, instead of any of the statistical parts. Statistics are pretty
rugged & general.)
> You received this message because you are subscribed to the Google Groups
> "NELL: Never-Ending Language Learner" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cmunell+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
--
--
John Ohno
http://firstchurchofspacejesus.blogspot.com/