Teaching assistant Jill Watson

[John F Sowa]

An example of how to build a helpful Q/A system for any subject.

It also shows why the Turing Test is useless as a test of whether
a computer system is "thinking". The messages that Jill W. sent
were written by human teaching assistants. With a good system
that matches questions to likely answers, you can develop a
robot TA that can even fool students who are taking an AI course.

URL and excerpts below.

John
___________________________________________________________________

From
http://www.news.gatech.edu/2016/05/09/artificial-intelligence-course-creates-ai-teaching-assistant

Professor Ashok Goel teaches Knowledge Based Artificial Intelligence
(KBAI) every semester. It’s a core requirement of Georgia Tech’s online
master’s of science in computer science program. And every time he
offers it, Goel estimates, his 300 or so students post roughly 10,000
messages in the online forums — far too many inquiries for him and his
eight teaching assistants (TA) to handle.

That’s why Goel added a ninth TA this semester. Her name is Jill Watson,
and she’s unlike any other TA in the world. In fact, she’s not even a
“she.” Jill is a computer — a virtual TA — implemented, in part, using
technologies from IBM’s Watson platform.

[Frank Guerino]

Hi John,

Also along these lines, I recently got to be a part of an IBM executive briefing that included this video…

Title: IBM Watson now powers a Hilton hotel robot concierge

Web Address: http://arstechnica.com/gadgets/2016/03/ibm-watson-hilton-robot-connie/ 

The entire Watson platform is growing in leaps and bounds and it appears IBM's goal is to keep automating knowledge workers, every opportunity they get, in every industry.  They stated that they’re going after largest database of Question-Answer sets, in as many industries as possible.

There are also numerous major pharma companies that are investing heavily in using Watson for things like clinical trials, drug experiences, treatment pathways, and even service desk speech analytics for caller experience optimization (voice based Q&A that responds to the callers emotional tones).  I also got to witness how Mt. Sinai hospital is working with IBM to leverage image analytics for medical scan anomaly identification.

I personally feel it’s pretty amazing progress.

My Best,

Frank

-- 
Frank Guerino, Chairman
The International Foundation for Information Technology (IF4IT)
http://www.if4it.com
1.908.294.5191 (M)

--

All contributions to this forum by its members are made under an open content license, open publication license, open source or free software license. Unless otherwise specified, all Ontolog Forum content shall be subject to the Creative Commons CC-BY-SA 4.0 License or its successors.

---

You received this message because you are subscribed to the Google Groups "ontolog-forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.

To post to this group, send email to ontolo...@googlegroups.com.

Visit this group at https://groups.google.com/group/ontolog-forum.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/baad53ed-e723-ab23-1b80-a800e635b9a3%40bestweb.net.

For more options, visit https://groups.google.com/d/optout.

[Alex Shkotin]

John,

is there any URL to ask Jill?

Alex

[Alex Shkotin]

John,​​

Nowadays we should complicate Turing Test for ELIZA Weizenbaum and Jill Watson: read this text and we'll test you:-)

Alex

2016-05-19 23:39 GMT+03:00 John F Sowa <so...@bestweb.net>:

[John F Sowa]

On 5/20/2016 4:27 AM, Alex Shkotin wrote:
> is there any URL to ask Jill?

The TA service was probably restricted to students who took
the course. But it would be interesting to see what kinds of
responses Jill W. would generate. You might check the web site
or send a message to Prof. Ashok Goel.

If you find anything useful, please send a note to Ontolog Forum.

John

[Alex Shkotin]

After https://developer.ibm.com/watson/blog/2015/11/11/watson-question-and-answer-service-to-be-withdrawn/ 

there is no doubt they did it :-)

And I'll wait for Russian as

Alex

--
All contributions to this forum by its members are made under an open content license, open publication license, open source or free software license. Unless otherwise specified, all Ontolog Forum content shall be subject to the Creative Commons CC-BY-SA 4.0 License or its successors.
--- You received this message because you are subscribed to the Google Groups "ontolog-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.
To post to this group, send email to ontolo...@googlegroups.com.
Visit this group at https://groups.google.com/group/ontolog-forum.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/5dd34fe3-fbdb-3a14-ef38-3c462f914e83%40bestweb.net.

[John Bottoms]

Alex,

I have not heard about an IBM agreement to make it publicly available.
However, the article says that some of the grad students are working on a work-a-like. I would imagine that the people-TA's are among those. I'll see if I can make contact with them since their names were published.

JohnS: Turing test useless? I agree, I know people who attribute insight and empathy to "their" slot machine.

-John Bottoms

On 5/20/2016 4:27 AM, Alex Shkotin wrote:

[John F Sowa]

Alex and John B,

AS

> After https://developer.ibm.com/watson/blog/2015/11/11/watson-question-and-answer-service-to-be-withdrawn/
> there is no doubt they did it

That's an example of a useful tool. But it's not a system that
understands language. I discuss Watson and other systems in slides
22 to 45 of http://www.jfsowa.com/talks/nlu.pdf

JB
> Turing test useless? I agree

It's useless as a test for whether a machine can think. But the
technology can be useful as a stimulus for developing better
methods of matching a pattern in a query to a pattern of info
in some document or collection of documents.

In fact, that's what Google and most search engines do. IBM's
Watson (and related systems) are more sophisticated versions.
But the systems can't be said to understand what they're doing.
See nlu.pdf.

> I know people who attribute insight and empathy to "their" slot machine.

Until the 13th century, most things that performed some kind
of service were driven by or connected to some human or other
kind of animal. Therefore, it was reasonable to treat anything
that did something useful or interesting as animate. That's
also why people are willing to treat systems like Siri as though
they were animate beings.

The nlu.pdf slides also discuss DNNs (deep neural nets), but the
word 'deep' in front of 'learning' is a misnomer. I sympathize with
Gary Marcus:
https://www.edge.org/conversation/gary_marcus-is-big-data-taking-us-closer-to-the-deeper-questions-in-artificial

But the technology for DNNs can be useful. For papers on the subject,
see http://dl4mt.computing.dcu.ie/DL4MT-ref.html

John

[Alex Shkotin]

JohnB,

I was impressed that Watson services become better and better, see http://www.ibm.com/cloud-computing/bluemix/watson/

And any kind of QAS is always interesting as well as a technology to build it. 

Do we have some kind of competition, benchmark for them besides Turing test?

JFS,

I agree "That's an example of a useful tool.  But it's not a system that

understands language." and congrats with http://kyndi.com/! I did not dive yet:-)

Alex

--
All contributions to this forum by its members are made under an open content license, open publication license, open source or free software license. Unless otherwise specified, all Ontolog Forum content shall be subject to the Creative Commons CC-BY-SA 4.0 License or its successors.
---
You received this message because you are subscribed to the Google Groups "ontolog-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.
To post to this group, send email to ontolo...@googlegroups.com.
Visit this group at https://groups.google.com/group/ontolog-forum.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/d2c5eea2-c2ad-8a50-bdfa-d1640d6f538f%40firststarsystems.com.

[John Bottoms]

Alex,

I've been working on this recently from the viewpoint of a system issue.

First, let's make some observations about the role of a TA.
- It involves the syllabus for the course with the requirements for competion.
- There is some notion of the level of excellence by the TA needed to successfully achieve the goal.
  That level of excellence is presupposed to match or exceed that of a human TA.
- The administration of the course is a cooperative process that involves expert assistance.
- Nominally questions are submitted by email. You can't go see Jill in her office.
- Jill need not respond immediately.
  She has the "graceful delay" of saying, "I will need to talk to the professor about this."
- This delay allows the editing of Jill's responses by an Watson system SME.

While this sounds like moving the goal posts, I think that observation had been made with Watson's work on Jeopardy. The IBM staff explained the difficulty with images and music identification. And the Jeopardy staff made adjustments to the rules of Jeopardy. I'm sure the IBM staff were looking at the likelihood of success in light of these changes. I would imagine a similar design task would be carried out in evaluating any new project. There are goals and it is both party's interests to complete the project successfully.

Some people may be put off by this process but I believe it is just what we need. IBM has moved Watson from being strictly a smart machine to a cooperative system that consults with experts in the field for which the system is being developed. That it cannot be done automatically is not a concern either. We have seen videos of student doing the rounds with a senior physician as they strive to achieve an acceptable confidence level in medicine.

This also becomes the prima facia reason why we cannot use the Turing Test as a framework for testing systems. There have been articles that discussed the acceptable level of performance by Watson. It is going to make mistakes and errors. And it is deemed ready for implementation when that level is an improvement compared to the performance of humans. Finally, in a professional setting that work can always be subject to continuous review by an expert.

This means that if we cannot use the Turing Test as a guideline then we need to consider an alternative. In general, in engineering we use the interview process in which a candidate interviews with several staff members, perhaps across time before a decision is made. With the creative disciplines an artist is expected to bring a portfolio or be able to demonstrate proficiency such as in ballet. Perhaps this should be done with intelligent systems.

Backing up from that scenario, I have an earlier issue. It is this: The medical student, the aspiring artist and the proficient dancer all have gone through rigorous training. The level of performance has been discussed at length and those students' performances reflect their school's curriculum. That is not true in the case of intelligence systems. We have no curriculum identified beyond that of the particular discipline. But, there has been little discussion of the ancillary tasks associated with professional performance. How do you dispense fluids from a pipette, and what do you do when you break a vial of a pathogen.

We are behind the 8-ball in this respect first, because the field is changing quickly, and 2nd because nobody wants to send Johnny-5 to 14 years of school. I would hope that at some point the Academy sees that this shortcoming is going to impair growth of these related industries until the infrastructure issues have been addressed.

-John Bottoms
 FirstStar Systems

[John F Sowa]

On 5/22/2016 5:16 PM, John Bottoms wrote:
> First, let's make some observations about the role of a TA.
> - It involves the syllabus for the course with the requirements for

> completion.

> - There is some notion of the level of excellence by the TA needed to
> successfully achieve the goal.
> That level of excellence is presupposed to match or exceed that of a
> human TA.

Jill W. did not do anything remotely like that. Today's AI technology
is incapable of matching human performance as a TA. And it won't come
close to human capability for many, many years to come.

There were 300 students in the course, and they asked many similar
questions. Furthermore, students did *not* address a particular TA
when they asked a question. They just sent them to a pool where any
of 9 TAs could select a question to answer. Eight of them were human,
and one was Jill W.

When a new question came in, Jill W. was the first to check whether it
was similar to any question that had previously been answered. If Jill
was 97% sure that the question was sufficiently similar to some previous
question, Jill would send the previous answer written by some human TA.

If Jill did not answer a question, one of the 8 humans would answer it.
Then that new Q/A pair would go into the pot from which Jill would be
able to select that answer if some new question happened to be similar.

John

[Alex Shkotin]

JB,

thank you for full-dressed respond. It's clear there a lot of to do for clever QAS and when we trust them: in Jepardy! we know answer:-)

The next step to thinking QAS: they should explain "Why?" and in some areas should be convincing.

And I like "use the interview process in which a candidate interviews with several staff members" for QAS.

And definitely we may have QAS to answer "How to dispense fluids from a pipette, and what to do when you break a vial of a pathogen."

Jill Watson (as I think JFS has described) is just FAQ-storage with QA-wrapper and human assistent when Jill does not know answer - we say "simply and tastefully" in this case.

Alex

--
All contributions to this forum by its members are made under an open content license, open publication license, open source or free software license. Unless otherwise specified, all Ontolog Forum content shall be subject to the Creative Commons CC-BY-SA 4.0 License or its successors.
---
You received this message because you are subscribed to the Google Groups "ontolog-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.
To post to this group, send email to ontolo...@googlegroups.com.
Visit this group at https://groups.google.com/group/ontolog-forum.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/2780e389-9210-efe8-5ee2-4f025ae76caa%40firststarsystems.com.

[John Bottoms]

On 5/22/2016 10:10 PM, John F Sowa wrote:

On 5/22/2016 5:16 PM, John Bottoms wrote:

First, let's make some observations about the role of a TA.
- It involves the syllabus for the course with the requirements for
completion.
- There is some notion of the level of excellence by the TA needed to
successfully achieve the goal.
  That level of excellence is presupposed to match or exceed that of a
human TA.

JohnS,

I was careful to say that we should consider the "role of the TA". This is a key component of the design process. I was not aware that they had said there were another 8 human TA's answering questions. In fact, it could have been just one, or just the professor, as I proposed.

To me, what's important here is the migration path. It now appears that IBM's strategy is to develop "thread communities" to address problems in technical domains. And, with their work in several areas they show that careful design of the technical team that includes Watson they are able to do just that. The unknown next step is how two different threads within the same domain would be joined? How could you cross over responsibility from one sub-discipline to another.

That does not seem to be an insurmountable problem using Watson's approach to knowledge. (And yes, Alex, it is an extension of knowledge management via the PageRank algorithm that Google uses.) And that process of sharing knowledge is being addressed in the development of federated apps by companies such as Google and Microsoft.

The contrary view is that Watson needs to be more intelligent through the addition of components such as inference engines and theorem provers. The following link points to an article that takes a very strong view on that position.

     "The fraudulent claims made by IBM about Watson and AI:They are not
       doing "cognitive computing" no matter how many times they say they are."

http://www.rogerschank.com/fraudulent-claims-made-by-IBM-about-Watson-and-AI

In response, "Cognitive Computing" increasingly appears as a marketing term that is an alternative to "Artificial Intelligence". This is because no company wants to reveal what is inside their secret sauce. No one wants to say that they have an ontology and associated functions that provide intelligence. That is IBM's prerogative. That reveal is an academic exercise, not a commercial one. If they were to mention what will be inside Watson in the future, they would be on a slippery slope.

These objections overlook a key factor that IBM has recognized. They have moved computing from a simple knowledge response system to a human-cooperative system through the integration of computing with existing knowledge communities. Their selected migration strategy diverts around the issues of "smarts", at least for now, and they have done that successfully.

And for now, those of us interested in the implementation of KM; we need to determine what the future of those systems look like that extend current systems, and what the payoff would be for one that was effective in solving problems that are not currently addressed. Again, I believe it comes down to the migration strategy and that strategy may vary from industry to industry. IBM has picked their strategy, it is applicable across multiple areas and it appears to be effective. Next, each company working in this space will need to do that also. Our task is  to show how ontologies can best be implemented for each domain. Should we be following IBM's blazed trail?

-John Bottoms
 FirstStar Systems

[Adrian Walker]

Alex,

You wrote The next step to thinking QAS: they should explain "Why?" and in some areas should be convincing.

That step is arguably already achieved, albeit in an unexpected way.  Please see

www.executable-english.com/internet_business_logic_in_a_nutshell.pdf

                              Cheers,  Adrian

Open Apps in Open English for Open Data
Online at www.executable-english.com  
Shared use is free, no advertisements

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/CAFxxROQk6SLcL7EGmVJay7Nuz3uCVU8SdsDR8a_wWF1_ynNX6Q%40mail.gmail.com.

[Alex Shkotin]

Adrian,

this is great! I remember IBL and ExEng, but just did not pay attention on explanation service.

Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/CABbsESexK30T58X%2B8%3D6m1jk-bfB7RcK-%3DWytP8-m9aBipk%2BDXw%40mail.gmail.com.

[Alex Shkotin]

And Adrian,

before the Watsom@IBM the question was show me your ontology but now - show me your QAS :-)

Watson has 2 demo-QAS 

"We have a few demo applications to get you started in the App Gallery:  What’s in Theaters and Questions on the Natural Language Classifier are great starting points."

This is a deal.

Alex

2016-05-23 20:24 GMT+03:00 Adrian Walker <adrian...@gmail.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/CABbsESexK30T58X%2B8%3D6m1jk-bfB7RcK-%3DWytP8-m9aBipk%2BDXw%40mail.gmail.com.

[Adrian Walker]

Alex -- and explanations?

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/CAFxxRORE%2B-8Ajx%3DRqjR7ik5MdragVBGB0%2BmYLjsRTFBAswnsqA%40mail.gmail.com.

[Alex Shkotin]

Adrian,

it would be nice to ask "Why?" and get something special:-)

Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/CABbsESeWtWYASVdqu%3D3Byco4MRny_b%3DZdg1AQGPrr%3DAxrXBLSA%40mail.gmail.com.

[John F Sowa]

On 5/24/2016 3:52 AM, Alex Shkotin wrote:
> it would be nice to ask "Why?" and get something special

"Why?" is the most important question and also the most difficult
to answer in general.

With today's technology, there are two ways to answer "why" questions:

1. Use an automated FAQ system (such as Jill W.), which returns
an answer that had previously been written by some human.

2. Use a theorem prover that answers a "why?" question by mapping
the previous step in a proof to a reply in some NL. (This is
the method that Adrian Walker implemented in his system.)

These methods are useful for some purposes. More difficult examples:

* Why did the stock market go down?

* Why did the voters choose candidate X?

* And the endless stream of "why?" questions that preschool children
ask their parents.

John

[Gary Berg-Cross]

An unsatisfactory answer to a "why" question can drive the search for a better understanding in humans. So it would be interesting to consider what an artificial system might do if it had a sense of this inadequacy.
This quickly gets into issues of data gathering, working hypotheses and the like.

Gary Berg-Cross

--
All contributions to this forum by its members are made under an open content license, open publication license, open source or free software license. Unless otherwise specified, all Ontolog Forum content shall be subject to the Creative Commons CC-BY-SA 4.0 License or its successors.
--- You received this message because you are subscribed to the Google Groups "ontolog-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.
To post to this group, send email to ontolo...@googlegroups.com.
Visit this group at https://groups.google.com/group/ontolog-forum.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/bb5c4644-724c-ad43-8db3-476f684a9f9b%40bestweb.net.

[Adrian Walker]

John,

You wrote..  More difficult example:   Why did the stock market go down?

We can push the envelope a bit.

The following example -- Why did economic growth go down?  -- shows the kind of data and executable knowledge that is needed for any technology to be able to answer a similar question:

www.executable-english.com/demo_agents/GrowthAndDebt1.agent

www.astd.org/Publications/Magazines/The-Public-Manager/Archives/2013/Fall/Social-Knowledge-Transfer-Using-Executable-English

                                   Cheers,  -- Adrian

Open Apps in Open English for Open Data
Online at www.executable-english.com 
Shared use is free, no advertisements

--
All contributions to this forum by its members are made under an open content license, open publication license, open source or free software license. Unless otherwise specified, all Ontolog Forum content shall be subject to the Creative Commons CC-BY-SA 4.0 License or its successors.
--- You received this message because you are subscribed to the Google Groups "ontolog-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.
To post to this group, send email to ontolo...@googlegroups.com.
Visit this group at https://groups.google.com/group/ontolog-forum.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/bb5c4644-724c-ad43-8db3-476f684a9f9b%40bestweb.net.

[Nadin, Mihai]

From the Why? question to Executable English—great take.

Computing with words, anyone? Zadeh was probably ahead of us (again).

 

Mihai Nadin

www.nadin.ws

www.anteinstitute.org

 

[Adrian Walker]

Mihai,

Thanks for your kind words. 

Please see also:

www.executable-english.com/demo_agents/Zadeh1.agent

www.executable-english.com/demo_agents/Zadeh2.agent

                                          Cheers,  -- Adrian

Open Apps in Open English for Open Data
Online at www.executable-english.com 
Shared use is free, no advertisements

--

All contributions to this forum by its members are made under an open content license, open publication license, open source or free software license. Unless otherwise specified, all Ontolog Forum content shall be subject to the Creative Commons CC-BY-SA 4.0 License or its successors.
---
You received this message because you are subscribed to the Google Groups "ontolog-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.
To post to this group, send email to ontolo...@googlegroups.com.
Visit this group at https://groups.google.com/group/ontolog-forum.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/8942fe91ef1f4820830a3307a050ab15%40utdallas.edu.

[John Bottoms]

JohnS,

Yes, I agree with 1 and 2 below. The first being a response in the sense
of syntax. The 2nd involves more rules and information processing that
exceeds strict syntax.

It is my understanding that Jill/Watson works in the syntax domain and
has no understanding of knowledge structure beyond similarities. In that
sense the question of "Why" that question remains in the syntax domain
and can be answered using the same approach Jill uses for other
questions. I don't see any translation of knowledge here.

-John Bottoms

[Alex Shkotin]

and just to add

0. an answer is a reference to source. So on "Why?" Watson@IBM may answer "It's written in Wikipedia" or child may answer "My daddy said so" - this is a power of factology. This is about a facts-system, not FAQ one.

Alex

--
All contributions to this forum by its members are made under an open content license, open publication license, open source or free software license. Unless otherwise specified, all Ontolog Forum content shall be subject to the Creative Commons CC-BY-SA 4.0 License or its successors.
--- You received this message because you are subscribed to the Google Groups "ontolog-forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ontolog-foru...@googlegroups.com.
To post to this group, send email to ontolo...@googlegroups.com.
Visit this group at https://groups.google.com/group/ontolog-forum.

To view this discussion on the web visit https://groups.google.com/d/msgid/ontolog-forum/bb5c4644-724c-ad43-8db3-476f684a9f9b%40bestweb.net.