Expert systems: what happened in the past decade?
Paolo Amoroso
so? Are they still used? Are they just another one of those "once it
works, it's not AI" technologies? Have expert systems been abandoned,
or replaced by something else?
Some context about my question. I have been reading about, and
experimenting with, expert systems for the past few months. I bought
from Amazon half a dozen well know books such as "Expert Systems -
Principles and Programming" by Giarratano, "Programming Expert Systems
in OPS5" by Brownston et al., "Building Expert Systems" by Hayes-Roth
et al., and a few more.
Although I have a basic knowledge of computing and AI history, I did
not closely follow recent AI research and applications. Many early
expert systems books contain a lot of what might now be considered
hype. But one of the books I bought, Feigenbaum's "The Rise of the
Expert Company", published in 1989, seem to tell several stories of
how this technology improved the bottom line of many companies.
To someone not deeply familiar with AI like me, all this seems to stop
around the AI winter. So, I always wondered what happened to expert
systems since then. I got Feigenbaum's book from Amazon at 0.01$ plus
shipping: is this any indication of the fate of expert systems? :)
Are expert systems alive and well?
Paolo
--
Lisp Propulsion Laboratory log - http://www.paoloamoroso.it/log
[ comp.ai is moderated. To submit, just post and be patient, or if ]
[ that fails mail your article to <com...@moderators.isc.org>, and ]
[ ask your news administrator to fix the problems with your system. ]
alex goldman
> Expert systems: what happened in the past decade?
People turned away from the theoretically unsound "certainty factors" in
favor of probabilistic reasoning and started calling their systems Bayesian
Networks instead of Expert Systems.
Ted Dunning
areas.
In financial fraud detection, particularly the problem of finding
credit card transaction fraud, expert systems were an early candidate.
Based on experts' experience, rules were posed which characterized
known fraud and the output on unseen data was used to figure out how
the rules would go wrong. These systems did as well as the state of
the art at the time which was considerably worse than the state of the
art now. That state of the art was defined by supervised training
algorithms running against specially defined and somewhat intricate
features of transactions and history.
The interesting and important thing that happened is that the fraud
characteristics evolved over time as, indeed, things in the real world
tend to do. The rule developers continued to add rules but as they
reached systems with hundreds to thousands of rules, the interactions
between the rules became the dominant determinant of the behavior of
the system. As a result, these systems became harder and harder to
maintain. The systems based on supervised learning had no such
difficulties; as new features were proposed or as the world shifted,
they simply retrained their model and performance was maintained and
the level of complexity did not increase.
Ultimately, the rules systems suffered so severely from bit-rot that it
became cheaper to buy the services of a trained system. One company
in particular, HNC Software, did a good enough job in this area so that
they eventually dominated the market for credit card fraud detection
software and services. Eventually, CITI was just about the only place
that a system based primarily on large numbers of human designed rules
persisted, and that was largely due to a large staff of modelers who
developed special techniques to allow them to retire old rules and
manage the interactions.
The really important lesson from all of this is not that rules are bad
and neural nets are good. The HNC system in the end had a rule
post-processor which was critical in assuring that the system could
comply with various regulatory and business requirements. For example,
there is a limitation period in the US after a first contact about a
possible fraud during which a customer cannot be contacted about the
same issue. Getting a neural net to understand and comply with
arbitrary and stiff requirements like this is a losing battle.
The features that were inputs into the system were also examples of
very simple rules if you define any hand-coded processing as a rule.
For example, you could define apparent velocity as the ratio of
distance and time between transactions and set a hard or soft threshold
for what constitutes "high" velocity. You could do the same to define
"large" amounts of cash advances. These might make very good input
variables for a learning machine, but you could also make the case that
they are just rules with soft outputs.
The difference, then, is really one of emphasis rather than of essence.
The systems that ultimately failed depended on large numbers of fairly
complex rules with binary outputs that were combined using human
specified rules in the style of expert systems from the 1980's. They
failed because the combination step quickly became unmaintainable when
they reached the software-engineering break-even where each bug fix
introduces as many new bugs as it removes old bugs.
The systems that succeeded used very simple rules with soft outputs
combined using a learning system that weighted these inputs and their
interactions so as to optimize performance. The commonly repeated
nostrum about neural nets and similar learning algorithms being
"uninspectable" as compared to traditional rule-based systems was shown
to be completely misguided because supervised learning systems often
provide an ability to quantify the average benefit of any particular
input. This ability to diagnose which inputs were useful and which
counter-productive combined with the ability to learn new patterns is
what really prevented bit-rot from making these systems progressively
less effective.
At this point, the lessons of history are pretty clear, at least with
respect to fraud detection. You can't do without rules in such a
problem, at least in the form of feature detectors, but these rules
have to be kept exceedingly simple and must be rigorously evaluated for
an benefit. You also absolutely must have a very sophisticated way of
combining the output of these rules that allows you to determine net
benefit and blame. Finally, you really will need some sort of ability
to impose business rules on the outputs.
I should point out that the market has spoken in other ways as well.
According to Amazon, at least, Feigenbaum's book is now literally not
worth the paper it is printed on. That is a pretty harsh judgement and
I think that he had some interesting things to say at the time, but we
have learned a lot since then. Peter Norvig's book, on the other hand,
is still selling. Conclude what you will, but I think that the fact
that he put in chapters on reasoning in the face of uncertainty and on
learning systems is a pretty major component of his success.
Nameless
news:422bc6e5$1...@news.unimelb.edu.au...
> What happened to expert systems in industry within the past
> decade or so? Are they still used? Are they just another one
> of those "once it works, it's not AI" technologies? Have
> expert systems been abandoned, or replaced by something else?
>
> Some context about my question. I have been reading about,
> and experimenting with, expert systems for the past few months.
> I bought from Amazon half a dozen well know books such as
> "Expert Systems - Principles and Programming" by Giarratano,
> "Programming Expert Systems in OPS5" by Brownston et al.,
> "Building Expert Systems" by Hayes-Roth et al., and a few more.
>
> Although I have a basic knowledge of computing and AI history,
> I did not closely follow recent AI research and applications.
> Many early expert systems books contain a lot of what might now
> be considered hype. But one of the books I bought, Feigenbaum's
> "The Rise of the Expert Company", published in 1989, seem to
> tell several stories of how this technology improved the bottom
> line of many companies.
>
> To someone not deeply familiar with AI like me, all this seems
> to stop around the AI winter. So, I always wondered what
> happened to expert systems since then. I got Feigenbaum's book
> from Amazon at 0.01$ plus shipping: is this any indication of
> the fate of expert systems? :)
> Are expert systems alive and well?
Of course they are, and their use is growing. One book you
didn't mention but which is probably the most informative
w.r.t. expert system use in industry or otherwise is
The Handbook of Applied Expert Systems
CRC Press
ISBN 0-8493-3106-4
HTH.
Bob Bechtel
> What happened to expert systems in industry within the past decade or
> so? Are they still used? Are they just another one of those "once it
> works, it's not AI" technologies? Have expert systems been abandoned,
> or replaced by something else?
>
> Some context about my question. I have been reading about, and
> experimenting with, expert systems for the past few months. I bought
> from Amazon half a dozen well know books such as "Expert Systems -
> Principles and Programming" by Giarratano, "Programming Expert Systems
> in OPS5" by Brownston et al., "Building Expert Systems" by Hayes-Roth
> et al., and a few more.
>
> Although I have a basic knowledge of computing and AI history, I did
> not closely follow recent AI research and applications. Many early
> expert systems books contain a lot of what might now be considered
> hype. But one of the books I bought, Feigenbaum's "The Rise of the
> Expert Company", published in 1989, seem to tell several stories of
> how this technology improved the bottom line of many companies.
>
> To someone not deeply familiar with AI like me, all this seems to stop
> around the AI winter. So, I always wondered what happened to expert
> systems since then. I got Feigenbaum's book from Amazon at 0.01$ plus
> shipping: is this any indication of the fate of expert systems? :)
> Are expert systems alive and well?
>
>
> Paolo
Grumble. If you'll allow a quick definitional move, I'll suggest that
what was called "expert systems" in the books you've been reading would
more correctly be termed "rule-based systems." Now, that's not
completely true (for example, Carl Engelman insisted that Macsyma is an
expert system, even though it doesn't have rules in the conventional
sense, but it does have expertise), but I think that it may suffice for
your current purposes.
Given that, I'll claim that it's partly "once it works, it's not AI,"
partly a name change ("rule-based" rather than "expert"), partly that
(like almost every other technology) rules alone usually aren't enough,
so the tools have gotten more complex and may be described differently.
There are numerous tools available, both commercial and non-commercial.
Some of them can be traced directly back to the giants of AI summer -
Gensym Corporation makes G2; Haley Systems has an OPS-descendant; as I
recall, there's still a descendant of ART*Enterprise going strong as
well, mostly in banking and insurance (I think).
A separate family holds CLIPS (originally NASA), JESS, a Java descendant
of CLIPS from Sandia, and LISA for Common Lispers.
In another renaming, rule-based technology has also been recast as
"business rules" - see BPEL, workflow, process control, etc.
You might look at comp.ai.shells (as in expert system shells or
rule-based shells) - while not super active, there is occasional useful
information.
There are also those that would argue that (if you squint a bit) logic
programming is just rules, opening up Prolog, constraints, and all those
wonderful technologies.
Rules are definitely alive. In the past year my company delivered a
custom application that used CLIPS to reason about information in
military messages - the use of rules made it possible for the customer
to tweak things to accommodate new message types without having to do
recompiles or have a full software development environment (Visual
Studio - they're a Microsoft shop) around.
bob bechtel
P.S. "The Rise of the Expert Company," being more a business book than a
technology book, has suffered the fate of most business books - after a
year or so, there's a new fad and a new book to read. I suspect his
Fifth Generation book is equally inexpensive.
Randolph M. Jones
> What happened to expert systems in industry within the past decade or
> so? Are they still used? Are they just another one of those "once it
> works, it's not AI" technologies? Have expert systems been abandoned,
> or replaced by something else?
>
> Some context about my question. I have been reading about, and
> experimenting with, expert systems for the past few months. I bought
> from Amazon half a dozen well know books such as "Expert Systems -
> Principles and Programming" by Giarratano, "Programming Expert Systems
> in OPS5" by Brownston et al., "Building Expert Systems" by Hayes-Roth
> et al., and a few more.
>
> Although I have a basic knowledge of computing and AI history, I did
> not closely follow recent AI research and applications. Many early
> expert systems books contain a lot of what might now be considered
> hype. But one of the books I bought, Feigenbaum's "The Rise of the
> Expert Company", published in 1989, seem to tell several stories of
> how this technology improved the bottom line of many companies.
>
> To someone not deeply familiar with AI like me, all this seems to stop
> around the AI winter. So, I always wondered what happened to expert
> systems since then. I got Feigenbaum's book from Amazon at 0.01$ plus
> shipping: is this any indication of the fate of expert systems? :)
> Are expert systems alive and well?
Depending on what you're willing to call an expert system, this article
describes one of the directions they went in the last 10 years:
Jones, R. M., Laird, J. E., Nielsen, P. E. Coulter, K. J., Kenny, P. G.,
& Koss, F., (1999). Automated intelligent pilots for combat flight
simulation. AI Magazine, 20(1), 27-41.
Paolo Amoroso
> Of course they are, and their use is growing. One book you
> didn't mention but which is probably the most informative
> w.r.t. expert system use in industry or otherwise is
>
> The Handbook of Applied Expert Systems
> CRC Press
> ISBN 0-8493-3106-4
I wasn't aware of this, thanks for the suggestion. I profit by the
occasion to also thank all those who contributed information and
comments--very interesting--to this thread.
Paolo
--
Lisp Propulsion Laboratory log - http://www.paoloamoroso.it/log
[ comp.ai is moderated. To submit, just post and be patient, or if ]
Lynn Hales
wrote:
>What happened to expert systems in industry within the past decade or
>so? Are they still used? Are they just another one of those "once it
>works, it's not AI" technologies? Have expert systems been abandoned,
>or replaced by something else?
>
>Some context about my question. I have been reading about, and
>experimenting with, expert systems for the past few months. I bought
>from Amazon half a dozen well know books such as "Expert Systems -
>Principles and Programming" by Giarratano, "Programming Expert Systems
>in OPS5" by Brownston et al., "Building Expert Systems" by Hayes-Roth
>et al., and a few more.
>
>Although I have a basic knowledge of computing and AI history, I did
>not closely follow recent AI research and applications. Many early
>expert systems books contain a lot of what might now be considered
>hype. But one of the books I bought, Feigenbaum's "The Rise of the
>Expert Company", published in 1989, seem to tell several stories of
>how this technology improved the bottom line of many companies.
>
>To someone not deeply familiar with AI like me, all this seems to stop
>around the AI winter. So, I always wondered what happened to expert
>systems since then. I got Feigenbaum's book from Amazon at 0.01$ plus
>shipping: is this any indication of the fate of expert systems? :)
>Are expert systems alive and well?
>
>
>Paolo
Check www.kscape.com.
We market and sell a very comprehensive AI system for real-time
control of mineral processing plants. We're as busy as we ever have
been. Lynn
Ted Dunning
KnowledgeScape? advanced process control software integrates
fuzzy logic, neural
networks, genetic algorithms, statistical process control
techniques and client server
architecture to provide an easy to use, on line, real time, global
optimization solution to
the process control industry.
Notice how little credit the rules-based heritage gets. This is pretty
ubiquitous in my experience; the rules part of it all doesn't get you
very far at all, but you have to have it around for the regulatory or
business logic parts where non-compliance is just not allowed.
Another interesting example of this was the recommendation engine that
I created for Musicmatch (now Yahoo! Music). The first use of the
system was to link web pages and recommend music. This worked well
with raw statistical AI as the output. The next use, however, was to
program radio stations. The statistical relatedness measures could
produce nice listenable radio stations after you tweaked them a bit by
putting in some ad hoc measures to encourage variety, but there was no
way that they would meet the requirements of the compulsory licenses
under the DMCA.
Thus, we hacked on a generic sort of rule engine that augmented the
statistical scoring with large penalties to prevent the random search
from violating the law.
The general principle is that rules alone will get you compliance with
some nasty little details, but the bits that can reasonably be called
"expertise" are best captured by other means.
Paolo Amoroso
> Check www.kscape.com.
>
> We market and sell a very comprehensive AI system for real-time
> control of mineral processing plants. We're as busy as we ever have
Your company's product, and what has been said in this thread, seem to
suggest that current systems are no longer purely rule-based (have
they ever been?), but are hybrid and rely also on neural, fuzzy and
other techniques.
Is this correct? Does it apply only to large systems?
If I recall correctly from my readings, classic (i.e. pre-AI winter)
expert systems were considered large when they contained more than a
few thousand rules. But systems with hundreds of thousand rules have
been mentioned in this thread.
Paolo
--
Lisp Propulsion Laboratory log - http://www.paoloamoroso.it/log
[ comp.ai is moderated. To submit, just post and be patient, or if ]
Ted Dunning
are useful for the essentially trivial parts of the system and the
learning components are useful for the content aspects.
Having worked on systems with thousands of rules in an academic setting
I would run, not walk from any project that tried to field a large
rule-based system in any practical setting.
Take as an example a large financial transaction processing company
that I had a hand in working with once. They had monthly updates to
their procedures manuals. They also had a team of 7 "knowledge"
engineers who were charged with reducing this material to a logical
form. The knowledge team never came close to being up to date.
Generally, logical updates were so late that by the time they became
available, they were old news and had often been superceded by
additional changes.
This was the most successful instance of a traditional AI project that
I have ever encountered personally. It didn't exactly blow apart. It
just never had any positive net value and wound up being completely
replaced. As it turned out, a text retrieval system could give people
access to the information they needed with a minimally complex system.
Contrast this with a system that I built over a decade ago. The task
was to look at radar returns and determine when birds were about to
land on a cooling pond next to a power plant. Since the cooling plant
had high levels of dissolved salts and such, the birds have to warned
off of landing on the pond. The system used standard image processing
techniques, some minor learning systems and about 5 rules for
increasing variety in response. The same system has been running
successfully without change for the last 11 years.
Anyway, my guiding principle has always been to avoid asking for
trouble. I avoid building large rule systems if I possibly can.
Lynn Hales
is crisp feedforward rules. Many use fuzzy rules in certain areas of
their control. The two are totally integrated.
Our system allows for limitless neural networks to be defined to model
the processes we control. New vector acquisition, training and
predicting all happen concurrently. The built in genetic algorithms
are used to answer the general question " given where I am at what set
point changes should I make to the process to improve where I'm going
to be in the near future"? The results of the neural network
predictions can be used in the rules, by the GA optimizers, trended
etc. Very flexible and powerful.
Rules are the core I would say. Regards, Lynn
On Thu, 10 Mar 2005 01:47:29 GMT, Paolo Amoroso <amo...@mclink.it>
wrote:
>Lynn Hales <lha...@xmission.com> writes:
>
>> Check www.kscape.com.
>>
>> We market and sell a very comprehensive AI system for real-time
>> control of mineral processing plants. We're as busy as we ever have
>
>Your company's product, and what has been said in this thread, seem to
>suggest that current systems are no longer purely rule-based (have
>they ever been?), but are hybrid and rely also on neural, fuzzy and
>other techniques.
>
>Is this correct? Does it apply only to large systems?
>
>If I recall correctly from my readings, classic (i.e. pre-AI winter)
>expert systems were considered large when they contained more than a
>few thousand rules. But systems with hundreds of thousand rules have
>been mentioned in this thread.
>
>
>Paolo
[ comp.ai is moderated. To submit, just post and be patient, or if ]
Lynn Hales
We, at KnowledgeScape, primarily work in the mineral processing
industry. Here our systems can contain hundreds of rules, many neural
network models with GA optimization. Rules, both crisp and fuzzy
rules certainly are the core however.
We don't run into any corporate compliance issues that require rules
versus other technologies. Rules are certainly understandable
however.
Regards, Lynn
Ted Dunning
> versus other technologies.
Lucky you. I would expect that you still have limit conditions that
would, for example, require plant shutdown. Mechanical systems tend to
have limit stops that serve the same function.
> Rules are certainly understandable however.
In isolation, they appear to be simple and comprehensible. In a large
mass with significant interactions and prioritization, however, they
are definitely not understandable.
In fact, the combination of apparent simplicity and actual
impenetrability is very dangerous.
Lynn Hales
<ted.d...@gmail.com> wrote:
>Yes, all of these systems have rule and learning components. The rules
>are useful for the essentially trivial parts of the system and the
>learning components are useful for the content aspects.
Our KnowledgeScape systems, controlling mineral processing plants are
anything but trivial. On average our systems are running and
controlling entire plants 95 to 100 percent of the time (24 x 7 x
365). New process set points are calculated every 15 seconds to 2
minutes on average and proceed to be implemented by the underlying
plant control system.
Each of these systems receives upto 2000 new process data points at
least every minute. Process data comes from sensors, instruments,
actuators , analyzers, real-time images, etc.
Our systems run in the largest minerals plants throughout the world
and are well known to increase the yield of the plant in many cases of
well over $100,000 per day.
>
>Having worked on systems with thousands of rules in an academic setting
>I would run, not walk from any project that tried to field a large
>rule-based system in any practical setting.
The success of our systems is so well known that all operators of
these types of minerals plants are not running from expert control but
have long since embraced it and continually expand their use within
their facilities.
>
>Take as an example a large financial transaction processing company
>that I had a hand in working with once. They had monthly updates to
>their procedures manuals. They also had a team of 7 "knowledge"
>engineers who were charged with reducing this material to a logical
>form. The knowledge team never came close to being up to date.
>Generally, logical updates were so late that by the time they became
>available, they were old news and had often been superceded by
>additional changes.
We're lucky that we don't have to deal with the generation of quality
input data like you mention here. We certainly have to deal with
poor, broken, poorly calibrated sensors but this sounds easy compared
to the human requirement you mention.
>
>This was the most successful instance of a traditional AI project that
>I have ever encountered personally. It didn't exactly blow apart. It
>just never had any positive net value and wound up being completely
>replaced. As it turned out, a text retrieval system could give people
>access to the information they needed with a minimally complex system.
>
>Contrast this with a system that I built over a decade ago. The task
>was to look at radar returns and determine when birds were about to
>land on a cooling pond next to a power plant. Since the cooling plant
>had high levels of dissolved salts and such, the birds have to warned
>off of landing on the pond. The system used standard image processing
>techniques, some minor learning systems and about 5 rules for
>increasing variety in response. The same system has been running
>successfully without change for the last 11 years.
Two our systems we like to brag about are in the jungle of the
Philippines which has been running continuously since we installed it
in 1987 as well as another located in an isolated mountainous area of
Mexico which has also been running continuously since 1986. I still
can't believe the Industrial IBM PC's we used in these cases, running
OS2, are still ticking, but they are.
Many of our systems go untouched after installation but for many more
we provide quarterly software updates to as well as plant visits to
monitor, turn, tweak and provide additional training. As an
interesting aside the project in the Philippines took six months to
implement which was extremely fast for the state of the industry at
that time. Last fall we installed a larger and more complex version
of our system in a plant adjacent to the rain forest of northern
Brazil in a total of two weeks.
>
>Anyway, my guiding principle has always been to avoid asking for
>trouble. I avoid building large rule systems if I possibly can.
We agree totally, some of our earlier systems had four times as many
rules as we routinely use nowadays. I guess practice makes perfect,
more or less. :) Lynn
Lynn Hales
<ted.d...@gmail.com> wrote:
>> We don't run into any corporate compliance issues that require rules
>> versus other technologies.
>
>Lucky you. I would expect that you still have limit conditions that
>would, for example, require plant shutdown. Mechanical systems tend to
>have limit stops that serve the same function.
Certainly, we have many physical and process limits. Based on feed
variability what if any of these limits actually contribute to new set
point changes continually changes. We often talk in terms of one role
the expert system has is to push the operation to a limit. As
conditions change the system moves away from that limit and then
pushes you towards another. We always track the percent of time we
are at each limit and provide that information to management so they
can consider capital expenditures that would reduce or eliminate a
given limiting situation.
>
>> Rules are certainly understandable however.
>
>In isolation, they appear to be simple and comprehensible. In a large
>mass with significant interactions and prioritization, however, they
>are definitely not understandable.
Ted we have greatly overcome the negative potential you refer to as
systems get larger and larger by a best practices use of
KnowledgeScape. Our system is a very intuitive object oriented system
that allows users to group their rules in a hierarchical way that is
based on the physical layout of the plant. It would take a long time
to totally explain this but this potential problem is really
non-existent in the systems we implement. Our standard rule sets for
the processes we control have been refined since 1976 so we feel they
are very tractable. Lynn