What is the main problem with learning in AI? (Especially to Curt Welch)
Burkart Venzke
You have an interesting but very long discussion in "No easy road to
AI". As we want to solve the problem of AI I also think that learning is
the central problem of it.
What do you think are the main problems about learning?
One interesting point I have read in the beginning of your discussion is
the connection of learning and perception and that an AI also has to
learn perception. Yes, it is important.
Another point is how far the internal representation of the AI system is
important. I think that a least we want to really write software we have
to think about the internal representation. But ok, we may have more
important problems before.
For me, (other) central points are
- how to motivate an AI to learn without programming it precisely
- what it really have to learn
- the basics of learning, philosophically seen. "What is learning?" etc.
Burkart
casey
Hi Curt, looks like you have won a convert ;)
JC
> Hi Curt Welch, hi all,
>
> You have an interesting but very long discussion in
> "No easy road to AI". As we want to solve the problem
> of AI I also think that learning is the central
> problem of it.
>
>
> What do you think are the main problems about learning?
>
>
> One interesting point I have read in the beginning of
> your discussion is the connection of learning and
> perception and that an AI also has to learn perception.
> Yes, it is important.
And yet many animals are up and running without taking
the time to learn to perceive.
> Another point is how far the internal representation
> of the AI system is important.
>
>
> I think that a least we want to really write software
> we have to think about the internal representation.
> But ok, we may have more important problems before.
>
>
> For me, (other) central points are
>
> - how to motivate an AI to learn without programming
> it precisely
Motivations have to be precisely programmed. In biological
systems this was done by an evolutionary process. To motivate
simply means to get moving. You "motivate" your automobile
with a press of the accelerator pedal. In brains different
kinds of inputs can "press" the brain's accelerator pedal.
We train animals by providing inputs that we know will cause
their accelerator pedal to be pressed.
> - what it really have to learn
That depends on its goal. Curt would say it has to learn
to maximize a reward signal.
> - the basics of learning, philosophically seen.
> "What is learning?" etc.
Learning is a change in the system that results in its
ability to do things it couldn't do before.
JC
Burkart Venzke
> On Jan 11, 3:44 am, Burkart Venzke <b...@gmx.de> wrote:
>
> Hi Curt, looks like you have won a convert ;)
>
> JC
Why "convert"? ;)
>> Hi Curt Welch, hi all,
>>
>> You have an interesting but very long discussion in
>> "No easy road to AI". As we want to solve the problem
>> of AI I also think that learning is the central
>> problem of it.
>>
>>
>> What do you think are the main problems about learning?
>>
>>
>> One interesting point I have read in the beginning of
>> your discussion is the connection of learning and
>> perception and that an AI also has to learn perception.
>> Yes, it is important.
>
> And yet many animals are up and running without taking
> the time to learn to perceive.
Perception is not a sufficent condition for intelligence, only
(something like) a necessary one.
>> Another point is how far the internal representation
>> of the AI system is important.
>>
>>
>> I think that a least we want to really write software
>> we have to think about the internal representation.
>> But ok, we may have more important problems before.
>>
>>
>> For me, (other) central points are
>>
>> - how to motivate an AI to learn without programming
>> it precisely
>
> Motivations have to be precisely programmed. In biological
> systems this was done by an evolutionary process. To motivate
> simply means to get moving. You "motivate" your automobile
> with a press of the accelerator pedal. In brains different
> kinds of inputs can "press" the brain's accelerator pedal.
> We train animals by providing inputs that we know will cause
> their accelerator pedal to be pressed.
Acknowledge.
I think that motivations are a precondition for general learning.
That's why I've mentioned them.
>> - what it really have to learn
>
> That depends on its goal.
Right.
For me, a very small set of goals is the base of a (real/strong) AI
(originally thinking of Asimov's three rules for robots).
My two goals/rules are:
1. Do what the AI's owner(-s; manhood) wants it to do.
2. Learn as much and good as it can.
The rules should be a start for "real" AI. Later, better (more
(precizely)?) rules will be necessary, especially when it acts in the
real world.
> Curt would say it has to learn to maximize a reward signal.
A reward signal is also important for me, even a quite direct one.
The maximization seems to be the best method as long as the AI has no
better possibilities to know what is good (better, right...).
>> - the basics of learning, philosophically seen.
>> "What is learning?" etc.
>
> Learning is a change in the system that results in its
> ability to do things it couldn't do before.
Yes, that was nearly the definition of my prof decades ago ;)
(I think he said something like "to do s.th. better than before".)
It is nice but not enough for me.
Who (or what) decides which things are those?
Who (or what) decides if "it could now" is true?
OK, sometimes it is easy to answer these questions but sometimes it
isn't, especially when human ideas etc. are involved (moral...).
Burkart
Curt Welch
> Hi Curt Welch, hi all,
>
> You have an interesting but very long discussion in "No easy road to
> AI".
Yes, I tend to ramble on endlessly.
> As we want to solve the problem of AI I also think that learning is
> the central problem of it.
>
> What do you think are the main problems about learning?
The implementation details. :)
> One interesting point I have read in the beginning of your discussion is
> the connection of learning and perception and that an AI also has to
> learn perception. Yes, it is important.
>
> Another point is how far the internal representation of the AI system is
> important.
It's highly important. You could say it's everything.
> I think that a least we want to really write software we have
> to think about the internal representation. But ok, we may have more
> important problems before.
Well, I guess you need to be clear about what you mean by the internal
representation. Representation of what? Of the environment? Of the
learning system? There are lots of internal represdnations you have to
get "right" to solve AI.
> For me, (other) central points are
> - how to motivate an AI to learn without programming it precisely
> - what it really have to learn
> - the basics of learning, philosophically seen. "What is learning?" etc.
>
> Burkart
Yeah, those are all basic questions that been been debated since the very
begin of AI and many also for far longer in areas like psychology.
If you look up learning theories in psychology, the lists you find will
seem endless. They are all attempts to answer the question of "what is
learning?"
To solve AI (make a computer act like a human), we have to figure out what
type of learning to build into the machine. Do we need 20 types of
learning modules working together to create all the types of learning
humans have been documented to contain? Or is there a smaller list of
learning abstractions that can cover all learning?
I believe reinforcement learning is the only important type of learning we
have to correctly implement to solve AI, and that all other forms of
learning fall out as learned behaviors of a strong reinforcement learning
machine. Reinforcement learning (a field of machine learning which is a
field of AI and Computer Science) is basically the same thing as operant
conditioning in psychology (The work of Skinner and other Behaviorists).
It's also closely related to genetic algorithms in AI - which is basically
a specialized sub-class of reinforcement learning in my view - though I
don't see others making that connection.
Before these terms became better formalized, reinforcement learning was
also generically known as trial and error learning since it's a method of
learning by experience.
If you are not familiar with reinforcement learning:
http://en.wikipedia.org/wiki/Reinforcement_learning
Sutton's Book which is on line here:
http://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html
On you quesiton about what is learning, it's just the fact that human
behavior is not constant. It changes over our life time. So at the most
abstract layer, we can just say that learning is a change in behavior. But
that gives us no answer as to the direction of change. Why might one
change in behavior be called learning, and another be called random
nonsense? It's because learning typically assumes a change for a purpose,
or a change for "the better".
But when we say these things, like "change for the better" we have just
pushed the question of what is learning, off to a problem of defining what
we mean by better. Reinforcement learning answers that questions.
Other answers people have used to the question is to describe learning
behavior as goal seeking. That is, our behavior changes to better reach
some goal. But then we are left asking the question of what a goal is, or
what the goal should be for our AI solution. Reinforcement learning
abstracts the definition of what the goal is to reward maximizing.
Reinforcement learning frames the problem of learning as an agent, which
interacts with an environment. Which means it has inputs from the
environment (sensors) and outputs to the environment (effectors). But in
addition, it also receives a single dimension reward signal from the
environment.
The goal of the agent in this abstract framework, is to maximize some long
term measure of the reward signal. There is no explicit end-goal in
reinforment learning. That is, learning never ends. The assumption is the
agent will continue to try and find ways to get more rewards - even though
there might not be any better ways to get rewards than what it's currently
doing.
Though we say in this framework that the reward comes from the environment,
when we build reinforcement learning into something like a robot, we also
have to build the hardware which generates the reward signal. But that
hardware, is conceptually outside the "reinforcement learning hardware"
module. From the perspective of the learning module, the reward comes from
the environment of the learning module.
The problem with reinforcement learning, is that's it's easy to specify,
but extremely hard to implement. In fact, no one has implemented a good
generic reinforcement learning machine.
There are many reinforcement learning algorithms that have already been
created. Most however, only work (or are only practical) for small toy
environments - like the game of tic tac toe or something similar. This is
because they require the algorithm to track a statistical value for every
state the environment can be in. When there are only thousands of states,
like the number of possible board positions in tic tac toe, and the agent
has enough time to explore all the states many times, then the current
algoerithms can converge on optimal (perfect) decisions fairly quickly.
But as the environment becomes more complex (more states it can be) the
simple RL algorithms fail - because it quickly becomes impossible to track
a value for each possible state, (not enough memory in the world to track a
value for every board game in Go). So some other approach has to be used.
These are called high dimension problem spaces because the number of states
of the sensory inputs have an effectively how number of dimensions - aka
multiple sensors acting in parallel - which just means the total state
space is huge.
No one has found a good generic solution to this problem but many people
are looking for them.
Here's a fairly recent paper on one approach to trying to apply RL to
robotics which discusses the issues, and the solutions they attempted to
explore in this paper:
http://www.cnbc.cmu.edu/~jp/research/publications/b2hd-provost-phd07.html
To make a workable system, that person ended up creating at least 4
different modules in his machine. One only was the module where the
reinforcement learning was done. The others, were defectively just
"adapters" which mapped the hard problem down to one which the simpler RL
algorithms could deal with.
Though the approach in the paper demonstrates how you cam use low dimension
RL algorithms in a high dimension environment and do some good with it,
it's not the correct solution yet.
When implemented correctly, I believe we will basically have solved AI.
So back to yiour questions, in reverse order:
> - the basics of learning, philosophically seen. "What is learning?" etc.
It's a reinforcement learning agent that can operate in the high dimension,
continuous, real time, problem space of the real world. To solve AI, we
have to build one of these machines that can learn as well in this
environment as a typical human brain can. It's imporant to understand
however that the RL problem we solve as humans, has no perfect answer, so
there is not perfect solutions. The brain simply does the best it can with
what it has to work wirth. A "perfect" (but impossible) solution would
require the brain (agent) to make perfect us of everything it has ever
sensed in making it's output decisions. To solve Ai, we don't have to
create a perfect solutions, we just have to build a learning machine that
can learn as much, and as fast, as a typical human. None of the current RL
algorithms come anywhere near human level learning for the general domain,
but many work better than humans in very small limited domains.
> - what it really have to learn
Trial and error learning where behavior is adjusted so as to maximize a
reward signal while the agent is interacting with the the environment (on
line learning).
> - how to motivate an AI to learn without programming it precisely
By building the hardware which generates the reward signal. This can be
very simple, or very hard, depending on what sort of motion we are
attempting to give it. Human rewards (and punishment - aka negative
rewards) can be understood as the normal low level conditions that we sense
as pain and pleasure.
> [how does perception fit into the answer]
To make RL work in a real world environment, we have to map complex, high
volume, high dimension sensory data, into some internal representation of
the environment (state), and at the same time, map state to actions some
how. We then have to statistical track state->action pairs relative to
rewards, to determine what actions, from what state, has produced the
highest average rewards in the past. Behavior is then adjusted to favor
the actions that produce the highest rewards.
Perception is the problem of creating the internal state which is the
systems representation of the current state of the environment. In trivial
RL domains, like tic tac toe, there is no perception problem. The board
position is the current state of the environment. But in hardware domains,
where we have a video feed from a robot eye, mapping that to some
simplified internal state of the environment, which we can then use to
track actions, is non trivial.
To create a generic solution to AI, we have to find systems that create the
internal representation automatically from the raw sensory data, without
having to use the intelligence of a human engineer to hard-code a special
case solution for each job we want our robot to perform. In the above
paper, the guy used a form of self organizing maps to solve the perception
problem for his application.
The devil is in the details here. How you implement a solution to the RL
problem, is the question of AI in my view. If you find the right
implementing, you will have solved AI.
Not everyone agrees with this framing of the AI problem, but it's the
framing I believe to be the one and only possible way to correctly frame
it. Nothing else can explain, or produce, the strong generic adaption
skills human have.
--
Curt Welch http://CurtWelch.Com/
cu...@kcwc.com http://NewsReader.Com/
Daryl McCullough
>I believe reinforcement learning is the only important type of learning we
>have to correctly implement to solve AI, and that all other forms of
>learning fall out as learned behaviors of a strong reinforcement learning
>machine.
It seems to me that there is a stage of learning that must happen before
any kind of reinforcement learning is possible, which is concept formation.
What I mean is this: in reinforcement learning, the learner is trying to
figure out a *policy* for dealing with the environment. In other words,
the learner is trying to discover a policy, or set of rules of the form:
If you are in situation S_i, then perform action A_j (in reinforcement
learning, the learner is trying to discover a policy that maximizes some
notion of expected reward).
But for a robot (or AI program) dealing with
the real world, the set of possible situations is limitless (or at least,
too large to effectively enumerate) as is the set of possible actions.
Further, no situation precisely repeats itself; there's always some
slight difference between the current situation and any situation that
has every occurred in the past. Similarly, actions are never perfectly
repeated. So there is no opportunity for the learner to ever come to
a useful conclusion of the form: If such and such is true, then do this...
unless, that is, the learner uses abstraction and factoring in order to
reduce the complexity of situations and actions to a manageable number
of broad categories of types of situations and types of actions.
So before the learner can even start doing reinforcement learning,
it has to be able to categorize its environment and its own behavior.
But how does it learn to categorize correctly? You could say that it
learns categorization through reinforcement learning, as well, but
that just pushes the problem back further: how does it learn what
the various categorization strategies are?
Ultimately, it seems to me that any learner must start with some
initial strategies for categorization, and must use these in order
to develop descriptions of the world in terms of situations and
actions. I don't think these initial strategies can be *learned*.
You have to start somewhere. What I think is the starting point
that all creatures capable of thought use is some notions of
continuity in space and time---that events physically close
together in space or time are more likely to be causally related.
I don't think that this starting point is sufficient to bootstrap
our learner to intelligence, but I think it is a pre-requisite
for any learning system.
--
Daryl McCullough
Ithaca, NY
Curt Welch
> casey schrieb:
> > Motivations have to be precisely programmed. In biological
> > systems this was done by an evolutionary process. To motivate
> > simply means to get moving. You "motivate" your automobile
> > with a press of the accelerator pedal. In brains different
> > kinds of inputs can "press" the brain's accelerator pedal.
> > We train animals by providing inputs that we know will cause
> > their accelerator pedal to be pressed.
>
> Acknowledge.
> I think that motivations are a precondition for general learning.
> That's why I've mentioned them.
Yes, if learning is change in behavior, something has to direct that
change. Why would one change be better than the million other changes not
taken? What makes the learning hardware pick one change over another?
Whatever the process the hardware uses to direct the change is what we can
all the motivation of the learning hardware.
> >> - what it really have to learn
> >
> > That depends on its goal.
>
> Right.
> For me, a very small set of goals is the base of a (real/strong) AI
> (originally thinking of Asimov's three rules for robots).
> My two goals/rules are:
> 1. Do what the AI's owner(-s; manhood) wants it to do.
> 2. Learn as much and good as it can.
Yes, those are clearly the type of goals we would like to build in our our
AI. Those ideas correctly captures the sort of ideals we want to end up
with in our AI. But how, using transistors, can we build a detector for
"what the owner wants"? And how do you make transistors "be good"?
You have a long road to go to translate such high level abstract ideas into
some wiring of a box of transistors. To some extent, that is exactly the
road AI research has been exploring for 60 years now - the mechanism of
good and bad and wanting.
> The rules should be a start for "real" AI. Later, better (more
> (precizely)?) rules will be necessary, especially when it acts in the
> real world.
Well man as been building smart machines for a few thousand years now.
Building "rules" into the behavior of the machine has been the standard
work of engineering since forever. And as our machines get more complex
with the help of advanced electronics and computers, the number of "rules"
we build into them grows. If you want to understand how to build rules
into machines, you should first master engineering in the field you would
like to build the AI from - electronics, computers, and mechanical
engineering are typical if you want to understand if you actually want to
tackle this problem for yourself as apposed to what we do here in this
group - which is to just talk about it mostly.
Once you choose to add learning to your machines, then you are accepting
the fact that you will lose some degree of control over the machine. Once
you add learning, you have given up some of your right to specify exactly
what it does. Because of this, learning is seldom used in modern
engineering. It amounts for less than 1% the total engineering done.
Probably more like less than .001%.
None the less, learning has it's place. It allows the engineer to
abstractly step back a layer and build a goal into the machine by building
learning rules, instead of building behavior rules. The behavior rules are
then explored and adjusted by the learning rules we build into the machine.
Such things are already built into our machines in some places, but they
are normally very simple systems for very limited domain learning problems.
In AI, games have been used since the beginning as a simple environment for
exploring how to build goals into machines and let them learn to make their
decisions on their own. The goal is to win the game, and the decisions the
learning system tries to master are moves in the game. Though a game is a
highly simplified environment, it devastates very quickly how difficult it
can be to build good learning algorithms.
Currently, most engineering doesn't use learning, because the engineer is
still smarter than most of our learning algorithms. He can study the
problem, and build fixed behavior systems to solve the problem faster, than
he can build a good learning algorithm to do solve the problem for him.
But slowly, over time, our learning systems have been getting better, and
finding wider use - especially in domains where the problem is too complex
for the engineer to hand-code behavior rules to solve the problems - such
as face recognition. It's very hard to hand code a face recognizer for a
specific person. But we have learning algorithms that can learn by
example, that do a much better job.
Leaning systems are used to solve behavior problems that are too complex to
hand-code a solution for. The drawback however is that you have to teach
the machine - or at least, give it time to learn.
> > Curt would say it has to learn to maximize a reward signal.
>
> A reward signal is also important for me, even a quite direct one.
> The maximization seems to be the best method as long as the AI has no
> better possibilities to know what is good (better, right...).
In any practical application we might find the need to hand-code some
behaviors. Either because we don't want the learning system to do anything
else in that condition, or because for our application, we don't want to
wait for the system to learn it. However, hand-coding fixed behaviors is
the opposite of intelligence in my view. You either give it the freedom to
figer out behavior programs for itself using it's own intelligence, or you
use the intelligence in your own brain to decide for it. When we use our
own intelligence to solve the behavior problems (the standard practice of
all engineering), we aren't building intelligence into the machine.
> >> - the basics of learning, philosophically seen.
> >> "What is learning?" etc.
> >
> > Learning is a change in the system that results in its
> > ability to do things it couldn't do before.
>
> Yes, that was nearly the definition of my prof decades ago ;)
> (I think he said something like "to do s.th. better than before".)
> It is nice but not enough for me.
> Who (or what) decides which things are those?
> Who (or what) decides if "it could now" is true?
> OK, sometimes it is easy to answer these questions but sometimes it
> isn't, especially when human ideas etc. are involved (moral...).
>
> Burkart
Well, the problem here is that humans for the most part have no clue what
they are actually talking about when we talk about the most basic and most
abstract aspects of our own behavior. What are morals? Where to do they
come from? What is good and bad? What do those ideas really mean?
We can certainly look up these terms in a dictionary and get an answer.
But at the same time, the definitions seldom answer anything. They just
create circular definitions with other words and in the end, fails to
ground any of the concepts in anything absolute - to anything physical.
You have to be a human, to have some grounding of what good is. But good
then just becomes abstractly, "the stuff I like", and bad just becomes "the
stuff I don't like". But how does the brain make us like some things and
not others?
Because no one has yet fully solved the problem of how the brain works, we
can't yet fully answer simple questions like "what is good", or "what is
pain", or even, "what is an idea"?
But, by assuming the brain is just a reinforcement learning machine, we can
actually answer all those questions. But until we can build a
reinforcement learning machine that duplicates human behavior, we can't
prove that assumption is actually correct, or the answers are correct.
I'm so convinced that human intelligent behavior is just the emergent
behavior of a generic reinforcement learning machine than I believe all
this to be fact, not just speculation. But the data is not as convincing
to others as it is to me, so as a society, we still don't know what "good"
actually is, other than "the stuff we like".
Curt Welch
Yes, I agree completely with all the above.
Solving those problems I see as implementation details. How we implement
the learning system determines how those issues are solved. I believe
finding an implementation to just those problems are the key to solving AI
and to solving the problem of making reinforcement learning in the
difficult problem space of continuous real time behavior in the real world.
> Ultimately, it seems to me that any learner must start with some
> initial strategies for categorization, and must use these in order
> to develop descriptions of the world in terms of situations and
> actions. I don't think these initial strategies can be *learned*.
Yes, the system must start with some sort of default behavior which is then
shaped by learning. That's just generally true of how all learning systems
work.
The trick to solving this problem, is figuring out what sort of system to
start with. It's easy to pick things to start with, an in general, to pick
a machine architecture, but most architectures either 1) don't lend
themselves to be being adjusted by a reward signal or 2) don't seem to
converge on anything useful. So we have to keep searching until we find a
design that does solve all the issues you bring up.
> You have to start somewhere. What I think is the starting point
> that all creatures capable of thought use is some notions of
> continuity in space and time---that events physically close
> together in space or time are more likely to be causally related.
Yes. I bring up that idea all the time. Categories should be defined by
temporal correlations. Things that tend to show up close together in time
should be put into the same category. Or more generally, we might say
something like their category strength between events should be adjusted so
as to reflect the expected probability of the second one occurring near
time time to the first.
This idea is basically the same idea we see emerge externally as classical
conditioning in animals and humans. And I argue that classical condition
is in fact the solution to the categorizing problem.
It's also what I call the perception problem. We can't learn to love money
by reinforcement, if we don't first have hardware that can correctly
categorize raw sensory data as "money". Reinforcement learning offers no
answer to how we solve the primary categorizing problem - how the hardware
solves the perception problem. However, I strongly believe that once you
create a machine to solve the categorizing problem, you must ALSO make it
adjust it's operation, by reinforcement. YOu can't do it as two
independent modules.
The paper I posted a link to created a design that used two separate
modules, and two separate algorithms for these two problems. It used a
self organizing map to simplify the high dimension state space of the raw
sensory data into some internal state representation, and then used a
reinforcement learning algorithm on that simpler internal state space.
Though you can certainly some some simple problems like that, the full
power of the brain can't be done as two separate modules like that.
Reinforcement learning must adjust the operation of the categorizing system
to bias it's operation based on what is important. I played around for
many years with designs that did the same sort of thing the paper did by
using two modules and realized things work far far better if you do both at
the same time.
> I don't think that this starting point is sufficient to bootstrap
> our learner to intelligence, but I think it is a pre-requisite
> for any learning system.
Right. So there's really two major types of learning needed.
Reinforcement learning is what gives the learning purpose - it's the highly
abstract implementation of a goal. And category learning is what gives the
machine it's understanding of the state of the environment that it's trying
to learn how to deal with. These two problems correspond to what
behaviorists have been talking about for 60 years. They are Classical and
Operant Conditioning.
I normally only emphasize reinforcement learning, because I see classical
conditioning and this perception problem as a prerequisite of reinforcement
learning as apposed to being a separate problem. But it's also quite valid
to simply say we have two learning problem to solve here instead of just
the one.
I like to say that the category learning problem is what allows the machine
to "understand" the world, and that the reinforcement learning is what
allows the machine to know what to do with that understanding. If you are
familiar with the Jeff Hawkins book "On Intelligence" I claim that, (at
least in everything talke about in that book) he has failed to see these
two problems. All is effort is based on the first problem - how to
understand the world - or how to solve the category problem you talk about.
Without adding reinforcement learning at the same time, the sort of system
he is developing has no purpose - no goal in life - no motivation to _do_
anything with it's understanding.
So let me explain also explain how I think we can solve the problems you
bring up.
First, is the issue of the domain being high dimension. This means that
there are so many variables in the sensory data that we will never likely
see the same combination twice. And there are likewise so many dimensions
on the output side, we will are likely to never perform the exact same
behavior twice. So we can't use reinforcement with the idea we will see
the exact same sensory data twice, or the need to perform the exact same
behavior twice. So we can't assume we will learn by trying something lots
of times. We won't, in our entire life, try anything exactly the same
twice. So how do we do reinforcement learning (which is a trial and error
type of learning) in such an environment?
The answer that sounds right to me, is that we must use an architecture
that decomposes the problem down into many separate modules which work
together to produce behavior, and then apply the training to all the
modules in parallel. The idea is that when working together, the modules
never end up doing the same thing as a whole, but individual, they perform
very simple behaviors, which repeat constantly. And as such, trial and
error learning can be applied to each module individual quite well.
Abstractly, this is no different than what we do when we form a company and
get lots of people working together to solve a larger problem. Each person
perform only one simple task, and becomes an expert at that simple task
though learning and trial and error, without for the most part, every
having to understand what the rest of the organization is doing. He must
however be given reward feedback as to whether his efforts are helpful.
And there must be some overall structure that decomposes the larger
problem, down to smaller tasks.
This approach abstractly I believe solves the high dimension problem, but
leaves the implementation details of how to do it totally undefined.
Generally speaking, all ANNs (artificial neural networks) take this
approach. They use a network of many smaller units, to solve a larger
problem. Each small unit or node of the network before some simple
function, and the behavior of the network as a whole, is the emergent
behavior of the combined efforts of the smaller units working together.
Back prorogation is one strategy for distributing error information though
the network to allow each smaller unit to learn - to know how to adjust
it's behavior. Behavior is produced by data flowing forward though the
network, and training works by sending error data backwards though the
network. But that's not reinforcement learning - it's learning abstraction
though mimicking. It's supervised learning. A useful type of learning in
high dimension problems spaces, but not one that is general enough to solve
AI.
The architecture I've been playing with for many years now to try and solve
AI with, is an ANN design with a different twist. First, I've switched to
using a pulse signal abstraction (like the brain), so instead of using
nodes that act as mathematical functions which calculate an output value
from an input value, I have nodes that produce pulses. Though you can also
think of this as a node that that's producing a 0 or 1 input, where 0 is no
pulse, and 1 is a pulse, I don't use that abstraction because in addition
to using pulses, I also use async signal formats where the pulses don't
have any requirement for happening at the same time. The pulses coming
from the sensors are totally independent, and conceptually also have a high
temporal resolution. To simulate such a system using binary value
calculating nodes would mean that most the times, the inputs of all the
nodes will be all zero (as if the clock cycle of thss system war running at
100,000 cycles per second, and the pulses were showing up on the input at
about 10 per second with most the information encoded in the spacing of the
pulses.
Exploring this signal format, many years back now, I came up with the idea
of doing pulse sorting, instead of the more traditional "pulse generation".
So instead of the nodes having a function that determines when they "fire"
on a single output, I've been playing with nodes that have two outputs, and
one or more inputs, and when they receive an input pulse, they make a pulse
sorting decision, to send the pulse out one of the two outputs.
As a whole, the network will receive a stream of aysnc pulse signals from
the sensory inputs, the pulses will flow though the network, taking
different paths (but never forking into two pulses) and them emerge out one
of the output paths.
Each node in this network is performing a very simple task that it does
over and over again - making a very simple binary pulse sorting decision.
What the nodes have to learn, is a very low dimension problem - only two
actions to pick from. But the combined efforts of the nodes, makes the
network as a whole, perform a very high dimension behavior, which is the
problem of mapping X high dimension real time temporal input signals, to Y
high dimension real time temporal outputs. And again, the idea is to train
the network as q whole by r3einforment, by distributing the reward signal
to the nodes that were most responsible for the recent behaviors.
In this paradigm, what still remains unanswered is the question of what
internal logical does each node use to make it's routing decision, and how
is the reward signal distributed across all the nodes, and how does each
individual node respond to the reward signal it receives. I've played with
many different variations of that, and have some that work to some extend,
but not to the full extent they must work to solve AI.
And the other big obvious unanswered question about this pulse sorting
architecture, is whether such a network, which can't produce pulses except
when it receives one, and which can't fork a pulse into two pulses, or
can't throw a pulse away, is strong enough to explain all the complex
behaviors we want AI to perform. I believe the answer to this is yes, but
I so far see no way to prove whether that's true or not.
But the design demonstrates nicely the concept of how the high dimension
nature of the learning problem can by attacked by decomposing the large
problem, into a large number of simple problems. Each node learns a very
simple two action policy by reinforcement, while the society of nodes
working together learns a policy for the high dimension problem space.
So whether this architecture proves to be useful or not is unproven, it
shows how the high dimension nature of the AI problem can be attacked.
So, let me now turn to the concept formation question you brought up.
In such a decomposed network, each node in my pulse sorting network ends up
producing two new output signals. The network as whole, is envisioned as
being a huge network with many layers, and in it, the system is going to
form lots of new signals. Each of these signals in my view should be
looked as as "new concept" which was created by the network. In order to
solve complex behavior problems, this network has to form _useful_ middle
layer signals. It has to form useful "concepts" from the raw sensory data.
If I want to feed such a network visual data, and I want it to learn by
reinforcement to turn a light on every time it sees a dog, how could that
happen? It can only happen if the network, on it's own, without the help
of any reinforcement, naturally forms a set of internal concepts (internal
middle layer signals) that can correlate nicely with "dog". If it for
example, forms internal signals like the "dog ear", and "dog noise", and
"dog fur", because they each represent a recurring type of visual pattern,
then those internal signals, can be correlated with the reward signal, to
make th3e network reconfigure itself to use those signals to control the
output light.
But if it doesn't first, on it's own, from some signals that are unique to
dogs, it's lost. The raw pixel inputs have no correlations with "dog". The
raw input signals mean "bright light in this pixel". Bright and dark
pixels do not correlate with "dog". A bright pixels could be a dog as well
as a leaf, or cat.
So the network must sort the pulses, in order to form middle layer signals
like "dog". If they do that, then reinforcement learning actually becomes
very easy in this pulse sorting archetype (I have that working to an extent
just fine).
But what I don't have working, is the correct concept formation. I've not
yet figured out how to make that work correctly.
But a key aspect of how it should work,is exactly what you said above. It
should be using temporal correlation of pulses to define how it creates
it's categories.
In this architecture, each node receives pulse inputs, and must send the
pulse to one of two outputs. In effect, it's solving the classification
problem at the level of each pulse. Each pulse must be classified as a
pulse of type A, or type B. The node can (and must) have memory of past
events (aka the timing of past pulse inputs) to make it's sorting decision,
but what should be the foundation of the logic it uses to make that
classification? I don't know, but can write a few million words on all the
ideas I've explored here.
But there are a few things I know, that give some insight into how
reinforcement is added ON TOP OF, what ever algorithm is at work doing the
classification. To start wtih, I think the default classification
behavior, should divide all output pulses evenly across the two outputs.
So however the two output categories are defined by the working of the
node, the two categories are equally probable over the long run.
Let me give an example to make this clear. If we have a single input that
represents light level in a pulse signal, we can divide it into two
categories based on the brightness represented by each pulse. The bright
would be defined by the pulse spacing - the amount of time which has passed
since the last pulse. The node could use this to create two categories for
the pulses - the "bright" pulses and the "dim" pulses. Pulses could be
sent out one path vs the other based on the pulse spacing, and we end with
two categories. By adjusting the the spacing we use which defines the
border between bright and dim light, we can adjust the system to produce
two equally probable output signals. That means the outputs represent "dim
light" or "bright light" equally probably. We have in effect, created
categories that divide the one input "pixel" into two separate feature or
category signals such that the two outputs have equal probability of
outputting a pulse. What the node has to learn for this type of
classification, is how to set the reference pulse width that defines the
boundary between the two classifications.
This problem of classification or forming categories can also be seen as a
feature extraction problem. We have extracted the features of "bright
light" and "dim light" from the input signal which was just "light". But
we have done it in a way to make the two features equally probable.
Using something similar (but better as the classification algorithm), we
can think of a large network of nodes that divides, and transforms, the raw
input signals into a large set of micro-features of the environment. The
current state of all these "micro features" (all the internal signals in
the network) in theory should accurately represent the current state of the
environment. If they work correctly, we should expect the network to be
forming features for everything it has been exposed to. It should learn to
"see" the features it is exposed to by creating categories for them. So a
large network might have high level features like "dog", and "cat". The
idea is that such a network, when shown an image of a dog,. or a video with
a dog in it, should be routing pulses so as to active the internal "dog"
signal. And when there is no dog in the sensory data, we would expect no
pulses to be routed to that signal.
But a fixed configuration network will have a fixed number of internal
signals it can use for creating these micro features. But with the idea of
making all features equally probable, we can think of the network as
determine what micro features to extract based on what shows up in the
sensory data. If the system is shown lots of pictures of dogs, it will
allocate lots of micro features to represent dog features. If it's never
been exposed to a tree, there will be no high level tree features, but
where till be lots of micro features that trees have in common with dogs
(lines and edges, areas of similar color, motion of edges across the visual
field). It should in effect, define features in a way, that they are
divide3d evenly over the feature space of the complex environment.
I like to think of this using the analogy of how digital camera will divide
a continuous image, into a 2D array of features we call pixels. Each pixel
covers an even amount of the image, and in the network, each image
represents the same amount of information in the input stream.
All that background is to set up how reinforcement learning can be added on
top of this. I think it is added by making the classification nodes, skew
their normal 50/50 information split. Based on rewards, it should shift
it's algorithm to send more or less pulses to each classification.
However, sending less to one side, always implies we have to send more
pulses to the other. Using the pixel analogy, this is like changing the
size of each pixel in the image field to include more, or less information.
It in in effect distorting the grid that defines how many pixels end up
being assigned to each pixel, and allows, the algorithm that slices up the
data into features, to zoom in, on one high resolution element of the data
space, to pick out very small features that are important to getting
rewards, or zoom out, to create large abstractions if they as well, are
important to getting rewards.
The issue here is that in this very high dimension data space, there are an
infinite number of features we can define by how we slice up, and combine
the data. In order to solve any particular reinforcement learning problem
(to learn what features of the environment are important, and to learn how
to respond to them), we start with slizing of the information spce into a
large number of features. We in effect, are equally interested in
everything, so the network represents the state of the entire environment
with a set of micro features as best as it can.
But then we train the feature network, by reinforcement, to make it extract
out all the data that is important for getting rewards, and learn to mostly
ignore (but never filter out) all the data in the stream that has little
current value. the idea is that the feature extraction network evolves by
reinforcement, to form the categories that are important to get rewards.
Like with a digital camera, if you slice up the image into pixels, all the
information in the raw seen is in the pixels. But some of it might be
mixed in a single signal with other data you don't care about, and some of
it might be spread across many pixels. With the help of reinforcement, the
feature extraction network has to reshape itself to find the features that
are important to getting rewards.
In this type of approach, one network is doing everything. We don't have
one system for defining categories, and another for creating actions. The
forming of categories IS the action process of the network. It's the only
action process of the network.
For my pulse sorting neworks, the sorting of pulses is the behavior of the
network. All high level, long term, goal directed behaviors, must be
emergent properties of the society of simple nodes working together. The
micro-action is just the binary decision of sorting a single pulse. And
that forms both the end behavior of the whole system, as well well as
performs the categorizing problem.
If you have an output of this network that causes the right arm of a robot
to move up or down, we can think of this output as being a category the
network must learn to recognize. It's the category of all possible sensory
states, that the robot should raise its right arm for. All outputs can be
thought of as categories of the sensory inputs, which is why we can (and I
believe must) solve in a network which solves the category learning, and
the reinforcement learning, at the same time in parallel.
But as you suggested, it must have the power of concept formation that
works even in the absence of any rewards. Which is the default behavior of
the nodes to slice the formation space into equally probable features.
I've built working networks that do all the above. But what they get
wrong, is that they are not forming the right features. And if you form
the wrong features, the network can't learn anything we would consider as
interesting. If it can't "see" dogs, it can't be trained by reinforcement
how to best respond to a dog. That I believe is the real problem with the
current networks I've built - what they can "see" is of little use. What
they can't see, is what they can't learn to respond to.
Basically, there are constraints in the input data that reduces the
information present there. Those constraints are created by the effects of
the universe on the sensory systems. They create correlations in the input
data. If you turn a light on in a room, you can expect all your light
sensor inputs to suddenly jump to a higher value. That's a temporally
correlated effect that happened across all the parallel inputs. And it's
the type of effect, that the network could extract out of the data, by
creating a single internal signal that represents something like overall
light level. My current network designs don't manage to do that.
I think the default categorizing can also be thought of as a data
compression problem, or as an information maximizing problem. That is, it
should attempt to transform the data into the categories that represent as
much about state of the environment as is possible, with the new of signals
it has to work with. Though I've been at odds trying to figure out what a
form measure of "maximum information" might be defined in order to guide
the search for the right categorizing algorithm.
Another approach is to think of the problem as having raw data that is "out
of focus" with lots of redundancy spread across the signals, and that the
job of the categorizing algorithm is to bring the data "into focus" by
removing the redundancy, or by moving all the most redundant data, into the
same signal path, so it gets "out of the way" of the rest of the data.
And again, the entire idea that things happen close together in time, or
that pulse show up close together in time, I think is a foundation of how
this system should work. When pulse A tends to be be predictive of pulse B
that shows up later, then B doesn't carry much extra information than what
we already gained when pulse A showed up. In the extreme case for example,
if every time a pulse showed up on input line A, there was a always a pulse
on line B 100 ms later, then that second pulse gives us zero addition
information. Once we see pulse A, we know B will show up. And as such, we
can remove B from it's signal path, and combine it on the signal path for A
into one signal that means "AB pulse stream", leaving the other pulses in B
to be analyzed for other correlations. This conceptually is the type of
thing the transform should be doing by default. It should be looking at
the temporal predictive powers of each pulse in each signal and using that
to define categories, and to produce signals based on those temporal
predictive categories.
I've looked at this from many direction for a few years now, and though I'm
sure the "right" algorithm for doing this type of transform exists, I've
not yet figured it out. But I believe it's the last big missing piece of
the puzzle I'm still missing. Without out, reinforcement learning has no
hope of learning anything really interesting. With the correct type of
categorizing system, I think we are going to seem some amazing behaviors
emerging from such a learning system.
There's more details here I've not touched on, but that's the overview of
how and why why we need the two types of learning (Classical and Operant
conditioning) and how they both need to be combined into one system that
deals with the high dimensionality space by a divide and conquer approach
of using micro-learning machines working together to create the macro
behaviors as emergent properties of the society of learning machines
working together.
Finding a workable architecture for implementing this type of learning
system is the challenge. The devil is in the details. Get the details
just a little wrong with this type of system, and it does _nothing_
interesting. I think my pulse sorting approach has some possibilities of
being a workable implementation, but again, the details of my designs are
not yet right, and so far, what it can do is not all that interesting. But
I feel this approach is workable, and close to producing some very
interesting results.
Daryl McCullough
>
>stevend...@yahoo.com (Daryl McCullough) wrote:
>> So before the learner can even start doing reinforcement learning,
>> it has to be able to categorize its environment and its own behavior.
>> But how does it learn to categorize correctly? You could say that it
>> learns categorization through reinforcement learning, as well, but
>> that just pushes the problem back further: how does it learn what
>> the various categorization strategies are?
>
>Yes, I agree completely with all the above.
>
>Solving those problems I see as implementation details.
Maybe, but I think that those details are exactly what's
so hard about AI. Human beings have an initial learning
strategy at birth that is sufficient to uncover all the
secrets of language, science, mathematics, and social
interactions. You say that this learning works by
reinforcement learning, and that might be true, but
if the *starting* point is not sufficient, then reinforcement
learning isn't going to get you very far. You can't teach
a flatworm to play the piano, no more how cleverly you
design its reinforcements.
>The trick to solving this problem, is figuring out what sort of system to
>start with.
Exactly. This is where we fail, in my opinion.
>This idea is basically the same idea we see emerge externally as classical
>conditioning in animals and humans. And I argue that classical condition
>is in fact the solution to the categorizing problem.
Well, as I said, conditioning only works on certain starting points.
With some starting points (a rock), it doesn't do anything. With other
starting points, a flatworm, you can only go so far. So conditioning
requires a learner that is capable of being conditioned, and *that's*
what we don't know how to create.
>It's also what I call the perception problem. We can't learn to love money
>by reinforcement, if we don't first have hardware that can correctly
>categorize raw sensory data as "money". Reinforcement learning offers no
>answer to how we solve the primary categorizing problem - how the hardware
>solves the perception problem. However, I strongly believe that once you
>create a machine to solve the categorizing problem, you must ALSO make it
>adjust it's operation, by reinforcement. YOu can't do it as two
>independent modules.
That sounds plausible.
I can agree with that, except that we may differ about which of the two
is the hardest and most important. I think perception and categorization
is the hard part.
>I like to say that the category learning problem is what allows the machine
>to "understand" the world, and that the reinforcement learning is what
>allows the machine to know what to do with that understanding. If you are
>familiar with the Jeff Hawkins book "On Intelligence"
Yes, a very thought-provoking book. I'm not 100% convinced that he's
got the whole story, but his presentation was very persuasive.
>I claim that, (at least in everything talke about in that book)
>he has failed to see these
>two problems. All is effort is based on the first problem - how to
>understand the world - or how to solve the category problem you talk about.
>Without adding reinforcement learning at the same time, the sort of system
>he is developing has no purpose - no goal in life - no motivation to _do_
>anything with it's understanding.
I think you are right, that motivation is needed to complete the picture.
Hawkins had the insight (I'm not sure if it is original with him) that
perception and prediction are closely linked, as are prediction and
action: we recognize that something is a horse or a ball because we can
predict how the horse will behave. We are able to swing a bat to hit
a baseball because we can predict the pattern of tensions in our muscles
and visual patterns that occur when we do so.
But what, in these terms, is motivation? Presumably, we can predict
the possible near futures of our own situation, and evaluate some as
producing a higher reward than others. Do you think that's what really
happens?
>So let me explain also explain how I think we can solve the problems you
>bring up.
>
>First, is the issue of the domain being high dimension. This means that
>there are so many variables in the sensory data that we will never likely
>see the same combination twice. And there are likewise so many dimensions
>on the output side, we will are likely to never perform the exact same
>behavior twice. So we can't use reinforcement with the idea we will see
>the exact same sensory data twice, or the need to perform the exact same
>behavior twice. So we can't assume we will learn by trying something lots
>of times. We won't, in our entire life, try anything exactly the same
>twice. So how do we do reinforcement learning (which is a trial and error
>type of learning) in such an environment?
>
>The answer that sounds right to me, is that we must use an architecture
>that decomposes the problem down into many separate modules which work
>together to produce behavior, and then apply the training to all the
>modules in parallel.
I think that's absolutely right. But that design is (in my opinion) the
hard part.
[Rest deleted---I don't have time to read it all, right now]
Thanks for the discussion.
Curt Welch
> Curt Welch says...
> >
> >stevend...@yahoo.com (Daryl McCullough) wrote:
>
> >> So before the learner can even start doing reinforcement learning,
> >> it has to be able to categorize its environment and its own behavior.
> >> But how does it learn to categorize correctly? You could say that it
> >> learns categorization through reinforcement learning, as well, but
> >> that just pushes the problem back further: how does it learn what
> >> the various categorization strategies are?
> >
> >Yes, I agree completely with all the above.
> >
> >Solving those problems I see as implementation details.
>
> Maybe, but I think that those details are exactly what's
> so hard about AI.
Oh, I agree completely. Finding a working architecture and implementation
for these high level concepts is what's so hard.
> Human beings have an initial learning
> strategy at birth that is sufficient to uncover all the
> secrets of language, science, mathematics, and social
> interactions. You say that this learning works by
> reinforcement learning, and that might be true, but
> if the *starting* point is not sufficient, then reinforcement
> learning isn't going to get you very far. You can't teach
> a flatworm to play the piano, no more how cleverly you
> design its reinforcements.
>
> >The trick to solving this problem, is figuring out what sort of system
> >to start with.
>
> Exactly. This is where we fail, in my opinion.
>
> >This idea is basically the same idea we see emerge externally as
> >classical conditioning in animals and humans. And I argue that
> >classical condition is in fact the solution to the categorizing problem.
>
> Well, as I said, conditioning only works on certain starting points.
> With some starting points (a rock), it doesn't do anything. With other
> starting points, a flatworm, you can only go so far. So conditioning
> requires a learner that is capable of being conditioned, and *that's*
> what we don't know how to create.
Right, that's obvious. In psychology "conditioning" is seen as something
we do _to_ the animal, but in AI, as an engineering problem, it's something
we have to build into our design. We have to build a machine that can be
conditioned.
Building machines with simple conditioning is trivial. But building
machines that can be conditioned as well as a human, is very hard. And as
you say, one of the biggest missing pieces (the piece that Behaviorists
haven't given us any real help with), is the perception problem. A system
can't be conditioned to push a button when a light comes on, if it cant'
"see" the light as a unique feature of the environment, and see "button
press" as a unique feature of the environment as well.
We tend to ignore the complexity of this because it's built into us. We
don't have to do anything hard to "see" a feature of the environment like a
pen laying on a desk. We just see it.
Reinforcement learning works very well already, if you bypass the hard
problem of extracting features from complex sensory data by either using
specialized sensor systems to detect only the features that are important
for the task and if the action space is small. But when either the action
space grows too large, or the sensory feature space grows too large,
reinforcement learning alone becomes unworkable. That's when you have to
bring in the system that can learn to categorize features automatically to
solve both problems.
Yes, I agree. The perception problem is the part that no one has correctly
and fully solved yet. So it's clearly the harder part at the moment.
However, I don't think it's actually hard. That is, once we have it solved,
I believe it will no longer be seen as hard by anyone. I think the
solution will actually be rather simple. If that turns out to be true,
then the only thing hard about it, was finding the simple answer.
> >I like to say that the category learning problem is what allows the
> >machine to "understand" the world, and that the reinforcement learning
> >is what allows the machine to know what to do with that understanding.
> >If you are familiar with the Jeff Hawkins book "On Intelligence"
>
> Yes, a very thought-provoking book. I'm not 100% convinced that he's
> got the whole story, but his presentation was very persuasive.
Yes, I found myself strongly agreeing with 99% of everything he wrote. The
only thing I had issue with was that he failed to see the need to add
reinforcement learning at the same time to give the system purpose. If you
fail to understand that need, and build software (as his company is doing)
without any consideration as to how reinforcement learning is added, the
implementation you end up with will probably not be workable. That is, if
it's not an approach that can be conditioned, then it can't be the right
answer to AI. None the less, it's all important work he's doing because he
is trying to solve that hard perception problem. And at this point, any
good solution to the perception problem will be helpful.
> >I claim that, (at least in everything talke about in that book)
> >he has failed to see these
> >two problems. All is effort is based on the first problem - how to
> >understand the world - or how to solve the category problem you talk
> >about. Without adding reinforcement learning at the same time, the sort
> >of system he is developing has no purpose - no goal in life - no
> >motivation to _do_ anything with it's understanding.
>
> I think you are right, that motivation is needed to complete the picture.
> Hawkins had the insight (I'm not sure if it is original with him) that
> perception and prediction are closely linked,
I think the idea goes back to before he was born. When you read older
works you find most the same idea being hashed over to some extent. But
he's clearly pushing the concept forward and bring it into perspective.
> as are prediction and
> action: we recognize that something is a horse or a ball because we can
> predict how the horse will behave. We are able to swing a bat to hit
> a baseball because we can predict the pattern of tensions in our muscles
> and visual patterns that occur when we do so.
Well, yes and no. Reinforcement learning systems are already amazing
prediction machines even without solving the feature extraction problem
(which I agree is also a prediction problem).
It's because to do reinforcement learning, the system has to predict future
rewards. I use the TD-Gammon program a lot as an example of what RL can
do. It's backgammon program that learns how to play the game from
experience. It is made to play itself millions of time, and after enough
training like that, it has learned to play the game at (or at least very
near) the same skill level of the most expert human players. It uses a
standard reinforcement learning algorithm, combined with a simple neural
network to solve the categorizing problem for the game.
The only reward the learning machine gets is at the end of the game. But
yet, every move it makes, has been conditioned based on the expected
results. The first move it makes, has to be made based on a prediction of
what will happen 20 moves later.
We have reinforcement learning algorithms already that fully explains how
this prediction works, and how these machines can converge on making
perfect predictions of the future though experience. So to some extent,
that's already been solved.
But finding a good automatic categorizing algorithm to work _with_
reinforcement learning, is what hasn't been done. It's the hard part still
left to figure out.
> But what, in these terms, is motivation?
It's creating behaviors for the purpose of maximizing all _future_ rewards.
> Presumably, we can predict
> the possible near futures of our own situation, and evaluate some as
> producing a higher reward than others. Do you think that's what really
> happens?
Yes, without a doubt. I know for sure that's what happens.
But it's not really a prediction of _what_ will happen. That is, it
shouldn't be confused with how we use our thought process to predict that
when we take the lid off the cookie jar, that we will see cookies inside.
That's a much higher level process that falls out of this lower level stuff
I'm talking about.
For example, in TD-Gammon, when it picks what move to make, there is
nothing in the system that is predicting what the board will look like in
10 moves (as happens if we used a classic game tree search algorithm to try
and figure out what move to make). All it's predicting, is an estimated
expected average future reward value. All it's prediction, is how _good_ a
given board position is. The system has no concept of _why_ the move is
good at this low level. All it's doing is predicting by the use of
accumulated statistics, that a given board position is more or less likely
to result in a win. This requires a statistical system that applies the
reward that showed up at the end of the game, back to all the board
positions that led to that reward.
So though it's prediction of the future, the only thing the reinforcement
learning side of the equation is predicting about the future, is how many
rewards are to be expected.
No matter how it's implemented, the reinforcement learning side works by
back-propagating reward predictions though environmental states and
actions, that led up to the reward (or lack of reward).
So when the system picks an action, it's based on this accumulated
knowledge of how likely that action, vs other actions, are to produce
higher rewards.
Much of what Hawkins talks about in his book when he talks about predicting
the future, is actually a job for reinforcement learning, but he doesn't
seem to grasp that. A categorizing system doesn't define what's good or
bad. It can make statical predictions about what will happen next, but
can't make any sort of judgment about whether that's a good result or a bad
result. A very high power categorizing system for example can predict that
when you drop an egg, it will shatter, and spread this yellow and white
stuff all over the floor. That type of prediction is made simply from the
past experience of seeing eggs hit the floor and seeing how the sensory
data changes in response to that. The sensory state of "egg falling" is a
statical predictor of what is expected to follow. It can also predict that
if the agent moves it's arm and hand correctly, the egg won't turn into
yellow liquid. But it makes ZERO prediction about whether that outcome is
better or worse, than letting it hit the floor. Reinforcement learning is
what adds purpose to the system - it's what allows it to predict that the
yellow on floor state is "bad" and the egg in hand state is "good".
But again, at the low level, it doesn't actually work by searching forward
though various predictions of the future to see which one leads to the best
result. It works backwards. It figures out form past experience, which
action is more likely to lead to higher future rewards.
You see the whole problem boils down to the question of what must the agent
do _NOW_. And at the low level, these real time systems don't have the
luxury of running some long tree search process to check the value of
various action choices with some prediction system. It must know what to do
now, without having to "think about it" (so to say). So the processing is
done instead, ahead of time. When the system gets a reward, it assigns
some amount of the blame/credit to the actions that led up to the reward.
This means the system must have some sort of persistent memory of what has
happened in the past, and use that persistent memory to assign credit for
the reward.
Then, in the future, when the next time it needs to make an action
decision, it already knows how "Good" or "bad" each action choice is -
without having any sort of memory of why a given action is good. It doesn't
know why turning left in the maze at this point is good, it only knows, by
looking at the accumulated statistics, that turning left at this point is
"better" than turning right.
When writing programs to play games, there is plenty of freedom of the
program to sit there and do some number crunching for 30 seconds before it
decides what move to make. But for real time interaction with a physical
world, there's not such option available for the low level hardware. It
must know what is best before it has to do it. So when it's time to make
the next decision, the system just looks at the action available, and their
associated "worth", and sees that turning right in the current context has
a value of .645 and turning left has a value of .435 so it picks turning
left. Those.654 and .435 "values are "predictions" of future reward, but
not predictions of what will actually happen other than rewards.
However, the categorizing system is in effect making predictions about what
will happen next. And it needs to define categories, based on what
category definition is the best predictor, of what will happen next in the
sensory data.
This sort of prediction is _not_ used (at the low level) for the purpose of
exploring alternative futures. It's instead, only used for creating higher
quality categories - that is, categories that are themselves, better
stimulus signals for controlling behavior. Or better stimulus signals for
assigning estimated values to - they are better categories for predicting
future rewards, and better categories for regulating behavior.
For example, without the help of a system for creating better categories,
we could have a raw stimulus signal which represents the light level of the
pixel at the top right corner of our vision sensor. How good is that small
feature of the environment for predicting future rewards? Not very good at
all is the answer. But if our agent is stuck in a cage where it can get
rewards when a light comes on, a sensory system that indicates when the
light is on, is very good predictor of future rewards. Every reward the
agent every got, came after the light was on.
But the light category had to be created (at least partially) before the
system could learn its value. But it can be created, because it's an
effect in the environment that is persistent over time. Once the light
comes up, it's expected to stay on for an extended period. The "light on"
signal at point T in time, is a very good predictor of "light on" at point
T+1.
When the agent moves it's eye to scan the light, the light pattern moves
across the visual field. But the persistent effect of the light, creates a
persistent set of changing patterns across the pixels. A "light on"
pattern at one pixel, and a "light on" pattern at the pixel next to it,
creates a spatial perception of "light on" in the third pixel next to the
second. When two pixels activate, it's more likely for the third one to
activate as well. This is a persistent statistical effect that can be used
to justify the creation of a category of "three pixels on on in row". And
when these patterns move across the visual field over time, it creates a
temporal pattern that gain, can be used to justify the creation of a
category of "three pixel pattern moving across the visual field to the
right".
The categories get created because they identify statistically significant
correlations of patterns in the sensory data. Though there is an infinite
number of possible patterns to check for, the system needs to learn on it's
own, and converge on the patterns (categories) which act as the strongest
predictors.
Categories that are the strongest predictors of future sensory data
changes, also I suspect, are the best categories for acting as state
signals for predicting future rewards, and acting as stimulus signals for
triggering actions - for defining the context in which action decisions are
made.
Many people over the years in AI research have declared that trial and
error reinforcement learning is "too simplistic" to explain complex human
behavior But it's not. The problem is that it only works, if you also
solve the perception problem at the same time. Without solving the
perception problem, the reinforcement learning system is effectively
"blind" to the true state of the environment. And "blind" systems are, for
sure, too simplistic to learn anything interesting. How much a system can
learn by reinforcement, is all a function of how well it can "see" what is
happening in the environment. And so far, our AI projects that try to use
reinforcement learning, are still to "blind" to learn anything interesting.
If we solve the perception problem, and solve it in a way that we can marry
it with reinforcement learning, then we will have solved the hard problems
of AI in my view. It will be like the point where the Wright brothers put
together the important pieces to create controlled powered flight, that
turned human flight from a dream, into a reality. Everything after that
was just straight forward R&D to advance the technology. I think the same
thing is going to happen in AI once this last piece of the puzzle is put in
place. It will instantly show just how highly complex intelligent
behaviors can emerge from this type of learning system. I think from the
time the first small demonstration program that solves these problems for a
small (but yet high dimension) environment is running, to full human AI,
will be less than a decade. And I think the solution to the problems I'm
talking about here could show up any day now. Lot of people are actively
looking at them, with Hawkins being one of them.
I made a second bet with a friend of mine about 5 years ago that it would
be solved within 10 years - by 2015. I've only got 5 more years to solve
this - or for someone else to solve it, or else I lose the second bet with
my friend. My first best with him was in 1974. I lost that bet, but I
don't intend to lose this one.
> >The answer that sounds right to me, is that we must use an architecture
> >that decomposes the problem down into many separate modules which work
> >together to produce behavior, and then apply the training to all the
> >modules in parallel.
>
> I think that's absolutely right. But that design is (in my opinion) the
> hard part.
Yes, for sure. I've been trying to solve it (and understand it) for about
35 years now. I'm normally able to solve complex problems like this far
faster than that. :)
> [Rest deleted---I don't have time to read it all, right now]
>
> Thanks for the discussion.
--
casey
> The answer that sounds right to me, is that we must use
> an architecture that decomposes the problem down into many
> separate modules which work together to produce behavior,
> and then apply the training to all the modules in parallel.
> The idea is that when working together, the modules never
> end up doing the same thing as a whole, but individual,
> they perform very simple behaviors, which repeat constantly.
> And as such, trial and error learning can be applied to
> each module individual quite well.
>
>
> Abstractly, this is no different than what we do when
> we form a company and get lots of people working together
> to solve a larger problem. Each person perform only one
> simple task, and becomes an expert at that simple task
> though learning and trial and error, without for the most
> part, every having to understand what the rest of the
> organization is doing. He must however be given reward
> feedback as to whether his efforts are helpful.
However a company is not a rabble of individuals they
exist in departments.
Another example may be an army. A monolithic army, where
each soldier acts alone as part of two mobs bashing away
at each other, is no match to a hierarchical of units in
a modern Army.
Rather than talk in terms of an error signal or a reward
system I would talk in terms of what they do, they select.
The ultimate selection criteria is survival. A company,
an army and an individual either survives or they don't.
All have internal selection mechanisms which may be
called a reward or punishment.
These mechanisms are themselves selected by the ultimate
selection mechanism called natural selection.
> And there must be some overall structure that decomposes
> the larger problem, down to smaller tasks.
Again an act of selection. A many to one process. An act
that throws away some things and keeps others according
to some selection criteria.
> This approach abstractly I believe solves the high
> dimension problem, but leaves the implementation details
> of how to do it totally undefined.
But we do have examples of how this is implemented
problems. You make reference to how can a system learn
these implementations without having them built in.
> Generally speaking, all ANNs (artificial neural networks)
> take this approach.
>
>
> Behavior is produced [in an ANN] by data flowing forward
> though the network, and training works by sending error
> data backwards though the network.
Aplysis starts with default connections that will produce
useful but modifiable behaviors. For example a light touch
to the siphon will cause a gill withdrawal reflex. Continual
light touches will teach Aplysia the stimulus is harmless
and it will cease to withdraw it gill. Continual stimulus
results in a weakening of the connection between the touch
sensitive sensory neuron and the motor neuron.
If however a mild shock is given to Aplysia's tail after
the siphon has been touched it will stimulate a pain
receptor into action which is connected to the gill
withdrawal circuit. It acts on this circuit to make the
connection stronger. Aplysia is said to have learned
that a touch on the siphon will be followed by a shock
to its tail.
touch neurons ------>[ circuit ]-------> motor neurons
^
|
pain neurons -------------+
In this simple case the analysis is done by two specialist
sensory neurons. There are two different ways of changing
the connections.
> But that's not reinforcement learning - it's learning
> abstraction though mimicking.
Any kind of learning involves changing the connections
regardless of the terminology used to describe the kind
of learning taking place. I would concentrate on what
it does and how it does it (see Aplysia above) rather
than what it is called.
JC
casey
> casey schrieb:
>
> > On Jan 11, 3:44 am, Burkart Venzke <b...@gmx.de> wrote:
>
> > Hi Curt, looks like you have won a convert ;)
>
>
We have different ways of approaching the problem so I assumed you
were a convert to the approach used by Curt.
JC
casey
> We tend to ignore the complexity of this because it's
> built into us.
Exactly. The behaviorist assumes what is to be explained.
> A categorizing system doesn't define what's good or bad.
> It can make statistical predictions about what will happen
> next, but can't make any sort of judgment about whether
> that's a good result or a bad result.
>
> ...
> at the low level, it doesn't actually work by searching
> forward though various predictions of the future to see
> which one leads to the best result.
>
> It works backwards. It figures out from past experience,
> which action is more likely to lead to higher future
> rewards.
>
> You see the whole problem boils down to the question of
> what must the agent do _NOW_.
To say a system cannot search forward isn't true if a system
has a internal model it can run forward to see if an outcome
would be rewarding. In a chess game it is called depth search.
The choice is not made on the NOW state but instead on the
reward value of a computed FUTURE state. TD-Gammon I think
does a depth search as well. The only difference is the value
is determined by an ANN which learns rather than a set of
inbuilt heuristics. But the end result is it takes time to
make a decision.
> And at the low level, these real time systems don't have
> the luxury of running some long tree search process to
> check the value of various action choices with some
> prediction system.
But the search doesn't take place at the low level. All
systems are real time at that level.
> When writing programs to play games, there is plenty of
> freedom of the program to sit there and do some number
> crunching for 30 seconds before it decides what move to
> make. But for real time interaction with a physical
> world, there's not such option available for the low
> level hardware.
At the low level everything is a reflex action taking
place in "real time". But processes can be made up of
a series of reflexes and that takes different lengths
of time.
If you are asked to press a button whenever a click sound
is heard there will be a cascade of stimulus/responses
through your brain which will have a temporal limit of
about 0.1 seconds to complete. No amount of practice will
change it. All physical systems have temporal limits. It
takes even longer to react to a flash of light.
----------> frequency of response
*
*
**
******************
**************************
*****************************************
******************************
**********************
************
******
***
***
**
**
*
*
*
If we introduce the need to make a decision it takes even
longer to respond to a stimulus. Let us say you have to
press one button in response to a red light and another
button in response to a green light. When a light comes
on you have decide which button to press. This will take
you 1/10 of a second extra to react, regardless of the
stimulus being a sound event or a light event.
Unlike the response curve for a simple reaction the curves
for a decision reaction produce regular peaks and valleys
suggesting a periodic process of 0.03 seconds.
JC
Burkart Venzke
> On Jan 11, 9:53 am, Burkart Venzke <b...@gmx.de> wrote:
>> casey schrieb:
>>
>>> On Jan 11, 3:44 am, Burkart Venzke <b...@gmx.de> wrote:
>>> Hi Curt, looks like you have won a convert ;)
>>
>> Why "convert"? ;)
>
> We have different ways of approaching the problem so I assumed you
> were a convert to the approach used by Curt.
I see.
I don't know which point of view such a convert normally has before or
which you have perhaps. (Sooo much has been written... Your contribution
here is nice because it is short.)
So, you think that learning is not so important? Or what is your approach?
Burkart
casey
On Jan 13, 7:14 am, Burkart Venzke <b...@gmx.de> wrote:
> casey schrieb:
>
> > On Jan 11, 9:53 am, Burkart Venzke <b...@gmx.de> wrote:
> >> casey schrieb:
>
> >>> On Jan 11, 3:44 am, Burkart Venzke <b...@gmx.de> wrote:
> >>> Hi Curt, looks like you have won a convert ;)
>
> >> Why "convert"? ;)
>
> > We have different ways of approaching the problem so I
> > assumed you were a convert to the approach used by Curt.
>
>
> I see.
>
>
> I don't know which point of view such a convert normally
> has before or which you have perhaps.
The statement was mostly a joke which is the reason it ended
with a winking icon.
> So, you think that learning is not so important?
Learning is everything for humans for without it we would
only have a few simple reflexes.
Where Curt I differ is he believes it can and should be
done as a single monolithic learning network without any
innate support and that the "generic" learning module of
the brain takes such a simple form.
> Or what is your approach?
That AI will evolve as all our other technology has evolved.
I don't know what its limits are but I see AI as already
existing in that there are machines that can do things
that if done by a human would be considered intelligent.
I don't see intelligence as a singular thing to be invented
rather I see it as a category word for things we do that
are different to the things we might do if we were showing
unintelligent behaviors.
JC
Burkart Venzke
> Burkart Venzke <b...@gmx.de> wrote:
>> Hi Curt Welch, hi all,
>>
>> You have an interesting but very long discussion in "No easy road to
>> AI".
>
> Yes, I tend to ramble on endlessly.
Too long for me. This thread is long enough for me ;)
>> As we want to solve the problem of AI I also think that learning is
>> the central problem of it.
>>
>> What do you think are the main problems about learning?
>
> The implementation details. :)
Not a detailed theory?
>> One interesting point I have read in the beginning of your discussion is
>> the connection of learning and perception and that an AI also has to
>> learn perception. Yes, it is important.
>>
>> Another point is how far the internal representation of the AI system is
>> important.
>
> It's highly important. You could say it's everything.
It is when we want to implement it.
>> I think that a least we want to really write software we have
>> to think about the internal representation. But ok, we may have more
>> important problems before.
>
> Well, I guess you need to be clear about what you mean by the internal
> representation.
That's true.
> Representation of what? Of the environment? Of the
> learning system? There are lots of internal represdnations you have to
> get "right" to solve AI.
It thought of the representation of the learning system.
Do you mean the real world by "environment"?
>> For me, (other) central points are
>> - how to motivate an AI to learn without programming it precisely
>> - what it really have to learn
>> - the basics of learning, philosophically seen. "What is learning?" etc.
>>
>> Burkart
>
> Yeah, those are all basic questions that been been debated since the very
> begin of AI and many also for far longer in areas like psychology.
>
> If you look up learning theories in psychology, the lists you find will
> seem endless. They are all attempts to answer the question of "what is
> learning?"
>
> To solve AI (make a computer act like a human), we have to figure out what
> type of learning to build into the machine. Do we need 20 types of
> learning modules working together to create all the types of learning
> humans have been documented to contain? Or is there a smaller list of
> learning abstractions that can cover all learning?
Yes :)
I think that there is only a small important list whereas the rest is
variation (new not too important aspects of item of the small list).
> I believe reinforcement learning is the only important type of learning we
> have to correctly implement to solve AI, and that all other forms of
> learning fall out as learned behaviors of a strong reinforcement learning
> machine. Reinforcement learning (a field of machine learning which is a
> field of AI and Computer Science) is basically the same thing as operant
> conditioning in psychology (The work of Skinner and other Behaviorists).
> It's also closely related to genetic algorithms in AI - which is basically
> a specialized sub-class of reinforcement learning in my view - though I
> don't see others making that connection.
>
> Before these terms became better formalized, reinforcement learning was
> also generically known as trial and error learning since it's a method of
> learning by experience.
>
> If you are not familiar with reinforcement learning:
>
> http://en.wikipedia.org/wiki/Reinforcement_learning
>
> Sutton's Book which is on line here:
> http://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html
So, what is the real basis of reinforcement learning? Why isn't it the
solution for AI learning?
Or do we have to look for further foundations?
For me, a possible AI is something with the goal "learn (and act) for
your teacher" with some sensor (e.g. to see) and actor peripherals in a
real or play world (or how would you say to an easier world?).
Assuming (at first) an easy world, the AI first act randomly but with a
direct connection to its teacher (evualating with good or bad).
So, it can easy learn which of its actions was good and which bad.
The teacher has to present good situations/"examples".
(Also, a channel for easy communication between AI and teacher would be
nice. "Good" and "bad" would be the first two words.)
> On you quesiton about what is learning, it's just the fact that human
> behavior is not constant. It changes over our life time. So at the most
> abstract layer, we can just say that learning is a change in behavior. But
> that gives us no answer as to the direction of change. Why might one
> change in behavior be called learning, and another be called random
> nonsense? It's because learning typically assumes a change for a purpose,
> or a change for "the better".
Purpose, "the better"...
My central point is that humans have goals which are their central
control - independently if they consious or not, after hard planning or
only emotion etc.
> But when we say these things, like "change for the better" we have just
> pushed the question of what is learning, off to a problem of defining what
> we mean by better. Reinforcement learning answers that questions.
Does a teacher decides or who (what)?
> Other answers people have used to the question is to describe learning
> behavior as goal seeking. That is, our behavior changes to better reach
> some goal. But then we are left asking the question of what a goal is, or
> what the goal should be for our AI solution. Reinforcement learning
> abstracts the definition of what the goal is to reward maximizing.
I like the idea of goals, "reward maximizing" is too abstract for me.
But perhaps we only fight for words and mean nearly the same.
Goals are something we can implement, we can imagine something - better
than an abstract function. Or is the function for reward maximizing
something else?
> Reinforcement learning frames the problem of learning as an agent, which
> interacts with an environment. Which means it has inputs from the
> environment (sensors) and outputs to the environment (effectors). But in
> addition, it also receives a single dimension reward signal from the
> environment.
So far, so well. Besides the teacher, the world also may be this
environment when there is a basis for good and bad.
A human has its emotions as a basis, but an AI?
> The goal of the agent in this abstract framework, is to maximize some long
> term measure of the reward signal. There is no explicit end-goal in
> reinforment learning. That is, learning never ends.
Right. It is also my idea of learning because no AI (or human) ever can
be perfect in a complicated (e.g. real) world.
> The assumption is the
> agent will continue to try and find ways to get more rewards - even though
> there might not be any better ways to get rewards than what it's currently
> doing.
>
> Though we say in this framework that the reward comes from the environment,
> when we build reinforcement learning into something like a robot, we also
> have to build the hardware which generates the reward signal. But that
> hardware, is conceptually outside the "reinforcement learning hardware"
> module. From the perspective of the learning module, the reward comes from
> the environment of the learning module.
>
> The problem with reinforcement learning, is that's it's easy to specify,
> but extremely hard to implement. In fact, no one has implemented a good
> generic reinforcement learning machine.
I think that the (or a) problem is that the "correct" reward signal is
not always unique. What is really "correct"? (Besides well-known
physical laws and something like that.)
> There are many reinforcement learning algorithms that have already been
> created. Most however, only work (or are only practical) for small toy
> environments - like the game of tic tac toe or something similar. This is
> because they require the algorithm to track a statistical value for every
> state the environment can be in. When there are only thousands of states,
> like the number of possible board positions in tic tac toe, and the agent
> has enough time to explore all the states many times, then the current
> algoerithms can converge on optimal (perfect) decisions fairly quickly.
>
> But as the environment becomes more complex (more states it can be) the
> simple RL algorithms fail - because it quickly becomes impossible to track
> a value for each possible state, (not enough memory in the world to track a
> value for every board game in Go). So some other approach has to be used.
> These are called high dimension problem spaces because the number of states
> of the sensory inputs have an effectively how number of dimensions - aka
> multiple sensors acting in parallel - which just means the total state
> space is huge.
>
> No one has found a good generic solution to this problem but many people
> are looking for them.
Here, for example for tic tac toe and further on for Go, symbols are my
idea like "line" or "area".
Also therefore, I want a communication between human and KI so that the
human can define a (for the AI) new word like "line".
OK more or less.
But I see one important dfference: The human has innate "goals" in form
of emotions etc. In opposite to a human we can define the goals for the
AI. For example, an AI can be easily seen as a machine or "slave"; a
human should not.
>> - what it really have to learn
>
> Trial and error learning where behavior is adjusted so as to maximize a
> reward signal while the agent is interacting with the the environment (on
> line learning).
Who defines the reward signal?
Do you have a simple toy world and AI in mind which would work with your
ideas?
Burkart
Burkart Venzke
> On Jan 13, 7:14 am, Burkart Venzke <b...@gmx.de> wrote:
>> casey schrieb:
>>
>>> On Jan 11, 9:53 am, Burkart Venzke <b...@gmx.de> wrote:
>>>> casey schrieb:
>>>>> On Jan 11, 3:44 am, Burkart Venzke <b...@gmx.de> wrote:
>>>>> Hi Curt, looks like you have won a convert ;)
>>>> Why "convert"? ;)
>>> We have different ways of approaching the problem so I
>>> assumed you were a convert to the approach used by Curt.
>>
>> I see.
>>
>>
>> I don't know which point of view such a convert normally
>> has before or which you have perhaps.
>
> The statement was mostly a joke which is the reason it ended
> with a winking icon.
OK :)
>> So, you think that learning is not so important?
>
> Learning is everything for humans for without it we would
> only have a few simple reflexes.
>
> Where Curt I differ is he believes it can and should be
> done as a single monolithic learning network without any
> innate support and that the "generic" learning module of
> the brain takes such a simple form.
What (or how much) innate support are you thinking of about?
>> Or what is your approach?
>
> That AI will evolve as all our other technology has evolved.
Has all other technology been evolved the same way?
I don't think so.
But perhaps you see a principal (special) difference to Curt's idea of
evolving it?
> I don't know what its limits are but I see AI as already
> existing in that there are machines that can do things
> that if done by a human would be considered intelligent.
I sounds like the question if the is a principal diffenrece between weak
and strong AI. There seems to be none for you.
> I don't see intelligence as a singular thing to be invented
> rather I see it as a category word for things we do that
> are different to the things we might do if we were showing
> unintelligent behaviors.
Perhaps we should speak more about "strong AI" then?
Sure, weak AI exists.
One difference seems to be (good) learning, right?
Burkart
Burkart Venzke
> Burkart Venzke <b...@gmx.de> wrote:
>> casey schrieb:
>
>>> Motivations have to be precisely programmed. In biological
>>> systems this was done by an evolutionary process. To motivate
>>> simply means to get moving. You "motivate" your automobile
>>> with a press of the accelerator pedal. In brains different
>>> kinds of inputs can "press" the brain's accelerator pedal.
>>> We train animals by providing inputs that we know will cause
>>> their accelerator pedal to be pressed.
>> Acknowledge.
>> I think that motivations are a precondition for general learning.
>> That's why I've mentioned them.
>
> Yes, if learning is change in behavior, something has to direct that
> change.
Right.
> Why would one change be better than the million other changes not
> taken?
Because it was more important relative to a or the intermal goal(s).
> What makes the learning hardware pick one change over another?
See above.
> Whatever the process the hardware uses to direct the change is what we can
> all the motivation of the learning hardware.
See above, too.
My idea:
- An AI needs few innate goals (as the twoe goals "act for a/the
human(s)" and "learn as much/good as you can").
- It interacts directly with a human (teacher, parent etc.) especially
to learn the further foundations, e.g. of the (its) world.
- Direct signals (from the human) and further on communication between
AI and human serves as reinforcement which may produce new (sub)goals.
Later (or additionally), the AI may learn more directly form its world
without human.
>>>> - what it really have to learn
>>> That depends on its goal.
>> Right.
>> For me, a very small set of goals is the base of a (real/strong) AI
>> (originally thinking of Asimov's three rules for robots).
>> My two goals/rules are:
>> 1. Do what the AI's owner(-s; manhood) wants it to do.
>> 2. Learn as much and good as it can.
>
> Yes, those are clearly the type of goals we would like to build in our our
> AI. Those ideas correctly captures the sort of ideals we want to end up
> with in our AI. But how, using transistors, can we build a detector for
> "what the owner wants"? And how do you make transistors "be good"?
"Good"/"bad" it send (at first) directly to the AI vers a special channel.
"what the owner wants": Communication channel human <-> AI.
> You have a long road to go to translate such high level abstract ideas into
> some wiring of a box of transistors. To some extent, that is exactly the
> road AI research has been exploring for 60 years now - the mechanism of
> good and bad and wanting.
Did they have the same or such a model I am describing?
>> The rules should be a start for "real" AI. Later, better (more
>> (precizely)?) rules will be necessary, especially when it acts in the
>> real world.
>
> Well man as been building smart machines for a few thousand years now.
> Building "rules" into the behavior of the machine has been the standard
> work of engineering since forever. And as our machines get more complex
> with the help of advanced electronics and computers, the number of "rules"
> we build into them grows. If you want to understand how to build rules
> into machines, you should first master engineering in the field you would
> like to build the AI from - electronics, computers, and mechanical
> engineering are typical if you want to understand if you actually want to
> tackle this problem for yourself as apposed to what we do here in this
> group - which is to just talk about it mostly.
I studied AI two decades ago e.g. with theorem proving, expert systems,
Prolog... Since then, thinking about strong AI is my hobby.
> Once you choose to add learning to your machines, then you are accepting
> the fact that you will lose some degree of control over the machine.
Ack.
> Once
> you add learning, you have given up some of your right to specify exactly
> what it does.
OK (more or less).
> Because of this, learning is seldom used in modern
> engineering. It amounts for less than 1% the total engineering done.
> Probably more like less than .001%.
I know, especially in (good) chess (programs).
> None the less, learning has it's place.
Sorry for a little hint here. My mother language is not English but
isn't "its place" instead of "it's place" correct?
Yes, artificial learning should become more and more important.
> It allows the engineer to
> abstractly step back a layer and build a goal into the machine by building
> learning rules, instead of building behavior rules. The behavior rules are
> then explored and adjusted by the learning rules we build into the machine.
> Such things are already built into our machines in some places, but they
> are normally very simple systems for very limited domain learning problems.
> In AI, games have been used since the beginning as a simple environment for
> exploring how to build goals into machines and let them learn to make their
> decisions on their own. The goal is to win the game, and the decisions the
> learning system tries to master are moves in the game. Though a game is a
> highly simplified environment, it devastates very quickly how difficult it
> can be to build good learning algorithms.
E.g. in chess or Go, true.
> Currently, most engineering doesn't use learning, because the engineer is
> still smarter than most of our learning algorithms. He can study the
> problem, and build fixed behavior systems to solve the problem faster, than
> he can build a good learning algorithm to do solve the problem for him.
Right - like search and node evaluation in chess.
> But slowly, over time, our learning systems have been getting better, and
> finding wider use - especially in domains where the problem is too complex
> for the engineer to hand-code behavior rules to solve the problems - such
> as face recognition. It's very hard to hand code a face recognizer for a
> specific person. But we have learning algorithms that can learn by
> example, that do a much better job.
>
> Leaning systems are used to solve behavior problems that are too complex to
> hand-code a solution for. The drawback however is that you have to teach
> the machine - or at least, give it time to learn.
Sure, that is the great challenge.
>>> Curt would say it has to learn to maximize a reward signal.
>> A reward signal is also important for me, even a quite direct one.
>> The maximization seems to be the best method as long as the AI has no
>> better possibilities to know what is good (better, right...).
>
> In any practical application we might find the need to hand-code some
> behaviors. Either because we don't want the learning system to do anything
> else in that condition, or because for our application, we don't want to
> wait for the system to learn it. However, hand-coding fixed behaviors is
> the opposite of intelligence in my view. You either give it the freedom to
> figer out behavior programs for itself using it's own intelligence, or you
> use the intelligence in your own brain to decide for it. When we use our
> own intelligence to solve the behavior problems (the standard practice of
> all engineering), we aren't building intelligence into the machine.
The better way would be to create an AI which learns the basics and to
copy it for further applications. But this is the second step.
I think, too, that we should (first) create a system which learns from
nearly nothing (by randomly use its harmless actors, direct learning
from human response etc.)
>>>> - the basics of learning, philosophically seen.
>>>> "What is learning?" etc.
>>> Learning is a change in the system that results in its
>>> ability to do things it couldn't do before.
>> Yes, that was nearly the definition of my prof decades ago ;)
>> (I think he said something like "to do s.th. better than before".)
>> It is nice but not enough for me.
>> Who (or what) decides which things are those?
>> Who (or what) decides if "it could now" is true?
>> OK, sometimes it is easy to answer these questions but sometimes it
>> isn't, especially when human ideas etc. are involved (moral...).
>>
>> Burkart
>
> Well, the problem here is that humans for the most part have no clue what
> they are actually talking about when we talk about the most basic and most
> abstract aspects of our own behavior. What are morals? Where to do they
> come from? What is good and bad?
"Good" for them is archieving their own goal (including their morals etc.)
> What do those ideas really mean?
Do you mean morals?
These are something like own emotions combined with knowledge about
others emotions.
> We can certainly look up these terms in a dictionary and get an answer.
> But at the same time, the definitions seldom answer anything. They just
> create circular definitions with other words and in the end, fails to
> ground any of the concepts in anything absolute - to anything physical.
> You have to be a human, to have some grounding of what good is.
Yes, we have our innate emotions.
The AI should only know "good" and "bad" (or a scale from -1..1 or
similar) as "helpful for a goal" or the opposite. The human has to teach
the rest.
> But good
> then just becomes abstractly, "the stuff I like", and bad just becomes "the
> stuff I don't like". But how does the brain make us like some things and
> not others?
In my mind, an AI has no innate (other) emotions because it should be a
helpful intelligence, not an egoistic rival of the human.
> Because no one has yet fully solved the problem of how the brain works, we
> can't yet fully answer simple questions like "what is good", or "what is
> pain", or even, "what is an idea"?
But we can define it for an AI. It is not necessary to rebuild the brain.
> But, by assuming the brain is just a reinforcement learning machine, we can
> actually answer all those questions. But until we can build a
> reinforcement learning machine that duplicates human behavior, we can't
> prove that assumption is actually correct, or the answers are correct.
>
> I'm so convinced that human intelligent behavior is just the emergent
> behavior of a generic reinforcement learning machine than I believe all
> this to be fact, not just speculation. But the data is not as convincing
> to others as it is to me, so as a society, we still don't know what "good"
> actually is, other than "the stuff we like".
Is it correct that you think of a rebuild of the brain?
My idea is a symbolic one.
Burkart
Daryl McCullough
>Curt Welch schrieb:
>> Why would one change be better than the million other changes not
>> taken?
>
>Because it was more important relative to a or the intermal goal(s).
But how would the AI *know* that, without trying all million
possibilities? To me, that's the hard problem.
Burkart Venzke
Which million possibilities do you mean?
An AI is in a special situation which correspondents with only few
similar knowledge/experience.
An AI doesn't *know* really much as a human doesn't really know really
much (besides only deductive things) - they assume normally and are only
relatively quite sure (philosophically seen).
An AI can become quite sure e.g. when the human(s) confirms something
(very often). Sometimes it can confirms something for itself when the
real world acts always the same, e.g. a natural law.
Burkart
Daryl McCullough
>
>Daryl McCullough schrieb:
>> Burkart Venzke says...
>>
>>> Curt Welch schrieb:
>>
>>>> Why would one change be better than the million other changes not
>>>> taken?
>>> Because it was more important relative to a or the intermal goal(s).
>>
>> But how would the AI *know* that, without trying all million
>> possibilities? To me, that's the hard problem.
>
>Which million possibilities do you mean?
You are trying to use a reward system to teach an AI something. That
something is of the form of a function (in the deterministic case):
a function that given the current situation, returns the most
appropriate action for that situation. Depending on the space of
situations and the space of actions, there could be millions, trillions,
or many, many more possible such functions. How does the AI pick one
possibility out of that space?
>An AI is in a special situation which correspondents with only few
>similar knowledge/experience.
Why are there only a few? As I said, typically there are huge numbers
of possible situations the AI could find itself in. Which ones are
similar to which others depends on a similarity metric (notion of
distance between situations). There is an even huger set of possible
similarity metrics. How does the AI choose a metric? How does it
know which situations are similar to which others?
In the real world, given any two situations, you can come up with
ways in which the situations are similar, and ways in which the
situations are different. For example: in these two situations,
a bell rang, so they are similar in that respect. But in the first
situation, the temperature was 10 degrees, while in the second situation,
the temperature was 20 degrees. Are they similar situations, or not?
You want the AI to do whatever worked in similar situations in the
past, but *every* situation in the past was similar in some ways, and
different in other ways.
casey
>> done as a single monolithic learning network without any
>> innate support and that the "generic" learning module of
>> the brain takes such a simple form.
>
>
> What (or how much) innate support are you thinking of about?
As much as it needs to get up and running within its lifetime
which depends in turn on its resources and time limits.
Learning is much more expensive in terms of time and resources.
There is a good balance for each species. Predators for example
find learning worthwhile and thus have a childhood during which
their survival depends on the parents while grazing animals are
up and running and chewing on grass from birth. Compare the
parentless start of the bush turkey to the start of an Eagle.
The fact that grazing animals can get up and walk so quickly
shows that the machinery to do so doesn't have to be learned.
Where people get confused is that this machinery is inhibited
during the early stages of a predator and what appears to
be learning is in fact the resumption of an interrupted
maturation process.
>>> Or what is your approach?
>>
>> That AI will evolve as all our other technology has evolved.
>
>
> Has all other technology been evolved the same way?
>
> I don't think so.
I do think so. It all starts simple and gets complicated over
time as new ideas are added and new materials become available.
Will some new idea appear (like Curt's imaginary generic
reinforcement learning pulse sorting network) which will
catapult AI into the realms of human intelligence? My hunch
is there will be a gradual improvement although as ideas and
new materials come together you may see it as sudden jumps
in improvement.
> But perhaps you see a principal (special) difference to
> Curt's idea of evolving it?
It is easy to have abstract day dreams and they are a good
start but in practice when you try and implement your ideas
you will find there are things you didn't think of or that
you imagined things in a way that was not really relevant.
I see Curt imagining things that aren't practical. The brain
is not a generic learning machine. It is a special purpose
learning machine designed to learn quickly about THIS world
not some abstract ANY kind of world. Our academic capabilities
are exaptations of the machinery used by our ancestors to
forage for food.
http://en.wikipedia.org/wiki/Exaptation
Our preprocessors are designed to decode THIS world (not a 4D
world for example). Due to extra resources and aided by body
changes we can out perform a chimp but we are not without our
own limits.
Our innate machinery is a parallel process but riding on top
of that is a serial process that makes up conscious thought.
We do not have access to the workings of the parallel processes
even though our choices such as learning to ride a bike, tie a
shoelace or even do arithmetic will change these systems to make
our physical actions and thinking more automatic.
I would suggest one reason we have had difficulty programming
a machine to "see" is because we haven't had access to how
we actually do it ourselves. We do however have access to how
we do arithmetic and thus can program a machine to do it for
us with the practical advantage of speed of execution.
Our serial thinking process is rather slow and our short term
memory limited requiring pencil and paper to execute complex
arithmetical tasks.
>> I don't know what its limits are but I see AI as already
>> existing in that there are machines that can do things
>> that if done by a human would be considered intelligent.
>
>
> I sounds like the question if the is a principal difference
> between weak and strong AI. There seems to be none for you.
And what would that principle difference be? I see biological
intelligence as a continuum of simple acts to more complex
acts which reflect the evolving capabilities of the brain.
There is research into the physical and mental differences
between us and the other Apes. A small difference can cause
a cascade of new abilities but the old machinery is still
there doing what it has always done.
Apart from a few house keeping tasks the old brain in the
human has become completely dependent on a functioning
cerebral cortex. But that is not the same as saying that
the cerebral cortex has replaced the old machinery -
The cerebral cortex still depends on the old machinery
to carry out its tasks and importantly to automate them.
>> I don't see intelligence as a singular thing to be invented
>> rather I see it as a category word for things we do that
>> are different to the things we might do if we were showing
>> unintelligent behaviors.
>
>
> Perhaps we should speak more about "strong AI" then?
> Sure, weak AI exists.
> One difference seems to be (good) learning, right?
Sure learning is essential and becomes dominate as the main
activity beyond processing a model of the special purpose
(not general purpose) world. At the bottom special purpose
innate circuits are faster and simpler than learning circuits
although they can also adapt and change with experience
within certain limits. All our high level serial processes
are learned but I believe they are dependent on the older
systems to provide processed data and to execute the actions.
If the cerebral cortex is deactivated (or missing) all we
see is a human vegetable with a few basic reflexes like
the eye following movements, reaction to pain, eating
reflexes, facial expressions to changing emotional states
and so on.
This doesn't mean the older parts have been replaced by
the cerebral cortex as suggested by Curt rather it is that
they can no longer make their own decisions.
JC
Burkart Venzke
> Burkart Venzke says...
>> Daryl McCullough schrieb:
>>> Burkart Venzke says...
>>>
>>>> Curt Welch schrieb:
>>>>> Why would one change be better than the million other changes not
>>>>> taken?
>>>> Because it was more important relative to a or the intermal goal(s).
>>> But how would the AI *know* that, without trying all million
>>> possibilities? To me, that's the hard problem.
>> Which million possibilities do you mean?
>
> You are trying to use a reward system to teach an AI something.
Yes, but my reward system is the human teacher, at least at first.
> That
> something is of the form of a function (in the deterministic case):
> a function that given the current situation, returns the most
> appropriate action for that situation. Depending on the space of
> situations and the space of actions, there could be millions, trillions,
> or many, many more possible such functions. How does the AI pick one
> possibility out of that space?
Now I understand your doubts. I also don't know where such a should come
from especially because there is not always *the* correct answer.
>> An AI is in a special situation which correspondents with only few
>> similar knowledge/experience.
>
> Why are there only a few? As I said, typically there are huge numbers
> of possible situations the AI could find itself in.
The room of all situations is very large but not necessarily the room of
simular situations.
> Which ones are
> similar to which others depends on a similarity metric (notion of
> distance between situations). There is an even huger set of possible
> similarity metrics. How does the AI choose a metric? How does it
> know which situations are similar to which others?
AIs shall have goals a little bit similar to ours. We have a lot of them
e.g. it form of needs as in Maslow's hierarchy of needs
(http://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs).
My two central goals for the AI are:
- Do what the human (teacher) wants you to do
- Learn as much and good as you can
Acting at first randomly through the world with the teacher's
evaluations "good (done)" and "bad (done)" lets the AI learn if its
actions was good or bad. If the AI lets fell down a glass with "bad" and
afterwards the same "bad" for a broken cup, the common of the situations
is something broken, so "broken" is (or at least "seems to be") bad.
> In the real world, given any two situations, you can come up with
> ways in which the situations are similar, and ways in which the
> situations are different. For example: in these two situations,
> a bell rang, so they are similar in that respect. But in the first
> situation, the temperature was 10 degrees, while in the second situation,
> the temperature was 20 degrees. Are they similar situations, or not?
There is no direct unique solution.
Perhaps, the AI might have learned that a bell rang means "something
important". Then, the two situations have something similar. Otherwise,
the AI may ask its teacher what the bell rang mean or at least if it is
important.
> You want the AI to do whatever worked in similar situations in the
> past, but *every* situation in the past was similar in some ways, and
> different in other ways.
Let us assume, the bell rang means "something important". Then, other
bell rang could make the AI to do something, e.g. inform its teacher
about it.
When I was a child a bell rang stands for "someone is a the door" or
"the telephone is ringing" which could be differed by the sound.
In such a situation an AI may open the door (door bell) or it may
establish the telephone connection - assuming that both situations have
been learned before.
Burkart
Burkart Venzke
> On Jan 13, 11:21 am, Burkart Venzke <b...@gmx.de> wrote:
>> casey schrieb:
>> ...>> Where Curt I differ is he believes it can and should be
>>> done as a single monolithic learning network without any
>>> innate support and that the "generic" learning module of
>>> the brain takes such a simple form.
>>
>> What (or how much) innate support are you thinking of about?
>
> As much as it needs to get up and running within its lifetime
> which depends in turn on its resources and time limits.
>
> Learning is much more expensive in terms of time and resources.
>
> There is a good balance for each species. Predators for example
> find learning worthwhile and thus have a childhood during which
> their survival depends on the parents while grazing animals are
> up and running and chewing on grass from birth. Compare the
> parentless start of the bush turkey to the start of an Eagle.
>
> The fact that grazing animals can get up and walk so quickly
> shows that the machinery to do so doesn't have to be learned.
> Where people get confused is that this machinery is inhibited
> during the early stages of a predator and what appears to
> be learning is in fact the resumption of an interrupted
> maturation process.
I see, you think of natural systems. They have to fight to survive; an
AI does not because it is as a robot a machine with software.
>>>> Or what is your approach?
>>> That AI will evolve as all our other technology has evolved.
>>
>> Has all other technology been evolved the same way?
>>
>> I don't think so.
>
> I do think so. It all starts simple and gets complicated over
> time as new ideas are added and new materials become available.
OK, it is e.g. a view of precision.
> Will some new idea appear (like Curt's imaginary generic
> reinforcement learning pulse sorting network) which will
> catapult AI into the realms of human intelligence? My hunch
> is there will be a gradual improvement although as ideas and
> new materials come together you may see it as sudden jumps
> in improvement.
Ack.
>> But perhaps you see a principal (special) difference to
>> Curt's idea of evolving it?
>
> It is easy to have abstract day dreams and they are a good
> start but in practice when you try and implement your ideas
> you will find there are things you didn't think of or that
> you imagined things in a way that was not really relevant.
Nobody thinks that AI is easy. We only look for a way to it.
> I see Curt imagining things that aren't practical. The brain
> is not a generic learning machine. It is a special purpose
> learning machine designed to learn quickly about THIS world
> not some abstract ANY kind of world. Our academic capabilities
> are exaptations of the machinery used by our ancestors to
> forage for food.
>
> http://en.wikipedia.org/wiki/Exaptation
>
> Our preprocessors are designed to decode THIS world (not a 4D
> world for example). Due to extra resources and aided by body
> changes we can out perform a chimp but we are not without our
> own limits.
Yes, we live in this world, the human race has been developing in it.
> Our innate machinery is a parallel process but riding on top
> of that is a serial process that makes up conscious thought.
> We do not have access to the workings of the parallel processes
> even though our choices such as learning to ride a bike, tie a
> shoelace or even do arithmetic will change these systems to make
> our physical actions and thinking more automatic.
>
> I would suggest one reason we have had difficulty programming
> a machine to "see" is because we haven't had access to how
> we actually do it ourselves. We do however have access to how
> we do arithmetic and thus can program a machine to do it for
> us with the practical advantage of speed of execution.
>
> Our serial thinking process is rather slow and our short term
> memory limited requiring pencil and paper to execute complex
> arithmetical tasks.
If we can develop AI in a similar (parallel) way, fine.
If we find another way to go a step forward to develop an AI which can
helps us it is also fine for me.
Burkart
casey
> ...
> I see, you think of natural systems. They have to fight to
> survive; an AI does not because it is as a robot a machine
> with software.
It is not about fighting to survive, it is about being selected.
When you design a machine it is an act of selection. The machine
will survive if it has whatever it needs for you to select it.
> Nobody thinks that AI is easy. We only look for a way to it.
AI is not a destination. You may choose a point along some
measure of intelligent behavior such as "human level" but
it is still a continuum.
---------------------------------------------------------->
Aplysia Rat chimp human ?
JC
Burkart Venzke
> On Jan 14, 8:58 am, Burkart Venzke <b...@gmx.de> wrote:
>> ...
>> I see, you think of natural systems. They have to fight to
>> survive; an AI does not because it is as a robot a machine
>> with software.
>
> It is not about fighting to survive, it is about being selected.
> When you design a machine it is an act of selection. The machine
> will survive if it has whatever it needs for you to select it.
The difference between nature and AI is that the natural beings want to
live (survive), the AI as a machine has no such will.
The latter selection is made by human (and not really enforced), the
first by nature.
>> Nobody thinks that AI is easy. We only look for a way to it.
>
> AI is not a destination.
For some persons it may be one e.g. in that sense to build helpful (and
intelligent) robots.
> You may choose a point along some
> measure of intelligent behavior such as "human level" but
> it is still a continuum.
>
> ---------------------------------------------------------->
> Aplysia Rat chimp human ?
That is also right, just another point of view.
Burkart
casey
> ...
>>> ...
>>> I see, you think of natural systems. They have to fight to
>>> survive; an AI does not because it is as a robot a machine
>>> with software.
>>
>>
>> It is not about fighting to survive, it is about being selected.
>> When you design a machine it is an act of selection. The machine
>> will survive if it has whatever it needs for you to select it.
>
>
> The difference between nature and AI is that the natural beings
> want to live (survive), the AI as a machine has no such will.
>
> The latter selection is made by human (and not really enforced),
> the first by nature.
Yes, one is natural selection the other is artificial selection
but the process of selection is the same, the only difference is
the selector and the difference in not relevant as to result
of an evolving system, only different as to what will evolve.
By the way "wanting to live" is a behavior that a machine can
have and it is no more forced or not forced in a man made machine
than in a biological machine.
Just to give one case of a subject that on diving to the bottom
of the pool suddenly felt no need to surface. He had lost all
interest in living or dying. Luckily he was saved and it was found
he had had a stroke damaging the part of the brain the produces
behaviors we call "an act of will".
You have to be careful about the levels of descriptions and levels
of explanation the words you use refer to.
JC
Burkart Venzke
> On Jan 14, 10:12 am, Burkart Venzke <b...@gmx.de> wrote:
>> ...
>>>> ...
>>>> I see, you think of natural systems. They have to fight to
>>>> survive; an AI does not because it is as a robot a machine
>>>> with software.
>>>
>>> It is not about fighting to survive, it is about being selected.
>>> When you design a machine it is an act of selection. The machine
>>> will survive if it has whatever it needs for you to select it.
>>
>> The difference between nature and AI is that the natural beings
>> want to live (survive), the AI as a machine has no such will.
>>
>> The latter selection is made by human (and not really enforced),
>> the first by nature.
>
> Yes, one is natural selection the other is artificial selection
> but the process of selection is the same, the only difference is
> the selector and the difference in not relevant as to result
> of an evolving system, only different as to what will evolve.
The (if you want "little") difference for me is that natural selection
continues running without special efforts in opposite to AI which could
also be stopped e.g. by human laws (sure, it is improbable but possible).
> By the way "wanting to live" is a behavior that a machine can
> have and it is no more forced or not forced in a man made machine
> than in a biological machine.
Yes, we *can* give the machine such a rule but we need not to do it. An
advantage is that humans might be less afraid of machines if they don't
defend their own existence; otherwise machine may more act against
humans (whether they do really or only humans only assume they do it).
If machines "wants to live" they really some day might to want the same
rights we humans have.
> Just to give one case of a subject that on diving to the bottom
> of the pool suddenly felt no need to surface. He had lost all
> interest in living or dying. Luckily he was saved and it was found
> he had had a stroke damaging the part of the brain the produces
> behaviors we call "an act of will".
>
> You have to be careful about the levels of descriptions and levels
> of explanation the words you use refer to.
In what sense more precisely?
Burkart
casey
> > You have to be careful about the levels of descriptions and levels
> > of explanation the words you use refer to.
>
> In what sense more precisely?
Wants and free will are like the gliders and oscillators
in Conway’s game of life vs. the rules at a lower level
that produce those higher patterns. There are no gliders
and oscillators mentioned in the rules. There are no
wants or free will at the level of the neuron.
JC
Don Stockbauer
That's what emergence and the metasystem transition and paradigm
shifts are all about. Here's an exercise for the reader: given a set
of systems and their attributes and their configuration and their
relationships, predict what their metasystem transition will be like.
Burkart Venzke
> On Jan 14, 9:51 pm, Burkart Venzke <b...@gmx.de> wrote:
>> casey schrieb:
>> ,,,
>>> You have to be careful about the levels of descriptions and levels
>>> of explanation the words you use refer to.
>> In what sense more precisely?
>
> Wants and free will are like the gliders and oscillators
> that produce those higher patterns. There are no gliders
> and oscillators mentioned in the rules. There are no
> wants or free will at the level of the neuron.
Your are right. But where is the problem? :)
Burkart
Don Stockbauer
> casey schrieb:
>
> > On Jan 14, 9:51 pm, Burkart Venzke <b...@gmx.de> wrote:
> >> casey schrieb:
> >> ,,,
> >>> You have to be careful about the levels of descriptions and levels
> >>> of explanation the words you use refer to.
> >> In what sense more precisely?
>
> > Wants and free will are like the gliders and oscillators
> > that produce those higher patterns. There are no gliders
> > and oscillators mentioned in the rules. There are no
> > wants or free will at the level of the neuron.
>
> Your are right. But where is the problem? :)
>
> Burkart
It would be nice to be able to predict emergent properties.
Of course, if you shot yourself through the head you'd have no more
problems, right?
Don Stockbauer
Meaning only the rather nihilistic thought that if one has no life one
has no problems. But don't try this at home, children.
Curt Welch
Well, these are the interesting questions of AI that need to be solved.
To start with, I think to deal with the high dimension nature of the
problem, it must be attacked using a distributed system. That is, there
needs to be lots of independent units working together to make the final
decision. They need to somehow vote to determine what to do. They don't
need to (and aren't expected to) agree, but some sort of majority rule sort
of effect should be in effect. However, not all units should get an equal
vote, they should be biased by the current context - by the current state
of the environment.
So to mirror your example from above, if a bell is ringing, the "bell is
ringing" unit should get a vote, and if it's 10 degrees outside, the unit
that represents that aspect of the environment should also get a vote. But
there won't be just 2 or 3 or 10 elements like that which define the
current context of the environment, there will be millions of
micro-features that define the current context, and they should all get
some share of the vote to determine what is the best behavior for the
current situation.
How these units end up voting, will be a statistical function of what sort
of rewards showed up after they get used. The system needs to
statistically track rewards, and assign them to the action selection units
based on how they have acted in the past and on average, what sorts of
rewards resulted.
This is effectively intelligence by a society of simple action units
working together.. They society of action units is trained by rewards,
where only the units that were recently used receive the credit or blame
for the action.
So actions are selected based on how many micro-features the current
environment had in common with past environments, and in how the
micro-features voted in the past, and in what rewards that resulted in.
With this approach, we are not really measuring closeness to a single past
event, but instead, defining the measure of "closeness" to millions of past
events that shared micro features. We don't have to actuality calculate
any metric of "closeness". It just happens automatically. The
environmental state selects which action units need to be us4ed, they all
vote based on their mirco-state information, and the action of the crowd
becomes a combined opinion of the crowd.
To show a highly simplified example to illustrate this, if we have a
decision to make about whether to turn left or right in a maze, when it 20
deg, we have found that turning right got an expected future reward of 10,
and when there was a bell ringing, turning left got the best expected
reward of 8. When ni the situation of 20 deg and bell, the votes are 8
units for left and 10 for right, so we end up turning right - the bell
"measure of closeness" is ranked by statistical rewards to be a "better"
action in this case than the "turn left for bell".
So, we need a good way to parse the environment into current micro-features
using the sensory data. Each micro feature needs to be an action-unit that
can make decisions of some type, and it's decision making needs to be
conditioned by rewards based on when it gets used. The action of the
overall system, needs to be some sort of combined "vote" of the action
units.
I think that approach is a high level answer to how the closeness "metric"
is created, and how actions are selected based on closeness.
But it fails to explain how to implement such concepts, or how to create
the best possible micro-features to define the state of the environment.
My pulse sorting neural networks are attempts to find a working
implementation for these concepts.
Curt Welch
Yes, intelligence is a continuum, but when people use the term "AI" like
Burkart did above, it generally means "the project mankind has been working
on for the past 60 years to crest machines with an complexity of behavior
equal to humans". In that sense, it very much IS a destination.
Curt Welch
> Curt Welch schrieb:
> > Burkart Venzke <b...@gmx.de> wrote:
> >> Hi Curt Welch, hi all,
> >>
> >> You have an interesting but very long discussion in "No easy road to
> >> AI".
> >
> > Yes, I tend to ramble on endlessly.
>
> Too long for me. This thread is long enough for me ;)
>
> >> As we want to solve the problem of AI I also think that learning is
> >> the central problem of it.
> >>
> >> What do you think are the main problems about learning?
> >
> > The implementation details. :)
>
> Not a detailed theory?
That's the same thing to me. If the theory is not detailed enough so that
it can be implemented, then it's really not a "detailed theory" in my book.
> >> One interesting point I have read in the beginning of your discussion
> >> is the connection of learning and perception and that an AI also has
> >> to learn perception. Yes, it is important.
> >>
> >> Another point is how far the internal representation of the AI system
> >> is important.
> >
> > It's highly important. You could say it's everything.
>
> It is when we want to implement it.
>
> >> I think that a least we want to really write software we have
> >> to think about the internal representation. But ok, we may have more
> >> important problems before.
> >
> > Well, I guess you need to be clear about what you mean by the internal
> > representation.
>
> That's true.
>
> > Representation of what? Of the environment? Of the
> > learning system? There are lots of internal represdnations you have
> > to get "right" to solve AI.
>
> It thought of the representation of the learning system.
> Do you mean the real world by "environment"?
Yes, assuming the real world is the environment you are using.
We can create AI algorithms and connect them to a toy environment created
as a computer simulation must as easily - like when we connect a learning
algorithm to a game as it's "environment". The environment is just the
thing the learning agent is interacting with. Of curse, they must always
connect to something in the real world, but when the something is a
computer creating a simple environment, the learning task of the of the
machine becomes much easier.
Reinforcement learning is only a framing of the problem - not the solution.
It's just one small step closer to implementation than saying "build a goal
seeking agent that interacts with an environment">
Reinforcement learning is only "build a goal seeking agent that has the
goal of maximizing a reward signal created by something in the
environment". So to say "reinforcement learning" is only to specify one
small part of the implementation - the goal.
> For me, a possible AI is something with the goal "learn (and act) for
> your teacher" with some sensor (e.g. to see) and actor peripherals in a
> real or play world (or how would you say to an easier world?).
Yes, but we are left with the very hard problem of how the agent is going
to identify the "teacher" in all that complex sensory data and how is it
going to determine what sort of action is "for the teacher" and what action
is "not for the teacher". How does the low level hardware know that making
the right arm move in circles is "for the teacher" or "not for the
teacher"?
And how does it resolve conflicts if the teacher seems to ask for two
conflicting behaviors (say in your seat, go pick up the trash)? Or if
there seem to be two teachers which are each asking for conflicting
actions?
The concept of "act for the teacher" is way too abstract to implement.
The concept of "maximize a reward signal" is precise and easy to implement.
The question is, of all the ways we might try to translate "act for the
teacher" down to hardware what can be implemented, which is the correct way
to do it? I claim that maximizing a reward signal is the one and only
correct way to implement all the ideas of intelligence.
> Assuming (at first) an easy world, the AI first act randomly but with a
> direct connection to its teacher (evualating with good or bad).
> So, it can easy learn which of its actions was good and which bad.
> The teacher has to present good situations/"examples".
> (Also, a channel for easy communication between AI and teacher would be
> nice. "Good" and "bad" would be the first two words.)
Yes, we see that sort of thing happening at the high level. But what is
the low level hardware doing? I claim it's got to be a reinforcement
learning system.
> > On you quesiton about what is learning, it's just the fact that human
> > behavior is not constant. It changes over our life time. So at the
> > most abstract layer, we can just say that learning is a change in
> > behavior. But that gives us no answer as to the direction of change.
> > Why might one change in behavior be called learning, and another be
> > called random nonsense? It's because learning typically assumes a
> > change for a purpose, or a change for "the better".
>
> Purpose, "the better"...
> My central point is that humans have goals which are their central
> control - independently if they consious or not, after hard planning or
> only emotion etc.
Well, these goals happen at many levels in us. When I walk down the hall,
it can be said I have the goal of walking in a roughly straight line and a
goal of not hitting the walls and a goal of not falling down. This sort of
walking behavior has been conditioned into us. That is, we have hardwired
connections in our brain to make us perform that behavior - and the wiring
of that network is such as to make us move as to "reach those goals". But
there probably is nothing int he wiring that makes the goal obvious. It's
just that we have a good collection of actions wired into us that make us
walk down a hall in that way. But how did our brain get wired that way?
That's a question to be answered by explaining how our learning system
works - that is, how the brain changes the way it's wired over time. So
those types of "goals" are "hidden" in the way our brain is wired.
But we also have short term goals, such as "I want to go to the store",
which will direct our action for the extended time it takes to get to the
store. That goal is not wired into us. It's something held in some sort
of short term memory. We are wired to response to that temporary goal by
producing the right actions, but the goal in that case is not in our
wiring, it's in some sort of temporary memory units of our brain. SO
that's yet enough type of goal with yet a different type of implementation
in the hardware of the brain.
And yet another are the goals implicit in the way our reward system is
wired. We have a reward system that makes damage to the body (like heat
high enough to burn) a signal that directs the learning system to change
the behavior to prevent a repeat of the same condition. So the learning
system has motivations built into it, that causes it to build behavior
generation circuits, that show goal behavior in our action - the goal of
not being burnt again. So by seeing how our behavior change - seeing how
we learn - we can see what the goals of the learning system is. And that's
yet another way that goals exist in the implement ion of our brain.
> > But when we say these things, like "change for the better" we have just
> > pushed the question of what is learning, off to a problem of defining
> > what we mean by better. Reinforcement learning answers that questions.
>
> Does a teacher decides or who (what)?
Evolution built into us a reward system (reward signal) that directs how
learning works. It's hard wired into us to make some things like damage to
the body, or an empty stomach, create a negative reward signal, and other
things, like the act of eating when hungry, or sex, generate a postie
reward signal.
The learning hardware than adjusts our behavior in response to rewards (and
more important, in response to estimations of future rewards).
The teacher is just part of the environment we learn to deal with so as to
get more rewards. If we don't respect the teacher, and produce the
behavior they want us to produce, we won't get as many rewards.
> > Other answers people have used to the question is to describe learning
> > behavior as goal seeking. That is, our behavior changes to better
> > reach some goal. But then we are left asking the question of what a
> > goal is, or what the goal should be for our AI solution. Reinforcement
> > learning abstracts the definition of what the goal is to reward
> > maximizing.
>
> I like the idea of goals, "reward maximizing" is too abstract for me.
Yes, it takes some time to understand. It's not easy to grasp at first.
> But perhaps we only fight for words and mean nearly the same.
> Goals are something we can implement, we can imagine something - better
> than an abstract function. Or is the function for reward maximizing
> something else?
I can build a little robot that runs away from bright light. By looking at
its behavior, we can say its goal is "to get away from the light". But all
I have done is built a very simple sensory system with some simple logic
for driving the wheels from the light sensors. No where in the schematic
do we see something that looks like "the goal of running away from the
light". Only by studying the schematic so as to understand what type of
behaviors this system will produce, can we then label the machine as
"waning to get away from the light".
Goals more often just natural langauge labels for complex behavior produced
by complex hardware. We say the behavior has a goal if we can identify
9wth words) some unifying purpose of the behaviors.
> > Reinforcement learning frames the problem of learning as an agent,
> > which interacts with an environment. Which means it has inputs from
> > the environment (sensors) and outputs to the environment (effectors).
> > But in addition, it also receives a single dimension reward signal from
> > the environment.
>
> So far, so well. Besides the teacher, the world also may be this
> environment when there is a basis for good and bad.
> A human has its emotions as a basis, but an AI?
The AI has it's reward as it's basis for good and bad. That's how humans
get their basis for good and bad. Well, at least, it's the foundation from
where all our ideas of good and bad grow from. It's just some stupid
hardware in us that makes us sense something things as "pain" and some as
"pleasure". pain is bad, pleasure is good, and all other forms of good and
bad grow from that simple fact of how the human body is wired.
Of course, our DNA is the source of how we got wired, and the DNA was
created by natural selection, so the way we get wired for "good" and "bad"
is held closely in sync with the things that help us survive, by this
process of evolution.
Eating a lot was "good" long ago, because it took a lot of work to get even
a little food. So our DNA wired us to really like food - so as to motivate
us to expend a lot of energy getting it. But now that food is far easier to
get, most of us are wired to like food too much - so many of us eat too
much. We are still wired to expend lots of our effort at food collection,
but that much effort in these days means we collect and consume too much
food. Natural selection will slowly kill off the people that eat too much,
and in the process reduce the human desire to eat to match the rough level
of effort we really need to expend to get food. But that will take many
generations.
> > The goal of the agent in this abstract framework, is to maximize some
> > long term measure of the reward signal. There is no explicit end-goal
> > in reinforment learning. That is, learning never ends.
>
> Right. It is also my idea of learning because no AI (or human) ever can
> be perfect in a complicated (e.g. real) world.
>
> > The assumption is the
> > agent will continue to try and find ways to get more rewards - even
> > though there might not be any better ways to get rewards than what it's
> > currently doing.
> >
> > Though we say in this framework that the reward comes from the
> > environment, when we build reinforcement learning into something like a
> > robot, we also have to build the hardware which generates the reward
> > signal. But that hardware, is conceptually outside the "reinforcement
> > learning hardware" module. From the perspective of the learning
> > module, the reward comes from the environment of the learning module.
> >
> > The problem with reinforcement learning, is that's it's easy to
> > specify, but extremely hard to implement. In fact, no one has
> > implemented a good generic reinforcement learning machine.
>
> I think that the (or a) problem is that the "correct" reward signal is
> not always unique. What is really "correct"? (Besides well-known
> physical laws and something like that.)
The reward signal is mostly ignored in the study of reinforcement learning.
There is no "right" or "wrong" reward signal. It's totally arbitrary. It
can be anything the builder picks. However, we are forced to build
hardware that generates the reward signal, and whatever that hardware does,
ends up defining the goal of the machine.
If I build a reinforcement learning robot rat, I can give it any reward
signal I want to. I can give it reward signals created from a light sensor
that makes it get more rewards for more light. Such a rat would learn
beahviors for keeping itself in the light.
Or I can wire it the other way around, and give it more rewards for LESS
light. The same rat would the learn behaviors for staying in the dark.
The learning algorithm is exactly the same whether I give it the goal of
"say in the dark" or "stay in the light". When we build the reward
hardware, we are indirectly giving it a goal.
Solving the problem of building better reinforcement learning algorithms
really has nothing to do with what reward signal we give it.
Humans however have a very complex system for generating our reward signal.
It has been tuned by the process of evolution to make us search for
behaviors that also happen to be good at helping us survive. So
indirectly, though the heavy hand of natural selection, we can say our goal
is to survive. But the how that is implemented, is by having hardware with
a goal of maximizing a reward signal, and a reward generating system, that
has a large selection of stuff defined as "good" and "bad" for us (and a
carefully turned relative measure of the different rewards.
But it must not be "multiple rewards". It must always boil down to a
single rewards system. If there are multiple rewards, the must still be
something that combines them down to a single reward.
> > There are many reinforcement learning algorithms that have already been
> > created. Most however, only work (or are only practical) for small toy
> > environments - like the game of tic tac toe or something similar. This
> > is because they require the algorithm to track a statistical value for
> > every state the environment can be in. When there are only thousands
> > of states, like the number of possible board positions in tic tac toe,
> > and the agent has enough time to explore all the states many times,
> > then the current algoerithms can converge on optimal (perfect)
> > decisions fairly quickly.
> >
> > But as the environment becomes more complex (more states it can be) the
> > simple RL algorithms fail - because it quickly becomes impossible to
> > track a value for each possible state, (not enough memory in the world
> > to track a value for every board game in Go). So some other approach
> > has to be used. These are called high dimension problem spaces because
> > the number of states of the sensory inputs have an effectively how
> > number of dimensions - aka multiple sensors acting in parallel - which
> > just means the total state space is huge.
> >
> > No one has found a good generic solution to this problem but many
> > people are looking for them.
>
> Here, for example for tic tac toe and further on for Go, symbols are my
> idea like "line" or "area".
> Also therefore, I want a communication between human and KI so that the
> human can define a (for the AI) new word like "line".
People have tried to solve AI at the level of words, and haven't gotten
very far. But there might be a way to make it work still.
whoever builds the machine.
There are plenty of reinforcement learning algorithms to look at. But if
you mean do any of them act like a human yet, no. None of the current
algorithms work well in the high dimension environment that humans operate
in. In the toy environments they do work in, their behavior is too
simplistic for anyone to see "obvious human intelligence" in it.
Curt Welch
> Curt Welch schrieb:
Your post was much longer, but I think I've basically covered all the other
points in other messages. But not this one:
> Is it correct that you think of a rebuild of the brain?
> My idea is a symbolic one.
Much work on AI in the first decades tried to attack the problem at the
abstract symbolic level of words (or abstract concepts at about that
level). The idea I guess is that since we can learn lots of important
information by reading books, an AI built at the level of symbol processing
should be able to learn to read, and "think" in symbolic (word) terms like
we do, and that we would then have a highly intelligent AI that could "read
the internet" and understand it like we do, and we could then talk to that
AI like we talk to each other using these words.
None of those projects reached the goal of a machine that could think using
symbols like we do. The general conclusion, is that no matter how much
information they tried to encode into it's system using symbols and
relationships between symbols, (humans have feet, humans haev names, Curt
is the name of a human, and on and on), the machine was still lacking a
huge bulk of "common sense".
Or, another way to look at it, if you tried to program into the machine,
the knowledge of a dictionary, and everything in wikipedia, the machine
would still be missing far more undersigning than it had. After all that
data is put into the machine, it might not know for example, that humans
didn't have tails - because that just happened to be a fact that wasn't in
the dictionary, or in wikipedia. It's almost as if the more information
you tried to put into the machine, the more you realized how much it was
missing. Each thing you taught it, was like one step forward, and 10 steps
backwards.
These sorts of symbolic based projects are generally known as GOFAI - Good
Old Fashion AI. And they are are generally considered as good ideas which
produced a lot of interesting results, they ultimately seem to have been a
dead end to creating full AI.
I think the lesson learned there is that you can't hand-code knowledge into
these machines. They have to learn it by experience instead. They have to
learn it by interacting with an environment.
Now maybe we can build a useful, and interesting AI, by building a machine
that has a dictionary of words, and which produces output by selecting from
the list of words - it produces a stream of words from it's diction as
output. Seeing how we can make such a machine learn though interaction
with humans would be fun, but ultimately, what such a machine could
understand, is limited to what it can sense - words. It would have no real
understanding of what it would be like to have a body that existed in a 3D
environment. It would have no ability to relate to what a smell was, or a
vision, or a warm touch. So even if it was highly intelligent and was
producing some real advanced sentences as we interacted with it, it would
be an odd thing - like an extreme case of trying to communicate to a blind
and death person what the sight of a sunset was like. Much of what such a
machine would read about on the intent it wouldn't be able to relate to.
It could learn to mimic how other people talked about their experience, but
it would have no real understand of what it was talking about. It would
only know the words.
Maybe such an AI would actually be very useful to us as agents that could
do things like search the internet for us and report back what it found.
It would be cool us such a thing could be created.
But ultimately, I would like to build intelligent robots - and as such,
they need far more IO power than a stream of words. They need the same
sort of bandwidth sensor and effectors we have, so they can interact in
eral time with humans, and do things like drive a car, like we do. And for
that, we need a level of "symbol processing" where the symbos are more like
spikes at rates more like a million per second.
A word-based-AI that was dealing with symbols at the rates of less than 5
per second (talking rate) would be a far easier machine to make work than
one ethat had to deal with sensory and effector information rates of 100
Mbits. But I think both would basically end up having very similar data
processing going on if they were to act intelligent in either domain.
Burkart Venzke
Precise interpretation of my intention :)
If it has to be more clear I would speak of weak (=existing) and strong
(our goal) AI.
Yes, it is also a continuum but a lot of things are continua like
colors: What is red? What is orange? Where is the border? It may be
defined but normally our language (everyday speech) is ambiguous with
such ideas.
Concerning to AI, the border (weak - strong) is also not clear, it is
also altered through the years...
I think it is only important that we know what we mean - and if not to
ask and make it clear for us.
Don Stockbauer
> Curt Welch schrieb:
Effective communication is always of benefit when forming higher level
systems than oneself.
Burkart Venzke
No question, right. Also AI needs communication to get to a higher level.
Burkart Venzke
> Burkart Venzke <b...@gmx.de> wrote:
>> Curt Welch schrieb:
>
> Your post was much longer, but I think I've basically covered all the other
> points in other messages. But not this one:
>
>> Is it correct that you think of a rebuild of the brain?
>> My idea is a symbolic one.
>
> Much work on AI in the first decades tried to attack the problem at the
> abstract symbolic level of words (or abstract concepts at about that
> level). The idea I guess is that since we can learn lots of important
> information by reading books, an AI built at the level of symbol processing
> should be able to learn to read, and "think" in symbolic (word) terms like
> we do, and that we would then have a highly intelligent AI that could "read
> the internet" and understand it like we do, and we could then talk to that
> AI like we talk to each other using these words.
Without a better base these system cannot work well because they have no
connection to the real world, to understand what the words really mean
in the world.
> None of those projects reached the goal of a machine that could think using
> symbols like we do. The general conclusion, is that no matter how much
> information they tried to encode into it's system using symbols and
> relationships between symbols, (humans have feet, humans haev names, Curt
> is the name of a human, and on and on), the machine was still lacking a
> huge bulk of "common sense".
That's right. Symbols alone cannot be the solution of AI. The AI also
needs contact with the world (input and output devices) and (a) goal(s).
> Or, another way to look at it, if you tried to program into the machine,
> the knowledge of a dictionary, and everything in wikipedia, the machine
> would still be missing far more undersigning than it had. After all that
> data is put into the machine, it might not know for example, that humans
> didn't have tails - because that just happened to be a fact that wasn't in
> the dictionary, or in wikipedia. It's almost as if the more information
> you tried to put into the machine, the more you realized how much it was
> missing. Each thing you taught it, was like one step forward, and 10 steps
> backwards.
I agree.
> These sorts of symbolic based projects are generally known as GOFAI - Good
> Old Fashion AI. And they are are generally considered as good ideas which
> produced a lot of interesting results, they ultimately seem to have been a
> dead end to creating full AI.
I may seem so. I know that this way is not the modern way (not the state
of art).
> I think the lesson learned there is that you can't hand-code knowledge into
> these machines. They have to learn it by experience instead. They have to
> learn it by interacting with an environment.
Exactly! But why shouldn't it be possible with a mainly symbolic approach?
> But ultimately, I would like to build intelligent robots - and as such,
> they need far more IO power than a stream of words. They need the same
> sort of bandwidth sensor and effectors we have, so they can interact in
> eral time with humans, and do things like drive a car, like we do. And for
> that, we need a level of "symbol processing" where the symbos are more like
> spikes at rates more like a million per second.
I think that we need both: Fast processing for sensor data, if necessary
also fast processing of effectors (but that is not so important), but
primarily a good internal processing which not necessarily needs to be
very fast - only "intelligent".
Don Stockbauer
> It's almost as if the more information
> > you tried to put into the machine, the more you realized how much it was missing.
Kinda describes humans too, innit?
Burkart Venzke
We could think so especially when thinking of Sokrates' "I know that I
know nothing".
I think it is a question of perspective, of individual curiosity, and of
how we weight the knowledge against what we don't know.
But that is not too important here ;)
The main difference is that the humans' knowledge is quite close to the
world so that humans are able to act quite rational in it.
Such an AI would only have knowledge on an abstract level without (much)
usable general knowledge. We could imagine that its knowledge won't let
act it rational at all e.g. it has no goals (or feelings or emotions
which may generate goals).
Curt Welch
> Curt Welch schrieb:
> > Burkart Venzke <b...@gmx.de> wrote:
> >> Curt Welch schrieb:
> >
> > Your post was much longer, but I think I've basically covered all the
> > other points in other messages. But not this one:
> >
> >> Is it correct that you think of a rebuild of the brain?
> >> My idea is a symbolic one.
> >
> > Much work on AI in the first decades tried to attack the problem at the
> > abstract symbolic level of words (or abstract concepts at about that
> > level). The idea I guess is that since we can learn lots of
> > important information by reading books, an AI built at the level of
> > symbol processing should be able to learn to read, and "think" in
> > symbolic (word) terms like we do, and that we would then have a highly
> > intelligent AI that could "read the internet" and understand it like we
> > do, and we could then talk to that AI like we talk to each other using
> > these words.
>
> Without a better base these system cannot work well because they have no
> connection to the real world, to understand what the words really mean
> in the world.
Yeah, I've wondered what could be done with a machine that simply
interacted with other humans by talking. It would have no eyes or ears or
arms and legs to allow it to gain an understanding of the world we live in,
but it would have sensors - it would sense what it was saying, and what
what person talking to it was saying. What sort of AI might that create?
How interesting would such an AI become just by spending thousands, or
millions of hours talking to millions of different people? It would need a
goal of course, but the goal could be simple - the humans it talks to would
give it rewards, or a score, and the AI would try to maximize those scores
from the humans. What it would be learning, is what humans like to talk
about - and what humans like to hear the AI talk about. With the right AI
software driving it, I think such an AI could be at minimal very
entertaining to talk to, even though it could have no real understanding of
the world we live in. But yet, it could learn how to correctly talk about
our world, even if it didn't know what the world was really like though
direct sensations.
I think with the right AI software driving it, such a machine would be very
interesting, and would seem highly intelligent to the people it talked
with.
Tim Tyler
> Yeah, I've wondered what could be done with a machine that simply
> interacted with other humans by talking. It would have no eyes or ears or
> arms and legs to allow it to gain an understanding of the world we live in,
> but it would have sensors - it would sense what it was saying, and what
> what person talking to it was saying. What sort of AI might that create?
Probably one like Google - but better. People chat with Google all
the time. No doubt in the future, the chats will more closely resemble
conversations.
--
__________
|im |yler http://timtyler.org/ t...@tt1lock.org Remove lock to reply.
casey
> Burkart Venzke <b...@gmx.de> wrote:
> Curt Welch schrieb:
>> > Burkart Venzke <b...@gmx.de> wrote:
>>>
>> >> My idea is a symbolic one.
>>>
>>>
>>> Much work on AI in the first decades tried to attack the
>>> problem at the abstract symbolic level of words (or abstract
>>> concepts at about that level). The idea I guess is that
>>> since we can learn lots of important information by reading
>>> books, an AI built at the level of symbol processing should
>>> be able to learn to read, and "think" in symbolic (word)
>>> terms like we do, and that we would then have a highly
>>> intelligent AI that could "read the internet" and understand
>>> it like we do, and we could then talk to that AI like we
>>> talk to each other using these words.
>>
>>
>> Without a better base these system cannot work well because
>> they have no connection to the real world, to understand
>> what the words really mean in the world.
>
>
> Yeah, I've wondered what could be done with a machine that
> simply interacted with other humans by talking. It would
> have no eyes or ears or arms and legs to allow it to gain
> an understanding of the world we live in, but it would have
> talking to it was saying. What sort of AI might that create?
>
> How interesting would such an AI become just by spending
> thousands, or millions of hours talking to millions of
> different people? It would need a goal of course, but the
> goal could be simple - the humans it talks to would give
> it rewards, or a score, and the AI would try to maximize
> those scores from the humans. What it would be learning,
> is what humans like to talk about - and what humans like
> to hear the AI talk about. With the right AI software
> driving it, I think such an AI could be at minimal very
> entertaining to talk to, even though it could have no real
> understanding of the world we live in. But yet, it could
> learn how to correctly talk about our world, even if it
> didn't know what the world was really like though direct
> sensations.
>
> I think with the right AI software driving it, such a
> machine would be very interesting, and would seem highly
> intelligent to the people it talked with.
The innate or learned framework that a child uses to build
language upon is the Rosetta stone required to translate
all this high level internet talk into useful knowledge to
enable it to have any sensible conversation.
Instead of the trying to feed in millions of internet
exchanges into the "right AI software" I would suggest
the much simpler approach of feeding it the exchanges
adults have with a child learning to speak.
JC
J.A. Legris
Maybe you should talk more to your dog. It has a built-in, thoroughly
debugged reinforcement learning system. Isn't that equivalent to the
"right AI" ?
--
Joe
Curt Welch
> On Jan 30, 3:41=A0pm, c...@kcwc.com (Curt Welch) wrote:
> Maybe you should talk more to your dog. It has a built-in, thoroughly
> debugged reinforcement learning system. Isn't that equivalent to the
> "right AI" ?
Sorry, my last dog died. We just have two cats now. She doesn't talk back
much anymore - she's just a box of ashes! :)
Yes, dogs certainly have the ability to understand a few words. But their
temporal pattern length and pattern matching set sizes are far too short to
correctly recognize English phrases. It's obvious not "the right AI" for
speaking English even if it's the right AI technology. That is, it's not
configured correctly for our level of langauge even if it's the same basic
learning technology we use for language (which is what I strongly suspect
is the case).
I think if we had the right AI technology for our computers, we might well
be able to configure it to be optimal for language, and nothing else, and
end up with something that creates some very interesting langauge
interaction with far less total size and complexity than what it would take
to also process all the other sense data.
--
Curt Welch http://CurtWelch.Com/
Wolfgang Lorenz
> Humans however have an increase quantity of parietal cortex
> called the inferior parietal lobule ...
>
> Damage to this area results in a condition called apraxia.
> The subject is unable to make a cup of tea or put on their
> clothes or any other complex sequential act of this kind.
>
> It main ability lost is the production of a proper sequence
> of actions to carry out a task.
How does the hippocampus fit into this picture?
I've read somewhere that there are loops within hippocampus, e.g. if the
symbols A, B, and C happen in reality, overlapped, and at a larger
timescale, then hippocampus will compress that sequence and run
A-B-C-A-... in a loop with a few hundred milliseconds duration.
Maybe hippocampus is only used for storing new sequences during the day
and writing them into cortex over night, and cortex will later be able
to play back that sequences alone. This would mean that people with a
damaged hippocampus will be able to play back the old sequences they
have learnt before the damage, but that they cannot learn new sequences.
--
Wolfgang
casey
> casey wrote:
>> Humans however have an increase quantity of parietal cortex
>> called the inferior parietal lobule ...
>>
>> Damage to this area results in a condition called apraxia.
>> The subject is unable to make a cup of tea or put on their
>> clothes or any other complex sequential act of this kind.
>>
>>
>> It main ability lost is the production of a proper sequence
>> of actions to carry out a task.
>
>
> How does the hippocampus fit into this picture?
I don't know how the hippocampus fits into the behavioral
disability of apraxia.
I actually deleted the post you quote from. Apparently deleting
in google didn't happen quickly enough? Anyone who wanted to
know about apraxia could always google it and I realized what
I had written wasn't really useful in understanding anything.
> I've read somewhere that there are loops within hippocampus,
> e.g. if the symbols A, B, and C happen in reality, overlapped,
> and at a larger timescale, then hippocampus will compress that
> sequence and run A-B-C-A-... in a loop with a few hundred
> milliseconds duration.
>
>
> Maybe hippocampus is only used for storing new sequences
> during the day and writing them into cortex over night, and
> cortex will later be able to play back that sequences alone.
> This would mean that people with a damaged hippocampus will
> be able to play back the old sequences they have learnt
> before the damage, but that they cannot learn new sequences.
Yes there is a lot of theories about the role of the hippocampus
but apart from saying it has something to do with memory there
isn't much I can write apart from regurgitating things I have
read which others can read for themselves if interested.
JC
Don Stockbauer
Innit the hippocampus where hippo-pot-a-musses go to school?
Burkart Venzke
I think, too, that an AI should be fed with (very) easy language (at
least at first). This should be associated with data from the real (or a
toy) world like
1. "(This is a) car" (a word or easy sentence)
and
2. a picture of or a real view to a car.
Only speech is not enough for real learning (my view).
casey
> ...
> Only speech is not enough for real learning (my view
Too high level rather than developing the foundations which makes
using a language possible,
JC
Burkart Venzke
Would you like a baby to learn his mother language without (other)
connections to the world?
Developing the foundations is fine. These foundations don't have to be
learned by an AI, right?
Do you perhaps want to learn something on base of these foundation
independently from the outer world? If so, what? And were should the
data to learn come from?
Curt Welch
Well, langauge is a highly complex learned behavior - one of the most
complex behaviors humans produce. If you want to start with something
simpler, don't start with language. Try making a robot learn to walk first
or learn to plug itself into a charger. Then after you got that done, try
and make it learn to talk about how it walks and how it plugs itself into a
charger.
Burkart Venzke
> Burkart Venzke <b...@gmx.de> wrote:
>> casey schrieb:
>>> On Feb 3, 11:08 am, Burkart Venzke <b...@gmx.de> wrote:
>>>> ...
>>>> Only speech is not enough for real learning (my view
>>> Too high level rather than developing the foundations which makes
>>> using a language possible,
>> Would you like a baby to learn his mother language without (other)
>> connections to the world?
>>
>> Developing the foundations is fine. These foundations don't have to be
>> learned by an AI, right?
>> Do you perhaps want to learn something on base of these foundation
>> independently from the outer world? If so, what? And were should the
>> data to learn come from?
>
> Well, langauge is a highly complex learned behavior - one of the most
> complex behaviors humans produce.
Language can be very complex but is it not necessarily so. So many
animals have a language often with an relatively easy syntax.
(An animal AI would be also a step towards the final goal (but not mine).)
> If you want to start with something
> simpler, don't start with language. Try making a robot learn to walk first
> or learn to plug itself into a charger. Then after you got that done, try
> and make it learn to talk about how it walks and how it plugs itself into a
> charger.
Yes, that is a plausible way to learn, to learn to act in a world with
may contain no other intelligence.
My idea is not (only) to learn from the unintelligent world but also
from our intelligence (in form of the teacher).
casey
> ...
>> On Feb 3, 11:08 am, Burkart Venzke <b...@gmx.de> wrote:
>>> ...
>>> Only speech is not enough for real learning (my view
>>
>> Too high level rather than developing the foundations which
>> makes using a language possible,
>
>
> Would you like a baby to learn his mother language without
> (other) connections to the world?
I don't see how words can have any meaning without reference
to the real world. Words become associated with things that
are already there, be they learnt or innate. People understand
reality independently of any words used to describe it.
> Developing the foundations is fine. These foundations don't
> have to be learned by an AI, right?
It can be an innate foundation resulting from natural selection.
JC
Burkart Venzke
> On Feb 3, 9:50 pm, Burkart Venzke <b...@gmx.de> wrote:
>> ...
>>> On Feb 3, 11:08 am, Burkart Venzke <b...@gmx.de> wrote:
>>>> ...
>>>> Only speech is not enough for real learning (my view
>>> Too high level rather than developing the foundations which
>>> makes using a language possible,
>>
>> Would you like a baby to learn his mother language without
>> (other) connections to the world?
>
> I don't see how words can have any meaning without reference
> to the real world. Words become associated with things that
> are already there, be they learnt or innate.
Agreed.
> People understand
> reality independently of any words used to describe it.
But don't words respectively notions help to understand reality better?
A word/notion is something like a kernel for a cloud of ideas.
E.g. a bridge is something for easier crossing of something else.
It's only one word which also can be taught (and mentally kept/learned)
as "bridge" with all its associations, learned first with its notion and
description (or attributes) and later be expanded e.g. with real bridges
like the "Golden Gate Bridge". (Or the other way round: First real
bridges as examples and then "bridge" as common notion.)
Words are even more important for more abstract things as philosophical
notions like "dialectic" or as statements like "cogito ergo sum".
>> Developing the foundations is fine. These foundations don't
>> have to be learned by an AI, right?
>
> It can be an innate foundation resulting from natural selection.
"Natural" selection of an "artificial" intelligence? ;)
Let us first construct an acceptable AI example before we think of
selection.
casey
> casey schrieb:
>
> > People understand
> > reality independently of any words used to describe it.
>
> But don't words respectively notions help to understand reality better?
> A word/notion is something like a kernel for a cloud of ideas.
But we must _start_ with something other than language.
JC
Burkart Venzke
I think we should start with the importance/relevance of the combination
of sounds (signs etc.) with something happening in the world.
casey
Have you read much about language? A popular thinker is Steven Pinker
who has written easy to read books such as "The Language Instinct" and
"The Stuff of Thought" which provide interesting insights into
language
and how we use it.
JC
Curt Welch
I argue that we start with a signal processing system. In the case of
using pulse signals, each pulse itself acts as very low level "symbol" on
which all higher level understanding is built in the sort of systems I work
with.
If you use binary as your low level signal format, then the 1's and 0's
become the low level symbols that you build all understanding from.
The meaning of these signals is defined by what causes them - by the
sensory systems that first generate them, and by the internal processing of
those signals that generate new signals with associated new meaning as the
result of the signal.
For example, if I have a wall contact sensor that generates a pulse when it
hits a wall, then the meaning of the pulses (and the symbol) coming from
that sensory is "we just hit something".
If we have a second sensory that detects if we are moving forward, and
genrates a pulse for each 1mm moved forward, the meaning of those pulse is
"just moved 1mm forward".
If we have a processing node that detects when these two pulse happen
within .2 seconds of each other, and produces a pulse each time it happens,
then the output of that node has meaning "hit something while moving
forward in the past .2 seconds".
The meaning of the low level signals is the meaning on which all higher
level understanding (and eventually natural langauge systems which is
output by the AI) is all grounded back to the the low level signals in the
system and their meaning as defined by the hardware (and associated
physical events) that are the cause of the signal.
Burkart Venzke
>> For me, a possible AI is something with the goal "learn (and act) for
>> your teacher" with some sensor (e.g. to see) and actor peripherals in a
>> real or play world (or how would you say to an easier world?).
>
> Yes, but we are left with the very hard problem of how the agent is going
> to identify the "teacher" in all that complex sensory data and how is it
> going to determine what sort of action is "for the teacher" and what action
> is "not for the teacher". How does the low level hardware know that making
> the right arm move in circles is "for the teacher" or "not for the
> teacher"?
>
> And how does it resolve conflicts if the teacher seems to ask for two
> conflicting behaviors (say in your seat, go pick up the trash)? Or if
> there seem to be two teachers which are each asking for conflicting
> actions?
>
> The concept of "act for the teacher" is way too abstract to implement.
>
> The concept of "maximize a reward signal" is precise and easy to implement.
>
> The question is, of all the ways we might try to translate "act for the
> teacher" down to hardware what can be implemented, which is the correct way
> to do it? I claim that maximizing a reward signal is the one and only
> correct way to implement all the ideas of intelligence.
First: Yes, the concept of "act for the teacher" is quite abstract and
must be specified better.
A 'young' learning AI should act as precisely as in can for the teacher.
So, if the teacher communicate with the AI, its most important task is
to listen to the teacher. Later it may say something like "shortly" to
the teacher when it is acting (making, doing) something valued as
important/swift.
Right, a reward signal is also important. But how is your AI acting if
it has currently no special task to do?
>> Assuming (at first) an easy world, the AI first act randomly but with a
>> direct connection to its teacher (evualating with good or bad).
>> So, it can easy learn which of its actions was good and which bad.
>> The teacher has to present good situations/"examples".
>> (Also, a channel for easy communication between AI and teacher would be
>> nice. "Good" and "bad" would be the first two words.)
>
> Yes, we see that sort of thing happening at the high level. But what is
> the low level hardware doing? I claim it's got to be a reinforcement
> learning system.
Agreed.
>>> On you quesiton about what is learning, it's just the fact that human
>>> behavior is not constant. It changes over our life time. So at the
>>> most abstract layer, we can just say that learning is a change in
>>> behavior. But that gives us no answer as to the direction of change.
>>> Why might one change in behavior be called learning, and another be
>>> called random nonsense? It's because learning typically assumes a
>>> change for a purpose, or a change for "the better".
>> Purpose, "the better"...
>> My central point is that humans have goals which are their central
>> control - independently if they consious or not, after hard planning or
>> only emotion etc.
>
> Well, these goals happen at many levels in us.
Exactly!
> When I walk down the hall,
> it can be said I have the goal of walking in a roughly straight line and a
> goal of not hitting the walls and a goal of not falling down. This sort of
> walking behavior has been conditioned into us. That is, we have hardwired
> connections in our brain to make us perform that behavior - and the wiring
> of that network is such as to make us move as to "reach those goals". But
> there probably is nothing int he wiring that makes the goal obvious. It's
> just that we have a good collection of actions wired into us that make us
> walk down a hall in that way. But how did our brain get wired that way?
> That's a question to be answered by explaining how our learning system
> works - that is, how the brain changes the way it's wired over time. So
> those types of "goals" are "hidden" in the way our brain is wired.
I don't think about a wired brain (although you and other do), I think
of a software (AI, robot) working with symbols. So, a set or better a
hierarchy or network of new goals can be learned.
> But we also have short term goals, such as "I want to go to the store",
> which will direct our action for the extended time it takes to get to the
> store. That goal is not wired into us. It's something held in some sort
> of short term memory. We are wired to response to that temporary goal by
> producing the right actions, but the goal in that case is not in our
> wiring, it's in some sort of temporary memory units of our brain. SO
> that's yet enough type of goal with yet a different type of implementation
> in the hardware of the brain.
For me, these are only _more temporary_ goals.
> And yet another are the goals implicit in the way our reward system is
> wired. We have a reward system that makes damage to the body (like heat
> high enough to burn) a signal that directs the learning system to change
> the behavior to prevent a repeat of the same condition. So the learning
> system has motivations built into it, that causes it to build behavior
> generation circuits, that show goal behavior in our action - the goal of
> not being burnt again. So by seeing how our behavior change - seeing how
> we learn - we can see what the goals of the learning system is. And that's
> yet another way that goals exist in the implement ion of our brain.
I think that a lot (most?) goals are only temporarily active.
When I sit somewhere the goal of walking a straight line is unimportant.
When I have just has been in a store (and have bought all I need) there
is no active goal "I want to go to the store".
>>> But when we say these things, like "change for the better" we have just
>>> pushed the question of what is learning, off to a problem of defining
>>> what we mean by better. Reinforcement learning answers that questions.
>> Does a teacher decides or who (what)?
>
> Evolution built into us a reward system (reward signal) that directs how
> learning works. It's hard wired into us to make some things like damage to
> the body, or an empty stomach, create a negative reward signal, and other
> things, like the act of eating when hungry, or sex, generate a postie
> reward signal.
Yes, we humans have a complex reward system. We may give an AI a similar
complex one, too, but it is not necessary and perhaps risky. I don't
want an AI to steal food (ok, electricity) only because of its need.
> The learning hardware than adjusts our behavior in response to rewards (and
> more important, in response to estimations of future rewards).
>
> The teacher is just part of the environment we learn to deal with so as to
> get more rewards. If we don't respect the teacher, and produce the
> behavior they want us to produce, we won't get as many rewards.
Yes, with your idea of environment and reward system.
For me, the teacher is the source of rewards, but he can 'say' "breaking
something is bad" so that this rule is substitution for an hard wired
reward.
>>> Other answers people have used to the question is to describe learning
>>> behavior as goal seeking. That is, our behavior changes to better
>>> reach some goal. But then we are left asking the question of what a
>>> goal is, or what the goal should be for our AI solution. Reinforcement
>>> learning abstracts the definition of what the goal is to reward
>>> maximizing.
>> I like the idea of goals, "reward maximizing" is too abstract for me.
>
> Yes, it takes some time to understand. It's not easy to grasp at first.
I am not sure if we think of the same problem.
The problem is a "reward maximizing" function:
Where does it come from? Is it fix? When yes: Why is so as it it?
>> But perhaps we only fight for words and mean nearly the same.
>> Goals are something we can implement, we can imagine something - better
>> than an abstract function. Or is the function for reward maximizing
>> something else?
>
> I can build a little robot that runs away from bright light. By looking at
> its behavior, we can say its goal is "to get away from the light". But all
> I have done is built a very simple sensory system with some simple logic
> for driving the wheels from the light sensors. No where in the schematic
> do we see something that looks like "the goal of running away from the
> light". Only by studying the schematic so as to understand what type of
> behaviors this system will produce, can we then label the machine as
> "waning to get away from the light".
A goal should not be only an interpretation of a system's behavior but
something an AI really wants to achieve.
>>> The problem with reinforcement learning, is that's it's easy to
>>> specify, but extremely hard to implement. In fact, no one has
>>> implemented a good generic reinforcement learning machine.
>> I think that the (or a) problem is that the "correct" reward signal is
>> not always unique. What is really "correct"? (Besides well-known
>> physical laws and something like that.)
>
> The reward signal is mostly ignored in the study of reinforcement learning.
> There is no "right" or "wrong" reward signal. It's totally arbitrary. It
> can be anything the builder picks. However, we are forced to build
> hardware that generates the reward signal, and whatever that hardware does,
> ends up defining the goal of the machine.
>
> If I build a reinforcement learning robot rat, I can give it any reward
> signal I want to. I can give it reward signals created from a light sensor
> that makes it get more rewards for more light. Such a rat would learn
> beahviors for keeping itself in the light.
>
> Or I can wire it the other way around, and give it more rewards for LESS
> light. The same rat would the learn behaviors for staying in the dark.
>
> The learning algorithm is exactly the same whether I give it the goal of
> "say in the dark" or "stay in the light". When we build the reward
> hardware, we are indirectly giving it a goal.
>
> Solving the problem of building better reinforcement learning algorithms
> really has nothing to do with what reward signal we give it.
I understand.
It seems for me that building/programming a world for reinforcement
learning algorithms either needs a real robot (e.g. such a mouse) or it
takes much time to write such a simulation. (The software for the robots
also needs 'enough' time to develop.)
I hope that it is possible to show comprehension of an AI with not so
much programming time investment.
> Humans however have a very complex system for generating our reward signal.
> It has been tuned by the process of evolution to make us search for
> behaviors that also happen to be good at helping us survive. So
> indirectly, though the heavy hand of natural selection, we can say our goal
> is to survive. But the how that is implemented, is by having hardware with
> a goal of maximizing a reward signal, and a reward generating system, that
> has a large selection of stuff defined as "good" and "bad" for us (and a
> carefully turned relative measure of the different rewards.
>
> But it must not be "multiple rewards". It must always boil down to a
> single rewards system. If there are multiple rewards, the must still be
> something that combines them down to a single reward.
This is a problem because in real world there are quite often conflicts
of possible rewards.
casey
> I argue that we start with a signal processing system.
> In the case of using pulse signals, each pulse itself
> acts as very low level "symbol" on which all higher
> level understanding is built in the sort of systems I
> work with.
>
> If you use binary as your low level signal format, then
> the 1's and 0's become the low level symbols that you
> build all understanding from.
>
> The meaning of these signals is defined by what causes
> them - by the sensory systems that first generate them,
> and by the internal processing of those signals that
> generate new signals with associated new meaning as the
> result of the signal.
>
> For example, if I have a wall contact sensor that
> generates a pulse when it hits a wall, then the meaning
> of the pulses (and the symbol) coming from that sensory
> is "we just hit something".
Unlike the written symbols "hit wall" the pulse can not
only represent "hit wall" it can also cause other things
to happen as a result. Thus pulses representing "self pity"
can cause you to weep just as peeling an onion. A state
of mind has causal powers because it is made up of pulses
that not only represent things but can also carry out
actions dependent of what they represent.
> The meaning of the low level signals is the meaning on
> which all higher level understanding (and eventually
> natural language systems which is output by the AI) is
> all grounded back to the low level signals in the
> system and their meaning as defined by the hardware
> (and associated physical events) that are the cause of
> the signal.
Essentially I agree with what I think is the view you are
expressing. Where we seem to differ is in how it is all
arranged. You see it as only two levels, the pulse level
and the high level behavior "emerging" out of every pulse
interacting with every other pulse whereas I see it as
much more orderly with independent, specialized operations
taking place in parallel without mutual interference.
I also see each level as being partially independent of
the levels above and below. I gave the example of an army
where you have the real time behaviors of each soldier
who make their own detailed decisions although within the
context of control from a higher level. In other words
arranged as a heterarchy with each level being partly
independent and yet partly dependent of those above and
below and beside each other. As data flows up the system
it becomes more abstract and as data flows down the system
it becomes more specific.
JC
Curt Welch
> On Feb 5, 10:58=A0am, c...@kcwc.com (Curt Welch) wrote:
> > I argue that we start with a signal processing system.
> > In the case of using pulse signals, each pulse itself
> > acts as very low level "symbol" on which all higher
> > level understanding is built in the sort of systems I
> > work with.
> >
> > If you use binary as your low level signal format, then
> > the 1's and 0's become the low level symbols that you
> > build all understanding from.
> >
> > The meaning of these signals is defined by what causes
> > them - by the sensory systems that first generate them,
> > and by the internal processing of those signals that
> > generate new signals with associated new meaning as the
> > result of the signal.
> >
> > For example, if I have a wall contact sensor that
> > generates a pulse when it hits a wall, then the meaning
> > of the pulses (and the symbol) coming from that sensory
> > is "we just hit something".
>
> Unlike the written symbols "hit wall" the pulse can not
> only represent "hit wall" it can also cause other things
> to happen as a result.
Written words work just as well at causing things to happen. It's the
exact same thing in a different media. It's a signal that has a cause and
an effect.
> Thus pulses representing "self pity"
> can cause you to weep just as peeling an onion. A state
> of mind has causal powers because it is made up of pulses
> that not only represent things but can also carry out
> actions dependent of what they represent.
As a "state of a book" has causal powers that not only represent things but
can also carry out actions.
> > The meaning of the low level signals is the meaning on
> > which all higher level understanding (and eventually
> > natural language systems which is output by the AI) is
> > all grounded back to the low level signals in the
> > system and their meaning as defined by the hardware
> > (and associated physical events) that are the cause of
> > the signal.
>
> Essentially I agree with what I think is the view you are
> expressing. Where we seem to differ is in how it is all
> arranged. You see it as only two levels, the pulse level
> and the high level behavior "emerging" out of every pulse
> interacting with every other pulse whereas I see it as
> much more orderly with independent, specialized operations
> taking place in parallel without mutual interference.
I network of a billion independent signal generators is how I see it.
Seems to me to fit what you just wrote above.
> I also see each level as being partially independent of
> the levels above and below. I gave the example of an army
> where you have the real time behaviors of each soldier
> who make their own detailed decisions although within the
> context of control from a higher level. In other words
> arranged as a heterarchy with each level being partly
> independent and yet partly dependent of those above and
> below and beside each other. As data flows up the system
> it becomes more abstract and as data flows down the system
> it becomes more specific.
>
> JC
Well, I don't see it as levels. I see it a massively parallel society of
smalls agents that are forced to work together by how their causality is
interconnected and who's overall actions become coordinated indirectly by
the influence of the reward signal shaping their behaviors.
We can choose to view the massively parallel systems at many different
levels, but those levels exist in our interpretation of the hardware and
not so much in the hardware itself. Just as we can take one of my networks
and draw a line around any set of nodes in the system and talk about those
nodes as being a "module". It's our choice how we draw such lines - the
lines don't exist in the hardware.
casey
> casey <jgkjca...@yahoo.com.au> wrote:
>
>> Unlike the written symbols "hit wall" the pulse can not
>> only represent "hit wall" it can also cause other things
>> to happen as a result.
>
>
> Written words work just as well at causing things to happen.
> It's the exact same thing in a different media. It's a
> signal that has a cause and an effect.
A written word on a piece of paper can do nothing unless it
is read and converted to pulses that can do something.
>> > The meaning of the low level signals is the meaning on
>> > which all higher level understanding (and eventually
>> > natural language systems which is output by the AI) is
>> > all grounded back to the low level signals in the
>> > system and their meaning as defined by the hardware
>> > (and associated physical events) that are the cause of
>> > the signal.
>>
>> Essentially I agree with what I think is the view you are
>> expressing. Where we seem to differ is in how it is all
>> arranged. You see it as only two levels, the pulse level
>> and the high level behavior "emerging" out of every pulse
>> interacting with every other pulse whereas I see it as
>> much more orderly with independent, specialized operations
>> taking place in parallel without mutual interference.
> A network of a billion independent signal generators is
> how I see it. Seems to me to fit what you just wrote above.
No, the difference is, I see signal processing of _different
kinds_ taking place in parallel. In my visual processing
efforts the input data fans out to different processors
as it does in the neocortex the results of which "come
together" at some higher level according to any higher level
requirement. Just as the general does not know the details
of each soldier's actions the higher level functions do
not know the details of the lower level i/o modules even
though they control them.
> We can choose to view the massively parallel systems at
> many different levels, but those levels exist in our
> interpretation of the hardware and not so much in the
> hardware itself. Just as we can take one of my networks
> and draw a line around any set of nodes in the system
> and talk about those nodes as being a "module". It's
> our choice how we draw such lines - the lines don't
> exist in the hardware.
As I wrote in another post that is not what is meant by
"modules" which are clearly definable not by some arbitrary
border but by some particular function. And a module's
level is defined by how it is connected to other modules.
Thus DrawLine(x1,y1,x2,y2,color) is at a higher level than
SetPixel(x,y,color). The decision by a general to move the
the army to another area is at a higher level than the
decisions of the officers who might choose where to place
the men within that new area which is at a different level
than the soldier who selects his area to bunker down or
duck a bullet. We do not choose the levels of a module we
define them by their relationship to other modules.
JC
Curt Welch
> On Feb 6, 5:55=A0am, c...@kcwc.com (Curt Welch) wrote:
> > casey <jgkjca...@yahoo.com.au> wrote:
> >
> >> Unlike the written symbols "hit wall" the pulse can not
> >> only represent "hit wall" it can also cause other things
> >> to happen as a result.
> >
> >
> > Written words work just as well at causing things to happen.
> > It's the exact same thing in a different media. It's a
> > signal that has a cause and an effect.
>
> A written word on a piece of paper can do nothing unless it
> is read and converted to pulses that can do something.
A pulse can do nothing unless it is read and converted to another pulse by
another neuron.
I see nothing of what you wrote to be different than anything I've ever
suggested. What you wrote is so general and nondescript it applies to just
about everything, including rocks and radios.
> > We can choose to view the massively parallel systems at
> > many different levels, but those levels exist in our
> > interpretation of the hardware and not so much in the
> > hardware itself. Just as we can take one of my networks
> > and draw a line around any set of nodes in the system
> > and talk about those nodes as being a "module". It's
> > our choice how we draw such lines - the lines don't
> > exist in the hardware.
>
> As I wrote in another post that is not what is meant by
> "modules" which are clearly definable not by some arbitrary
> border but by some particular function.
If you have transistors that make up a 16 bit adder, you can draw a circle
around a collection of atoms in the IC and call that "module" a
"transistor".
You draw it around a few more atoms and you call it a NAND gate.
Draw it around a few more atoms, and call the module an XOR gate.
Draw it around a few more atoms and call the module a half adder.
Draw it around a few more atoms and call the module a full adder.
Draw it around a few more atoms, and call it a 4 bit full adder.
Draw it around a few more atoms, and call the module a 16 bit full adder.
Draw it around a lot more atoms, and call it a CPU.
Draw ir around a lot more and call the module a mother board.
Draw it around a lot more and call it a PC.
Where we draw the line and what word we make up to label the atoms we have
drawn the line around is not inherent in the atoms themselves. It's an
arbitrary division of the universe into "modules" that are useful for us to
talk about and think about.
When you draw lines around a connection of gates in my type of network, you
are defining modules with different functions, whether you happen to have a
name for the module you have defined by drawing a line is of no concern.
The fact remains, it performs a different function and is as much a
"module" as anything you talk about as being a module.
casey
> casey <jgkjca...@yahoo.com.au> wrote:
>> On Feb 6, 5:55=A0am, c...@kcwc.com (Curt Welch) wrote:
>>> casey <jgkjca...@yahoo.com.au> wrote:
>>>
>>>> Unlike the written symbols "hit wall" the pulse can not
>>>> only represent "hit wall" it can also cause other things
>>>> to happen as a result.
>>>
>>>
>>> Written words work just as well at causing things to happen.
>>> It's the exact same thing in a different media. It's a
>>> signal that has a cause and an effect.
>>
>>
>> A written word on a piece of paper can do nothing unless it
>> is read and converted to pulses that can do something.
>
>
> A pulse can do nothing unless it is read and converted to
> another pulse by another neuron.
I think you are just being argumentative instead of just
accepting the point being made. Clearly the context should
have told you I was talking about things represented by
pulses in a working computational system vs. things being
represented by static words in a book.
When people ask how their non material thoughts can cause
actions the explanation is that their thoughts are in fact
material pulses which do the causing and different thoughts
are just different patterns of pulses acting in different
parts of the brain to produce different outcomes.
> ...
> What you wrote is so general and nondescript it applies to
> just about everything, including rocks and radios.
Well I don't see rocks as having a heterarchy of modules
with the ability to represent their environment and learn
about it. Radios are modular and even have feedback
systems but they do not evolve or learn.
>> > We can choose to view the massively parallel systems at
>> > many different levels, but those levels exist in our
>> > interpretation of the hardware and not so much in the
>> > hardware itself. Just as we can take one of my networks
>> > and draw a line around any set of nodes in the system
>> > and talk about those nodes as being a "module". It's
>> > our choice how we draw such lines - the lines don't
>> > exist in the hardware.
>>
>>
>> As I wrote in another post that is not what is meant by
>> "modules" which are clearly definable not by some arbitrary
>> border but by some particular function.
>
>
> If you have transistors that make up a 16 bit adder, you
> can draw a circle around a collection of atoms in the IC
> and call that "module" a "transistor".
But that is not, as you suggested about your nodes, just ANY
set of atoms. We are able to group the atoms according to
some criteria.
> You draw it around a few more atoms and you call it a
> NAND gate.
But they are not just ANY group of atoms, they are the
group that perform the NAND function. We define the
collection by their function.
> Draw it around a few more atoms, and call the module an XOR gate.
>
> Draw it around a few more atoms and call the module a half adder.
>
> Draw it around a few more atoms and call the module a full adder.
>
> Draw it around a few more atoms, and call it a 4 bit full adder.
>
> Draw it around a few more atoms, and call the module a 16 bit full adder.
>
> Draw it around a lot more atoms, and call it a CPU.
>
> Draw it around a lot more and call the module a mother board.
>
> Draw it around a lot more and call it a PC.
And notice that each group of atoms makes up a functional module.
You can't draw a line around ANY half of the atoms making up a PC
and say it has any modular function.
> Where we draw the line and what word we make up to label the
> atoms we have drawn the line around is not inherent in the
> atoms themselves.
I never said it was inherent in the atoms, I said it was a
collection of atoms that perform a particular function.
The only reason you can give the above examples, which are
indeed real modules, is because they have a function.
You can't draw a line around ANY collection of atoms and
say you have a functional module.
A protein for example has a recognizable function and thus
can be seen as a module. A collection of random amino acids
have no function and are not defined as a module.
> It's an arbitrary division of the universe into "modules"
> that are useful for us to talk about and think about.
It is NOT arbitrary and the rest of your sentence confirms
that, it is one that is useful for us to talk about and
think about because it has a definable function.
An arbitrary division most of the time will not produce a
collection that can be called a functional module. Why do
you have such difficulty understanding what a module is?
> When you draw lines around a connection of gates in my
> type of network, you are defining modules with different
> functions, whether you happen to have a name for the
> module you have defined by drawing a line is of no concern.
> The fact remains, it performs a different function and is
> as much a "module" as anything you talk about as being a
> module.
If it does not provide you with a means to describe its
function then it is, by definition, not what is meant by
a module, to everyone except apparently yourself.
We define something as a module by defining its function.
That is what "module" means. You only have one module,
a monolithic network. Yes it may have distributed functions
but you cannot identify those functions or talk about
them as such because you cannot identify them thus they
are not modules. A modular network is one in which the
demarcation of the functions is recognizable.
You can write a working program without modules. You
cannot then arbitrary choose a set of statements in
the program and say they are "modules" just to pretend
your program is now made up of modules and thus is an
example of modular programming.
JC
Burkart Venzke
> On Feb 5, 4:24 am, Burkart Venzke <b...@gmx.de> wrote:
>> casey schrieb:
>>
>>> On Feb 4, 11:22 pm, Burkart Venzke <b...@gmx.de> wrote:
>>>> casey schrieb:
>>>>> People understand
>>>>> reality independently of any words used to describe it.
>>>> But don't words respectively notions help to understand reality better?
>>>> A word/notion is something like a kernel for a cloud of ideas.
>>> But we must _start_ with something other than language.
>> I think we should start with the importance/relevance of the combination
>> of sounds (signs etc.) with something happening in the world.
>
> Have you read much about language?
No, I haven't since my studies of computer science with AI about two
decades ago.
> A popular thinker is Steven Pinker
> who has written easy to read books such as "The Language Instinct" and
> "The Stuff of Thought" which provide interesting insights into
> language and how we use it.
Would are his main theses?
Language as an instinct? As something innate?
The question especially for an AI is what part of language comprehension
should be "innate" and what part should be learned.
Burkart Venzke
> Burkart Venzke <b...@gmx.de> wrote:
>> Curt Welch schrieb:
>>> Burkart Venzke <b...@gmx.de> wrote:
>>> Your post was much longer, but I think I've basically covered all the
>>> other points in other messages. But not this one:
>>>
>>>> Is it correct that you think of a rebuild of the brain?
I am not sure if an AI without understanding of the world would be able
to talk in a way that is seems highly intelligent; I would ask it
questions with need of (good) understanding.
Curt Welch
> Curt Welch schrieb:
> > I think with the right AI software driving it, such a machine would be
> > very interesting, and would seem highly intelligent to the people it
> > talked with.
>
> I am not sure if an AI without understanding of the world would be able
> to talk in a way that is seems highly intelligent; I would ask it
> questions with need of (good) understanding.
Well, if it had enough experience talking to other people it seems to me
that it could develop any level of understanding of the world you would
like to test for by talking to it. It might be very slow, and take a long
time, but in the end, I think maybe it could understand it. That is, you
have to have an internal model that matches the way the world is, but it
seems to me that enough talking with other people about the world would
give the AI the needed internal model.
Anything you could ask it, could be about a feature of the world the AI had
talked to many people about in the past, and it would develop a model of
the way the world is - even though the model would be all word based.
I think the trick is that it would take a great deal of talking to this AI
to educate it to the level that it could understand the world well enough
to make someone believe it really had experienced the world directly even
though it never had. If we just think about the complexity of the sensory
experience we receive when we pick up a complex object and look at it, such
as it's basic shape, and then also how it changes shapes as we turn it, and
how it changes color and how it feels in our hand, to the sounds it makes.
We gain a huge amount of experience by spending 1 minute with some object,
where as to describe all that with words might take hours.
Another way to think about this is the problem that the brain must develop
an understanding of the world without every really experiencing anything
directly. It's all second hand information the brain is working with.
That is, all the brain can experience directly, is pulse signals from the
sensors. The sensors experience the sensation directly and translate it to
pulse signals. So it's all second hand information the brain is forced to
work with. But yet, even with the brain's only access limited to pulse
signals, it can still develop a rich understanding of our universe.
Likewise, if the AI only has words coming in, it's in the same position as
the brain is. All the data it receives about the world is second hand.
It's just using the people it talks to as its "sensors" of the real world.
The big problem is that the bandwidth is so small compared to our real
sensors, that do develop a good understanding, it will take a lot longer
just to be exposed to all the data it needs. But in the end, it will have
to do what the brain does, which is develop models of the data it has to
work with. And if it can develop the correct models, it should be able to
talk as if it had direct experience of the real world, becuase in a sense,
it has had direct experience of the real world by using the people it talks
to as sensors.
It would be very interesting to see an AI like that develop over time and
see just how many silly "mistakes" it would make that shows how much it's
model was wrong but yet, over time, it would improve it's model with each
sill mistake it made until the point that it was very hard to catch it
making a mistake.
casey
> casey schrieb:
>
>> On Feb 5, 4:24 am, Burkart Venzke <b...@gmx.de> wrote:
>>> casey schrieb:
>
>>>> On Feb 4, 11:22 pm, Burkart Venzke <b...@gmx.de> wrote:
>>>>> casey schrieb:
>>>>>> People understand reality independently of any words
>>>>>> used to describe it.
>>>>> But don't words respectively notions help to understand
>>>>> reality better? A word/notion is something like a kernel
>>>>> for a cloud of ideas.
>>>> But we must _start_ with something other than language.
>>> I think we should start with the importance/relevance of
>>> the combination of sounds (signs etc.) with something
>>> happening in the world.
>> Have you read much about language?
> No, I haven't since my studies of computer science with AI
> about two decades ago.
It is unlikely the armchair ramblings of those who have very
little understanding of a subject will ever produce fruits.
>> A popular thinker is Steven Pinker
>> who has written easy to read books such as "The Language
>> Instinct" and "The Stuff of Thought" which provide
>> interesting insights into language and how we use it.
> Would are his main theses?
> Language as an instinct? As something innate?
>
> The question especially for an AI is what part of language
> comprehension should be "innate" and what part should be
> learned.
Do you want me to regurgitate Steven Pinker's books?
There is no simple answer I can give in a short post.
A short answer is it depends on innate abilities but
what they are is for us to find out. Steven Pinker
gives us insight and arguments as to what these
innate abilities might be based on experiments not
on armchair philosophizing from ignorance based
on a few subjective observations and some arcane
philosophy based on a couple of rat experiments.
JC
Curt Welch
> Curt Welch schrieb:
> I don't think about a wired brain (although you and other do), I think
> of a software (AI, robot) working with symbols. So, a set or better a
> hierarchy or network of new goals can be learned.
Software is hardware. It's just hardware that is easy to change.
You have mentioned symbols a lot but I've never seen you define what a
symbol is. How do you build hardware that works with symbols? What in
hardware terms, is a symbol to you? Or, if I built some hardware, who
could you tell if it was "working with symbols"? How could I tell if my
hardware met your definition of "working with symbols"?
> > And yet another are the goals implicit in the way our reward system is
> > wired. We have a reward system that makes damage to the body (like
> > heat high enough to burn) a signal that directs the learning system to
> > change the behavior to prevent a repeat of the same condition. So the
> > learning system has motivations built into it, that causes it to build
> > behavior generation circuits, that show goal behavior in our action -
> > the goal of not being burnt again. So by seeing how our behavior
> > change - seeing how we learn - we can see what the goals of the
> > learning system is. And that's yet another way that goals exist in the
> > implement ion of our brain.
>
> I think that a lot (most?) goals are only temporarily active.
> When I sit somewhere the goal of walking a straight line is unimportant.
> When I have just has been in a store (and have bought all I need) there
> is no active goal "I want to go to the store".
Most of what you talk about seems to be a reference to the things we are
consciously aware of. That's all fine becuase in the end, any AI we build
will have to have a similar sort of conscious awareness.
But talking about our high level conscious awareness normally tells us
almost nothing about what type of hardware we have to build to implement
the subconscious systems that our conscious awareness emerges from. Many
people in AI work have tried to directly code the processes they believe
exist in their own conscious awareness. None of that work has really
gotten very far because the more they code the more they realize that
what's missing, are the subconscious systems that make it all work - and
for the most part they seem to have no idea how to code the subconscious -
after all - it's something we can't even sense that's in us.
We use lots of language to talk about what's happening in our head - such
as the "understanding" word. A lot of that langauge makes us believe we
understand what's happening there. But in fact, most the language doesn't
explain anything. It only acts a a label of what's happening, and not the
cause of why it's happening. When people in AI set out to try and code all
those descriptions, they soon figure out that they don't know _why_ it's
happening.
For example, we can program a robot to walk, or run, or talk. But what is
the subconscious process at work that determines _when_ the robot walks,
when it stops walking, and when it talks? Just because we have a word for
"run" and can make the robot produce the "run" behavior, that doesn't tell
us how it decides when to run, and when not to run.
You have to figure out how to make a robot make decisions. How does it
decide what to do next? I believe the answer to this is to build a
decision machine that learns how to make decisions by reinforcement
learning. I believe that's the low level hardware we have to build, and all
the stuff you seem to talk about, such a learning how to act from a
teacher, is not something we should build into the machine, but simply a
learned behavior that emerges from the low level decision hardware we have
to build.
In other words, what I think we have to build, is something we have no
direct conscious awareness of. It's the invisible low level process that
produces all these high level actions we are aware of. It's what has made
AI so hard. What we have to build, is not what we see.
On the question of making decisions, if we hard code a robot to run, we
have to hard-code a lot of real time decisions into the machine to make it
move all it's arms and legs at just the right time to stay balanced, and to
keep moving forward in a running gait. We then have to build a high level
process in to determine _when_ to start running, and when to stop running.
It has to learn how to make good decisions. But if you can build hardware
which learns when to run, and whyn not to run, along with, when to reach
for an object, and when not to reach for the object, you can also make it
learn to run, instead of having to hard code all that complex decision into
the robot. Because learning to run is no easier, or harder, than all the
other higher level decisions. The only difference is whether the systems
is learning to make 1000 decisions per second, or 1 per second.
> >>> But when we say these things, like "change for the better" we have
> >>> just pushed the question of what is learning, off to a problem of
> >>> defining what we mean by better. Reinforcement learning answers that
> >>> questions.
> >> Does a teacher decides or who (what)?
> >
> > Evolution built into us a reward system (reward signal) that directs
> > how learning works. It's hard wired into us to make some things like
> > damage to the body, or an empty stomach, create a negative reward
> > signal, and other things, like the act of eating when hungry, or sex,
> > generate a postie reward signal.
>
> Yes, we humans have a complex reward system. We may give an AI a similar
> complex one, too, but it is not necessary and perhaps risky. I don't
> want an AI to steal food (ok, electricity) only because of its need.
Well, that's the problem with intelligence. You can't control it directly.
We control it only indirectly though the goals we give it. That's what
makes it intelligent - it finds optimal behaviors on it's own.
If you want total control, you don't create intelligence, you do what we
always have done, which is to hard code all the behavior. If you create
intelligence, you lose some of that control - but you gain the ability of
the system to find behaviors you never thought of or which are so complex
we couldn't have coded them directly even if we wanted to.
If you give a goal of keeping it's batteries charged, it can at times do
the same things our pets do - tear things apart to get what it needs. I've
thought it would be fun for a robot that was motivated to keep itself
charged, to have electric sensors to detect the electromagnetic and
electrostatic charge of electric wires. It could sense where the
electricity was even hidden in a wall, and could learn to tear up a wall
and cut though the wires to get to it! (not good for the house, but fun to
watch it learn to do it!).
> > The learning hardware than adjusts our behavior in response to rewards
> > (and more important, in response to estimations of future rewards).
> >
> > The teacher is just part of the environment we learn to deal with so as
> > to get more rewards. If we don't respect the teacher, and produce the
> > behavior they want us to produce, we won't get as many rewards.
>
> Yes, with your idea of environment and reward system.
> For me, the teacher is the source of rewards, but he can 'say' "breaking
> something is bad" so that this rule is substitution for an hard wired
> reward.
Yes, but I think if you try to hard-code human level AI at that level, you
will fail completely. Human behavior at that level is far too complex to
hard-code.
> >>> Other answers people have used to the question is to describe
> >>> learning behavior as goal seeking. That is, our behavior changes to
> >>> better reach some goal. But then we are left asking the question of
> >>> what a goal is, or what the goal should be for our AI solution.
> >>> Reinforcement learning abstracts the definition of what the goal is
> >>> to reward maximizing.
> >> I like the idea of goals, "reward maximizing" is too abstract for me.
> >
> > Yes, it takes some time to understand. It's not easy to grasp at
> > first.
>
> I am not sure if we think of the same problem.
> The problem is a "reward maximizing" function:
> Where does it come from? Is it fix? When yes: Why is so as it it?
What does what come from? The reward signal which the learning system
tries to maximize, or where does the learning system which changes behavior
so as to try and maximize future rewards come from? They both come from
our genetics.
> >> But perhaps we only fight for words and mean nearly the same.
> >> Goals are something we can implement, we can imagine something -
> >> better than an abstract function. Or is the function for reward
> >> maximizing something else?
> >
> > I can build a little robot that runs away from bright light. By
> > looking at its behavior, we can say its goal is "to get away from the
> > light". But all I have done is built a very simple sensory system with
> > some simple logic for driving the wheels from the light sensors. No
> > where in the schematic do we see something that looks like "the goal of
> > running away from the light". Only by studying the schematic so as to
> > understand what type of behaviors this system will produce, can we then
> > label the machine as "waning to get away from the light".
>
> A goal should not be only an interpretation of a system's behavior but
> something an AI really wants to achieve.
"Wants"? You have to explain what want are before you can say things like
that.
Well, if you try to simulate a real 3D world full of objects and gravity
sure, that's a lot of work. But you can also make very simple simulations
that create very complex learning problems for a generic learning algorithm
that has no built in assumptions about the nature of the environment it's
interacting with. Simple problems like balancing a pole in 2D space or
balancing a pole on top of a pole is a fairly simple environment to
simulate but one which can be very hard to learn to deal with depending on
what type of sensors the agent is given. Such problems are simple enough
to study and simulate, but hard enough to show what's missing in the
learning algorithm.
> I hope that it is possible to show comprehension of an AI with not so
> much programming time investment.
>
> > Humans however have a very complex system for generating our reward
> > signal. It has been tuned by the process of evolution to make us search
> > for behaviors that also happen to be good at helping us survive. So
> > indirectly, though the heavy hand of natural selection, we can say our
> > goal is to survive. But the how that is implemented, is by having
> > hardware with a goal of maximizing a reward signal, and a reward
> > generating system, that has a large selection of stuff defined as
> > "good" and "bad" for us (and a carefully turned relative measure of
> > the different rewards.
> >
> > But it must not be "multiple rewards". It must always boil down to a
> > single rewards system. If there are multiple rewards, the must still
> > be something that combines them down to a single reward.
>
> This is a problem because in real world there are quite often conflicts
> of possible rewards.
You are just not understanding the point I was trying to make. I'm saying
that internally, all rewards and all estimations of future rewards must be
measured in a single unit of measure - a single "currency" of "value" if
you will. You can't have two different units of rewards or else the
learning agent is faced with an unsolvable problem of comparing apples to
oranges. If for example, it got 2 green rewards for pickup up a block, and
5 orange rewards for pickup up a ball, and it' sable to do either at the
moment, what should it do? Pick up the ball, or pick up the block? If
there are two units of measure for rewards, it becomes impossible for the
AI to decide which is "better". Is 2 green better than 5 orange? Or is 5
orange better than 2 green? This is not something it can decide on it's
own. It's something that has be defined for it. You have to translate
both units to a common measure of "worth" so that the AI cam make a
decision. If we define 1 green as being equal to 3 orange, then the AI can
determine which is better. It knows the 2 green is better than 5 orange
because 2 green is equal to 5 orange, so it can judge which of the two
actions are better.
All actions must be evaluated for their worth in terms of a common measure
of reward or else the AI will have no way to make decisions between
conflicting choices of behavior (of which there will always be millions of
options to pick from in real-world interaction). There can only be one
global measure of reward and only one global external reward value driving
the learning system.
That's what I meant by "a single reward _system_". There a of course
billions of conflicting choices as to what reward to "seek". Every micro
action we make is our brain's decision about what reward to seek at that
moment in time becuase every action has been evaluated by the system to
determine what its estimated reward is. So every decision made to lift a
finger, or move a toe, is all tied back to the systems estimation of how
likely that little action is to produce higher expected future rewards than
any other action the brain could have selected at that moment in time.
It's millions of decisions between conflicting actions all guided by a
single measure of worth internally in the system.
casey
> ...
> Software is hardware.
Software is not a subcategory of hardware.
We can talk about there being a problem with the software
without there being a problem with the hardware because they
are not the same thing.
Software is an _arrangement_ of physical matter when embodied
in hardware and until then it is a _description_ as to how that
physical matter can be arranged.
Hardware is not an arrangement it simply has an arrangement.
Hardware is not a description of how hardware is to be arranged.
Hardware is the stuff that gets arranged and software is a
description of how to arrange it.
Your inability to abstract the description from the thing being
described amazes me. Rocks are rocks no matter how they are
arranged but a circle pattern of rocks is not a square pattern
of rocks.
It is as silly as confusing a house plan with a house because
they are both physical. Duplicating a "house plan" is not the
same as duplicating the "house" it is a plan of. In the computer
world hardware refers to the "house" and software refers to
the "plan of the house".
Excel and Word exist as patterns of matter and as such, yes,
they exist as physical matter but the words themselves "Excel"
and "Word" do not refer to the matter they are made of they
refer to the arrangement of that matter.
> It [software] is just hardware that is easy to change.
No, the hardware, the rocks, remain the same, no matter how
easy or hard it is to rearrange them. The pattern, however
can be fixed or changeable. Software that is burnt into the
hardware is not easy to change, it is impossible to change
in any practical sense.
JC
Curt Welch
> On Feb 10, 11:53=A0am, c...@kcwc.com (Curt Welch) wrote:
> > ...
> > Software is hardware.
>
> Software is not a subcategory of hardware.
That's becuase it's not a subcategory. It's just hardware. Like the CPU
is not a subcategory of hardware. It's just hardware.
> We can talk about there being a problem with the software
> without there being a problem with the hardware because they
> are not the same thing.
Only by social convention of word usage - not by reality.
I could just as easy define the word software as a label for the CPU and
claim that the CPU was not hardware like you are claiming software is not
hardware. I can then talk just like you are doing about my CPU when it is
broken by saying "there's a problem with my software". When asked, do you
man the hardware, I respond "no, the software - the CPU - not the hardware
you stupid idiot!".
Software is just one of the many hardware modules of a computer.
> Software is an _arrangement_ of physical matter when embodied
> in hardware and until then it is a _description_ as to how that
> physical matter can be arranged.
Hardware is an _arrangement_ of physical matter when embodied
in hardware and until then it is a _description_ as to how that
physical matter can be arranged.
Your words are just as accurate when describing hardware.
When I talk about "a wheel" that is a description of an arrangement of
phsyical mater. When I make a real wheel, I rearrange physical matter to
create it.
When I talk about software, or write software with a pen on paper, that's a
descritpi9on of how I'm going to build some hardware. When I enter it in
the computer, I have built physical hardware according to my description.
It's exactly the same for all hardware and all software. It's only social
convention that some types of hardware ended being called software.
Software is just a type of hardware like a CPU is a type of hardware.
> Hardware is not an arrangement it simply has an arrangement.
Don't be an idiot John. Of course it's "an arrangement". How on earth do
you think we make a CPU? We take physical matter and change it's
arrangement.
> Hardware is not a description of how hardware is to be arranged.
Well, yes, there's some truth there, but only a half truth. If we draw a
picture of a wheel on a white board that picture of course is physical, but
it's a symbolic representation of the hardware we want to build. But we do
sometimes draw these pictures and talk about them as if they were the
hardware. So the distinction between the picture, and the real hardware
gets blured so it's hard at times to know which we are talking about.
The same thing happens with software. When I write code on a white board
is that the software or not? It's only a description of how I want the
hardware arranged but yet we talk about the description as being "the
software" sometimes just like we do with hardware. The actual hardware and
the language representations of it are often blurred.
> Hardware is the stuff that gets arranged and software is a
> description of how to arrange it.
Yes, you like it when you make up this nonsense and pretend everything is
just so simple don't you?
Software exists in my brain before I write it. It exists in paper when I
write it down with a pen. It exists as a physical data file when I enter
it into the computer by reading what I wrote on the paper before it's been
complied. It exists as other physical data files after it's been complied.
It exists as arrangements of electrons in the memory when it's loaded into
memory to be executed. It exists as voltage levels in the logic gates of
the CPU as it's making the CPU do things like add numbers.
At every step it exists in a physical different form. Each physical form
causes the next form to be be created in a straight forward physical chain
reaction. Every form was just hardware of one type or another.
"Description" is a mental term we use in our language - the langauge that
is based on the concept that dualism is a fact of reality - which is an
obvious error. Obvious to some of us at least.
The terms hardware and software came into common usage in our computer
technology becuase it was a nice parallel to the error of dualism that
already existed in our language. It fit nicely into that erroneous
foundational language concept.
And now, you are using the error of the langauge, to argue that the reality
of the universe is actually dualistic and that it's "obvious" that software
is not hardware to you.
> Your inability to abstract the description from the thing being
> described amazes me. Rocks are rocks no matter how they are
> arranged but a circle pattern of rocks is not a square pattern
> of rocks.
I have no problems of abstraction. I have the power to see the things how
they really are, unlike you, who are so befuddled by langauge that you can
only see the world based on how you were taught to talk about it.
> It is as silly as confusing a house plan with a house because
> they are both physical. Duplicating a "house plan" is not the
> same as duplicating the "house" it is a plan of. In the computer
> world hardware refers to the "house" and software refers to
> the "plan of the house".
Nope, it doesn't. It referees to both. It refers to the descriptions we
create of how we want the hardware arranged as well as the hardware when it
gets arranged like that.
And often, in fact it doesn't refer to the description,. When I think of
how I'm going to write the software in by head, the description has been
first created in by brain. But yet, people don't refer to my brain as "the
software". When I write software in a book that is clearly a description
of how I want the hardware arranged. But we don't say "run that software
please" in reference to what I wrote on paper. Not until the hardware has
been reconfigured so that it can run the software, do we really start to
talk about it as software. Not until it exists as part of the hardware, do
we call it software.
> Excel and Word exist as patterns of matter and as such, yes,
> they exist as physical matter but the words themselves "Excel"
> and "Word" do not refer to the matter they are made of they
> refer to the arrangement of that matter.
When we make reference to a pattern, we are in fact making reference to
physical hardware. When I talk about a circle, I'm in fact making
reference to the hardware in my brain that is able to recognize the circle
pattern. that's how all patte4rns we can talk about are defined for us -
by the hardware in our brain that is able to recognize it.
In our language whoever, we pretend mental events are not physical, and
that things like "patterns" like circles are not physical. But again,
that's the error of dualism built into our langauge talking and not the
truth about reality. Patterns are very much real in both how they exist,
like the circle of rocks, and in the hardware in our brain (and in our
machines) that can detect the patterns).
Excel is used in some context to talk about our pattern recognition
hardware and i9n others, it's used to talk about the hardware. I can hold
a CD and say "this is the most recent version of Excel". When we talk like
that, we are talking about the physical hardware ()the CD) and talking
about the fact that it's on the "Excel" pattern.
But the word is making reference to either the physical CD or the physical
hardware in our brain that is bale to recognize the Excel pattern - or some
unspecified implied combination of both. Such as by implying if you were
to examine the patterns on the CD your "excel" pattern recognizing hardware
would indicate a match. So I'm making reference to both in that case.
Likewise, when I use the word CPU, it's both a reference to some pattern
hardware that can recognize a CPU as well as potentially some specific CPU
hardware depending on context.
Everything we make reference to when we talk about software is a direct
parallel to how we talk about hardware. There is no difference because
software is hardware. Why there seems to be such a difference is becuase
we use this stupid language of ours which makes everything seem as if it's
dualistic when it's not really dualistic at all.
> > It [software] is just hardware that is easy to change.
>
> No, the hardware, the rocks, remain the same, no matter how
> easy or hard it is to rearrange them. The pattern, however
> can be fixed or changeable. Software that is burnt into the
> hardware is not easy to change, it is impossible to change
> in any practical sense.
True, software is not always easy to change. "easy to change" is not how
we identify which part of the hardware to talk about as software. Tires
are easy to change, but we don't talk about them as software.
However, there is no definition of software that is clear enough to allow
us to look at any hardware and know for sure what part of it is the
software, and what part is the hardware. It's mostly just social convention
that defines it.
In general, when one part of the hardware has the power to control the
behavior of another, and the one part is meant to be easy to reconfigure
over time, then we call the part that is easy to reconfigure software. But
even though that's roughly what happens in computers, when we do the same
thing in other hardware, it is seldom called software.
Back in the days of punched cards, for those of you old enough to remember
those days, we used card punch machines to manually type software into the
cards. These machines were a bit like typewriters, but they were also
programmable. You could set tab stops on them and some other features.
The "software" however was itself a punched card that you would type ahead
of time to define the tab stops and then you loaded this card onto a drum
that allowed it to control the card punch. The "software" was all
mechanical. And as far as I remember, no one every called it "software".
But that card was just as much "software" as any of our computer hardware
is today. It was software that regulated the behavior of the card punch and
could be easily reconfigured on the fly. But why was it not called
software?
Likewise, when we have switches on electronics that control how the system
behaves, they too are "software". That is, they are part of the physical
machine we can "reprogram" on the fly. But yet, no one talks about the
switches as part of the software even though they are. Why is that?
It's becuase it doesn't fit so well with the dualistic errors of our
language. Mental events are things that "exist" but yet we can't sense
directly with our physical senses. Software is the part of the hardware
that seems to be "physical" but yet can't be sensed so easily. When he
software is encoded in a punch card, or a switch, it doesn't seem very
"non-physical", so we call it hardware. But when it exists as invisible to
our normal senses pits in a CD, or magnetic patterns on a disk, or
elections in memory cell, then it fits better with the illusi9on of
dualism. It's there, but yet it doesn't seem to have a physical form, so it
feels right for us to make a parallel between that type of hardware, and
our dualistic mental terms. We use the term "software" as if it were the
"mind" of the computer because it's a close parallel to what we are able to
sense about our own brains and how we like to use terms like "mind" to
refer to the parts of the hardware in our brain.
But when the programmable hardware features of the machine become obvious
physical, like with switches, or punched cards acting as the software, then
it stops feeling "right" to talk about them as software, we just just go
back to calling it hardware.
It's only social convention which defines what part of the hardware we
label as being the software, and that social convention is highly biased by
our social conventions for how we talk about our own brain because of the
illusion of dualism that most people are fooled about.
And what always drives me crazy, is when people try to explain how to solve
AI by talking about the "software" being like the "mind" as if that was
some basic answer to how computers will solve AI when in fact, the only
reason the social definition of the word "software" matches the concept of
"mind" in any way is because it was derived from our confusion about how
the brain worked. The concept of software doesn't help us understand the
brain. Our confusion about the brain is what spilled into our confusion
about our how we talk about our computers.
As much as you want to throw social word usage convention back at me as if
it were the truth of how reality was structured, it's not. It's just
arbitrary social convention of word usage.
You do this all the time. You are so focused on how words are defined
realities to other words by social conventions, you seem to forget to think
about the physical world they are trying to describe. When I try to talk
about truths of the physical world, when I say something like "software is
hardware", you come back and argue social word usage conventions at me and
use other social world usage conventions like "description" as the
foundation of your argument. Instead of using facts about the physical
world to argue you point, you use social word usage conventions to argue
your points. My whole point (the one you jump on and posted the whole
article about trying to prove me wrong) was that social convention for word
usage with "software" and "hardware" was misleading and that we should look
past it to see the truth. And instead of you being able to look past
social word usage conventions to see the truth, you only seem able to to
argue social word usage conventions as if they were the only truth.
casey
> casey <jgkjca...@yahoo.com.au> wrote:
>> On Feb 10, 11:53=A0am, c...@kcwc.com (Curt Welch) wrote:
>> > ...
>> > Software is hardware.
>>
>> Software is not a subcategory of hardware.
>
You are the one who wrote that "software" is "hardware that is
easy to change". That makes it a subcategory of hardware, the
category of hardware that is easy to change.
>> Hardware is the stuff that gets arranged and software is a
>> description of how to arrange it.
>
>
> Yes, you like it when you make up this nonsense and pretend
> everything is just so simple don't you?
Nothing to do with pretending it is just so simple. It has to
do with the ability to use abstractions in order to communicate.
> The terms hardware and software came into common usage
> in our computer technology because it was a nice parallel
> to the error of dualism that already existed in our language.
And of course I knew that was the bug in your brain over the
use of software vs. hardware. If I was to say I have a problem
with my software who would you send in to fix it? The hardware
engineer or the software engineer? You seem to think they are
the same thing so it shouldn't matter. If my computer screen
goes blank I ask the question is that a software error or is
that a hardware error? According to the above you would say I
am talking nonsense because software is hardware there is no
difference.
>> Your inability to abstract the description from the thing
>> being described amazes me. Rocks are rocks no matter how
>> they are arranged but a circle pattern of rocks is not a
>> square pattern of rocks.
>
>
> I have no problems of abstraction. I have the power to
> see the things how they really are, unlike you, who are so
> befuddled by language that you can only see the world
> based on how you were taught to talk about it.
No Curt you are lousy at abstractions. You can't tell the
difference between hand waving and an abstraction. You
can't tell the difference between imagining something and
a practical working abstraction. They are all the same to
you like running and the runner.
> In our language whoever, we pretend mental events are not
> physical, and that things like "patterns" like circles
> are not physical.
You know very little about our language and how it works
and how it is used. Your notions are based on your subjective
observations of the use of language and some behaviorist
notions of language that seemed to fit your subjective views.
> ... there is no definition of software that is clear enough
> to allow us to look at any hardware and know for sure what
> part of it is the software, and what part is the hardware.
The hardware is the physical stuff it is made out of and the
software is a description of how it is arranged to cause it
to behave in a certain way.
> ... we call the part that is easy to reconfigure software.
The RAM memory chips are hardware and they are easy to change
and we don't call memory chips software. What we call software
is the description of how we are going to change the hardware.
The software itself may be unchangeable such as a read only dvd.
> Likewise, when we have switches on electronics that control
> how the system behaves, they too are "software". That is,
> they are part of the physical machine we can "reprogram" on
> the fly. But yet, no one talks about the switches as part
> of the software even though they are. Why is that?
Because the switches are the hardware and the software is the
state of the switches. Nothing to do with the dualistic crap
you go on about.
> We use the term "software" as if it were the "mind" of the
> computer because it's a close parallel to what we are able
> to sense about our own brains and how we like to use terms
> like "mind" to refer to the parts of the hardware in our
> brain.
The word "mind" refers to the things that make up our
experiences. We didn't know where they existed until they
were located in the organ we call the brain. We didn't know
what form they took until we found the experiences correlated
with brain activity. Because we couldn't relate firing neurons
to experiences it was hard to believe they were the same thing.
But now we know they are just different views of the same
thing as I have explained in a later thread.
> And what always drives me crazy,
Oh dear we don't want that to happen!
> ... is when people try to explain how to solve AI by talking
> about the "software" being like the "mind" as if that was
> some basic answer to how computers will solve AI
I have never heard anyone say viewing "software" as being
like the "mind" is going to _solve_ AI??
The usual view is that what people have been calling "the mind"
is in fact what the brain does rather than what some imagined
ghost in the machine does. To the extent that software is a
description of what the PC hardware is doing (Excel or Word)
is to the extent you might say that mind is a description of
what the brain hardware is doing (adding numbers, recognizing
something and so on).
> My whole point (the one you jump on and posted the whole
> article about trying to prove me wrong) was that social
> convention for word usage with "software" and "hardware"
> was misleading and that we should look past it to see
> the truth.
It is your misuse of language I jump upon. Your belief that
calling a verb a noun is "the truth" because you need a
physical object, noun, to do something, verb.
We use words in many ways and one of those ways is to make
useful discriminations. It has nothing to do with your
imagined behaviorist inspired belief that people are confused
by "mind" terms. Technical people are not confused when they
use such "mind" terms when talking about computers. On the
contrary it is you that seem to get confused and take the
words literally so you actually seem to believe that a simple
computer program is "understanding" something by its behavior
alone. Do you really believe the program, Dr Eliza, actually
understands the conversations it has?
JC