Re: [nupic-dev] Mentioned presentation on action with CLA?

[SeH]

i imagined that something like OpenBECCA http://openbecca.org could be integrated with NuPIC for a motor control system

https://github.com/brohrer/becca

https://github.com/brohrer/becca/blob/master/doc/becca_0.4.5_users_guide.pdf

from the opposite direction: part of BECCA's perceptual components may be enhanced (or replaced) with NuPIC

https://github.com/brohrer/becca/blob/master/core/agent.py

On Mon, Aug 19, 2013 at 8:25 PM, Thompson, Jeff <jef...@remap.ucla.edu> wrote:

Thank you for the quick reply.  I'm in Los Angeles, so I hope someone does record your presentation at NASA.

A similar question arose when I read "Thinking, predicting, and doing are all part of the same unfolding of sequences moving down the cortical hierarchy." (On Intelligence, p. 158.) I'm sure this is a FAQ, but do you have some writings or presentations about how a CLA would receive feedback signals coming down the hierarchy?   

Thank you,

- Jeff

From: Jeff Hawkins <jhaw...@numenta.org>
Reply-To: "NuPIC general mailing list." <nu...@lists.numenta.org>
Date: Sunday, August 18, 2013 11:35 AM
To: "'NuPIC general mailing list.'" <nu...@lists.numenta.org>
Subject: Re: [nupic-dev] Mentioned presentation on action with CLA?

Jeff,

I wrote this presentation a couple years ago for a workshop on sensory motor integration.  That workshop was held at the Santa Fe institute and I don’t believe it was recorded.  The genesis of the workshop was a paper written by Murray Sherman and Raymond Guillery where they point out that every region of the neocortex (as far as they have looked) has cells in layer 5 that have a motor function.  The big idea is that every region of neocortex does sensory inference and generates behavior.  There are no pure “sensory” regions and no pure “motor” regions.  It is one of those beautiful results that make you slap your head and say “of course!”

 

I have always envisioned the CLA as modeling a section layer 3 in a region of the neocortex.  Layer 3 is the primary input layer and is therefore doing inference on the input to that cortical region.  Layer 5 is driven by layer 3 and has the cells that innervate muscles, or more often project to some sub-cortical area that generates behavior.  I see how two CLAs, one for layer 3 and the other for layer 5 can work together to learn a sensory motor model of the world where today’s CLA is purely sensory.  There is a lot I don’t understand but there is enough that I think we can make progress. 

 

I gave this presentation again earlier this year at Numenta.  It wasn’t recorded.  It looks like I might give it again this fall at NASA Ames here in Silicon Valley as there are a few roboticists there interested in it.

 

I don’t mind recording it if someone could take care of the logistics.

Jeff

 

From: nupic [mailto:nupic-...@lists.numenta.org] On Behalf Of Thompson, Jeff
Sent: Saturday, August 17, 2013 4:57 PM
To: NuPIC general mailing list.
Subject: [nupic-dev] Mentioned presentation on action with CLA?

 

Hello. 

 

In the introduction for the NuPIC Hackathon Kickoff, Jeff Hawkins talks briefly about the need for CLA integration with action.  In response to a question, he says "We haven't done experiments with motor interaction.  I have a presentation, I think about it."  Is the presentation about motor interaction with CLA available?

http://www.youtube.com/watch?feature=player_detailpage&v=yShNQvJEP6A&t=2188

 

Thank you,

- Jeff Thompson 

_______________________________________________
nupic mailing list
nu...@lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org

[Brandon Rohrer]

SeH,
Thanks for the plug. I met with Jeff Hawkins about a year ago and he recognized that NiPIC was lacking an action selection mechanism. If integration was feasible that would be great. I think that both Jeff and I are hoping that BECCA and NuPIC will solve the combined perception/control problem but if a marriage of the two works better I'm all for it.
Brandon

--
You received this message because you are subscribed to the Google Groups "BECCA_users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to becca_users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[SeH]

I've mentioned RL-Glue before:  RL-Glue provides a standard interface that allows you to connect agents, environments, and experiment programs together, even if they are written in different languages. 

http://glue.rl-community.org/wiki/About_RL-Glue

https://code.google.com/p/rl-glue-ext/wiki/Python

http://glue.rl-community.org/wiki/Task_Spec

It could be a good framework for comparing the relative performance of the different cognitive architectures & implementations, in terms of: task effectiveness, CPU, and memory usage.

I have no idea of the relative compute cost of the various components, for different problem sizes.  BECCA would at least need ported to C/C++ to compete with NuPIC fairly, unless the native functions implemented by numpy are responsible for the vast majority of the calculation.

https://github.com/numenta/nupic/tree/master/nta/regions

https://github.com/numenta/nupic/tree/master/nta/algorithms

"Grouper is the temporal pooler: it discovers temporal dependencies between spatial coincidences."

it would be nice to use a consistent set of terminology between the projects.  grouper, pooler, etc..  maybe BECCA perception constitutes a new kind of NuPIC "grouper" or "pooler"...  not sure yet.  i should know more after becoming more familiar with NuPIC and the latest BECCA design.

[SeH]

BECCA "feature extractor" ?= NuPIC "classifier"

[Brandon Rohrer]

Approximately. I believe the NuPIC classifier performs feature extraction as part of its function. The feature extractors from the two approaches are trying to solve the same general problem, but with different assumptions about the data they are working with.

[SeH]

---------- Forwarded message ----------

From: Thompson, Jeff <jef...@remap.ucla.edu>
Date: Tue, Aug 20, 2013 at 11:34 AM
Subject: Re: [nupic-dev] Mentioned presentation on action with CLA?
To: "NuPIC general mailing list." <nu...@lists.numenta.org>

Hello SeH,

 

While I appreciate your pointing out this open source project of which I was not aware, it seems to go against my question. I started paying attention to work on the CLA (again after many years) when I heard Jeff Hawkins speaking as he does below that "There are no pure “sensory” regions and no pure “motor” regions".  It gave me hope that this work might avoid the pitfall of the classic "input-processing-output" loop of classic AI, which BECCA clearly seems to follow (see the attached diagram).

 

We now know that there are just as many feedback connections going to back down to the "input" regions, and that action is not so different from perception (in that it uses similar machinery of prediction), and that "input" is intimately tied to the actions active during the input (instead of having "action" on the other side of world from "input", as in the BECCA diagram). 

 

I'm hopeful to see a diagram soon of many CLA modules for action and perception connected in a hierarchy which shows how action comes from similar prediction machinery as perception and how to avoid the pitfall of "input on one end, output on the other end."

 

Thank you,

- Jeff T

---

From: nupic [nupic-...@lists.numenta.org] on behalf of SeH [seh...@gmail.com]
Sent: Monday, August 19, 2013 6:26 PM
To: NuPIC general mailing list.; becca...@googlegroups.com

Subject: Re: [nupic-dev] Mentioned presentation on action with CLA?

[Brandon Rohrer]

Thanks for this SeH. Jeff's criticisms are fair. The next version of BECCA will have a different structure that he may find more biologically plausible. But that will probably be a few months in coming.
Brandon

--

[SeH]

in general, we can experiment with both biologically-pure designs as well as hybrid non-biological ones.  to be biased towards only the emulating of nature is incomplete.  

"In my viewpoint, there is no meaning to the word "artificial."  Man can

only do what nature permits him to do.  Man does not invent anything.

He makes discoveries of principles operative in nature and often finds

ways of generalizing those principles and reapplying them in surprise

directions.  That is called invention.  But he does not do anything

artificial.  Nature has to permit it, and if nature permits it, it

is natural.  There is naught which is unnatural."

--- Buckminster Fuller, Education Automation

[Brandon Rohrer]

I have to agree. I like your Fuller  quote. Biological fidelity and high performance are two separate goals. They aren't often antagonistic but they can be orthogonal, that is, making progress toward one doesn't necessarily get you closer to the other.

[SeH]

---------- Forwarded message ----------

From: Michael Ferrier <Michael...@brown.edu>
Date: Tue, Aug 20, 2013 at 2:54 PM
Subject: Re: [nupic-dev] Mentioned presentation on action with CLA?
To: "NuPIC general mailing list." <nu...@lists.numenta.org>

The impression that I get from the neuroscience literature is that there are two basic types of learning in the brain. The first type could be called "model learning", it is what the cortex specializes in, and it's about learning hierarchical spatio-temporal models of input, from both external sensors and from other brain areas, representing the outside world, the body, and other internal states, and how they change over time. The second type is reinforcement learning, which uses built-in "reward" and "punishment" signals (such as pain or the taste of sugar) to learn what cortical patterns should be activated within a particular context of the activity of other cortical patterns, so as to maximize reward and minimize punishment. In the brain, reinforcement learning takes place in the basal ganglia, but uses input from many different areas of the cortex, and affects the activation of patterns within prefrontal and motor cortex to result in the control of attention, working memory and movement.  

For a more detailed discussion, see e.g. chapter 7 here: http://grey.colorado.edu/mediawiki/sites/CompCogNeuro/images/8/89/ccnbook_01_09_2012.pdf 

It's this dichotomy that I think the BECCA system is getting at, with their distinction between a "feature creator" and a "reinforcement learner". All cortical regions contribute in some way to motor output, if only by providing contextual information to the basal ganglia or to other subcortical structures involved in shaping motor output, such as the cerebellum or superior colliculus. But the final output to the spinal cord that actually produces movement comes mostly from the motor areas. 

CLA strikes me as being potentially a major advance in simulating the cortex and its spatio-temporal "model learning", but I think the addition of reinforcement learning will be necessary in order to approach the problems of action selection, attention, working memory and cognition in a brain-like way.

-Mike  

_____________
Michael Ferrier
Department of Cognitive, Linguistic and Psychological Sciences, Brown University
michael...@brown.edu

[Brandon Rohrer]

Thanks for your assessment Mike. I strongly agree that some reinforcement-flavored learning is likely necessary in order to get interesting behaviors. Are you working in this area?
Brandon

[SeH]

---------- Forwarded message ----------

From: Jeff Hawkins <jhaw...@numenta.org>
Date: Tue, Aug 20, 2013 at 5:19 PM
Subject: Re: [nupic-dev] Mentioned presentation on action with CLA?
To: "NuPIC general mailing list." <nu...@lists.numenta.org>

Two neuro-scientists, Ray Guillery and Murray Sherman have pointed out that in every region of the neocortex they have looked, they find cells in layer 5 that project to muscles, the spinal cord, or other behavior related parts of the brain.  For example in primary visual areas V1 and V2 there are layer 5 cells that project to the Superior Colliculus which generates saccades and other eye movements.    I don’t believe they counted the basal ganglia as a “motor” destination.  Sherman and Guillery have proposed that this is the normal state of affairs, that all areas of the cortex have a motor output.  This is a beautiful idea and certainly mostly true.

 

Sherman and Guillery have written extensively about these layer 5 cells.  The axons from these cells split.  One branch goes to the muscle or motor area and the other half goes to the next region up in the hierarchy.  Thus all regions of the cortex have some motor output command, but that same command is passed up the hierarchy.  The next region thus knows what behaviors are being generated.  Layer 3 receives both sensory and motor input.

 

Layer 3 is the primary feed forward layer.  It is what I think of when thinking of the CLA.  In the general case layer 3 is building a model of sensory data plus motor commands.  Layer 5 is similar to layer 3 in many ways. I believe it is learning the same sequence of column activations and thus the  same sequences.  The unfolding patterns of layer 5 cells then associatively link to other motor areas and thus learn to control them.   It is a bit hard to describe without images.

 

Conventional wisdom says that the basal ganglia does not create behavior directly.  It seems to be responsible for selecting between alternate motor plans stored in the cortex.

 

I believe we can build a simple system consisting of one CLA representing layer 3 and another CLA representing layer 5.  The Layer 5 CLA is driven by layer 3 and associatively links to some pre-existing motor generator.  The system would learn to string together pre-existing behaviors in novel ways.  I don’t know if we would need a basal ganglia equivalent.  There are several unknowns but the basic idea seems sound.  I have a talk that goes into this idea.  We hope to record it and make it available.

Jeff

 

From: nupic [mailto:nupic-...@lists.numenta.org] On Behalf Of Michael Ferrier
Sent: Tuesday, August 20, 2013 11:54 AM

To: NuPIC general mailing list.

Subject: Re: [nupic-dev] Mentioned presentation on action with CLA?

 

The impression that I get from the neuroscience literature is that there are two basic types of learning in the brain. The first type could be called "model learning", it is what the cortex specializes in, and it's about learning hierarchical spatio-temporal models of input, from both external sensors and from other brain areas, representing the outside world, the body, and other internal states, and how they change over time. The second type is reinforcement learning, which uses built-in "reward" and "punishment" signals (such as pain or the taste of sugar) to learn what cortical patterns should be activated within a particular context of the activity of other cortical patterns, so as to maximize reward and minimize punishment. In the brain, reinforcement learning takes place in the basal ganglia, but uses input from many different areas of the cortex, and affects the activation of patterns within prefrontal and motor cortex to result in the control of attention, working memory and movement.  

 

For a more detailed discussion, see e.g. chapter 7 here: http://grey.colorado.edu/mediawiki/sites/CompCogNeuro/images/8/89/ccnbook_01_09_2012.pdf 

 

It's this dichotomy that I think the BECCA system is getting at, with their distinction between a "feature creator" and a "reinforcement learner". All cortical regions contribute in some way to motor output, if only by providing contextual information to the basal ganglia or to other subcortical structures involved in shaping motor output, such as the cerebellum or superior colliculus. But the final output to the spinal cord that actually produces movement comes mostly from the motor areas. 

 

CLA strikes me as being potentially a major advance in simulating the cortex and its spatio-temporal "model learning", but I think the addition of reinforcement learning will be necessary in order to approach the problems of action selection, attention, working memory and cognition in a brain-like way.

 

-Mike  

_____________
Michael Ferrier
Department of Cognitive, Linguistic and Psychological Sciences, Brown University
michael...@brown.edu

 

On Tue, Aug 20, 2013 at 11:34 AM, Thompson, Jeff <jef...@remap.ucla.edu> wrote:

Hello SeH,

 

While I appreciate your pointing out this open source project of which I was not aware, it seems to go against my question. I started paying attention to work on the CLA (again after many years) when I heard Jeff Hawkins speaking as he does below that "There are no pure “sensory” regions and no pure “motor” regions".  It gave me hope that this work might avoid the pitfall of the classic "input-processing-output" loop of classic AI, which BECCA clearly seems to follow (see the attached diagram).

 

We now know that there are just as many feedback connections going to back down to the "input" regions, and that action is not so different from perception (in that it uses similar machinery of prediction), and that "input" is intimately tied to the actions active during the input (instead of having "action" on the other side of world from "input", as in the BECCA diagram). 

 

I'm hopeful to see a diagram soon of many CLA modules for action and perception connected in a hierarchy which shows how action comes from similar prediction machinery as perception and how to avoid the pitfall of "input on one end, output on the other end."

 

Thank you,

- Jeff T