Re: [hansonrobotics/HEAD] ROS NMPT Salience Detector: Degree Values and Salient Feature Tracking (#117)

[Linas Vepstas]

I have multiple issues with the so-called "saliency detector" in this pull request: https://github.com/hansonrobotics/HEAD/pull/117 that I'm thinking its better to discuss these in general, rather than simply in the context of one pull request.

So:...

On Wed, Jul 27, 2016 at 10:06 AM, natnaelargaw <notifi...@github.com> wrote:

the frequency of occurrence of a salient point with in a fixed turn around time t. Further more, this method norm

First off, the entire module seems to be mis-named -- it has nothing at all to do with "saliency" -- rather, the device seems to be a motion-tracker.

Something can be highly salient, and completely still, but that is not what this does, based on the description in the README.  

Next, I'd like to see an API change. Put yourself in the place of opencog.  Suppose you were blind-folded,  and someone was whispering in your ear: "hey, there is something important happening in the upper left. Oh wait, now its less important. ... now its more important, again!  Oh and also, there's something to your right, but its not very important."

What can you possibly do with that data?  WTF? How can I possibly build a smart robot, when its effectively blind-folded, and has the above as sensory input?

Thus, I suggest completely eliminating the "degree of saliency" measure from the API, and replacing it by three things: the *size* of the moving area, *how fast* its moving, and its color.  

Now, blind-folded, you get the message: "its very small, its moving slowly, and its black", and you might guess that its a fly buzzing around. "its big and its green and its up high and its bouncing" and you might guess its a helium balloon.  "its medium and its skin toned" and you might guess a waving hand or arm.  

Size speed and color should be easy to add, and would really be useful.

For bonus points, I'd like a spinning/waving measure.  So, for example, a ceiling fan is "moving" but it never changes position. Similarly, a TV set in the background is "moving", without changing its location.  Likewise, anyone who is waving their hand -- the hand is moving but really stays in the same place.    This is very different from things the move and also change position -- someone walking across the room is both moving and changing position.  

Another bonus measure: is it moving up, down, sideways? 

There's four more bits of data that would be useful, unrelated to motion: 

-- a general sense of how bright or dark the room is, 

-- a general sense of the contrast in the image (sharp black/what, or mostly washed-out grey)

-- sudden changes in overall lighting

-- average color for the entire visual field.

Sunlit rooms tend to have high contrast; lots of blue suggests the sky is visible; a sudden change of lighting suggests that someone put their hand over the camera. 

Apparently, people cover the camera with their hands a lot during demos, and I would really really like to know when that happens.

There's other stuff I'd like to see, that would allow the robot to actually *learn*, but that is for a some future email.  The short version is that I would like to be able to send the detector some little tiny snippets of pseudo-code, and have it run that pseudo-code, and report when it triggers. The pseudo-code would consist of little algorithmic combinations of speed, color, movement, size, etc. and opencog would generate these and use them as feature detectors.  I.e. rather than hard-coding (hand-coding) the feature detectors, they would be generated dynamically, on the fly, by opencog itself, rather than by human programmers.   This is an idea for the future, though.  Important, but not urgent.

-- Linas

[Linas Vepstas]

Oops, hit "send" on the email too soon.

Perhaps the #1 most important thing on the list is the "sudden change in brightness" detector.   During the demos people either cover her camera or get up in her face, and say things like "can you see me now", and I would really like to have the robot know when this is happening -- this is probably the #1 most important demoable feature, more important than saliency, movement, anything else.

The #2 most important feature would be size-of-movement -- another thing people do is to wave their hand directly in front of the camera:  From the camera point of view, it would look like everything is moving everywhere -- the entire visual field is moving.  It doesn't have a location, because its everywhere.  Again, this is a very important situation to recognize, and relay to the robot.

Both of the above should be easy to implement, and are more important, demo-wise, than "saliency".

--linas

[Ben Goertzel]

Hi LInas, many of your suggestions are good ones and should be added to the list

However, the "visual saliency" term was not invented by us -- and the
addition of this saliency detector was especially suggested by David H
because in his prior work he has found similar code to be useful with
his robots.... So I guess I will trust him that it's a useful
indicator to have, even though I agree with you that lots of other
stuff is also useful

The reason for adding a "Degree" was that without it, the saliency
detector gave way too many false positives.... Having a degree lets
us tune a threshold denoting what is salient enough to bother doing
anything about...

ben

--
Ben Goertzel, PhD
http://goertzel.org

Super-benevolent super-intelligence is the thought the Global Brain is
currently struggling to form...

[Linas Vepstas]

OK, well, if you just write a teensy sliver of code that simply turns the head towards any direction where something "sufficiently salient" happened, then sure, that makes for a livelier, more entertaining demo. And so maybe that is worthwhile.

But it also keeps her at the "dumb robot" level, reacting to a stimulus without any clue as to why -- its like a single-neuron circuit -- whenever one end of the neuron gets tickled, the other end turns a motor.   I want to move past single-neuron reactive circuitry.   I mean, doors that swing out of your way when you approach them were were in all the supermarkets by the 1960's -- turning your head to look at something "salient" is at about the same level of sophistication.  Its been 50 years. Yes, get to that, but then go farther.

--linas

[Ben Goertzel]

For sure we all want to go further! But we need to demonstrate
moderately cool stuff in the immediate term, and this saliency
detector (appropriately leveraged within Psi rules) should be helpful
in that regard...