OpenCV for preliminary visual processing?

158 views
Skip to first unread message

Noah Bliss

unread,
Sep 16, 2016, 11:37:31 AM9/16/16
to opencog
I'm going to be showing a great deal of ignorance in this post, but who knows, it might help.

I understand an issue recently discussed with embodiment concerns methods for processing visual input. It's well known that at this time sending raw video into atomspace is a bad idea and that humans have built in visual processors that assist our conscious minds in understanding what our eyes see. (Obvious simple example being that the image is preflipped).

I understand opencog has (in some form) a python api which leads me to think using the visual processing engine OpenCV may not be a bad idea. It has a fantastic python api, allows for exporting specific data from raw video such as "33% of the screen is red", or there are 2 lines in the field of view." it also has a PHENOMINAL foreground/background separation engine that allows only a processing of new or moving objects in the field of view.

While a more mature opencog engine may prefer a more "raw" processor, I see OpenCV as a great place to start for getting useful information into atomspace quickly.

I have yet to start work on this, heck, I have yet to fully learn the ropes of the current opencog system, but I wanted to at least drop the info here in case anyone else had comments or wanted to get a head-start on me.

Best regards my friends.
Noah B.

PS: My personal experience with OpenCV was specifically dealing with automated turrets. There are great YouTube examples of using OpenCV for face-tracking webcams attached to servos, and blob isolating security cameras if you wanted specific examples to look up.

Linas Vepstas

unread,
Sep 16, 2016, 12:25:40 PM9/16/16
to opencog
Hi Noah,

Sounds like a good idea!  We currently do not have any clear-cut plans, but let me tell you a little about what has been done so far.   Currently, the main visual interface is in the repo https://github.com/opencog/ros-behavior-scripting/ ... and its pretty pathetic as vision goes.   It does use OpenCV, but only as input into a hacked version of pi_vision, and that is used to detect human faces, and map them to 3D locations.  Actually, I think that the pi_vision has been replaced by the CMT tracker, recently, which seems to work a bit better, maybe.  The ID's of the faces are placed as atoms into the atomspace.  Its super-simple, and super-low-bandwidth: basically a handful of atoms that say "I can see face 42 now".... and that's it. The 3D locations of the faces are NOT kept in the atomspace -- they are kept off-line, mostly because of bandwidth concerns.  30 frames a second of x,y,z points is not a lot, but is pointless, because we currently can't do reasoning with that info, anyway.

Re: new or moving objects: someone recently added support for "visual saliency", and I flamed them a bit for how it was done: the information pumped into the atomspace was a very simple message: "something is happening in the visual field!" which is kind-of useless.  Tell me, at least, is it big, or is it small, near or far, moving fast or moving slowly?  Is it "windmilling" i.e. moving-without-moving, like clapping hands?  or just someone standing there, swaying side to side?

With that kind of info, one can, at least, do some sort of scripted reactions: the robot can say "Hey I think I see a fly" or "what's that going on behind your left shoulder?"  Anyway, that general kind of input is handled by    https://github.com/opencog/ros-behavior-scripting/ .. the actual "state" of what is seen, what's going on is in src/self-model.scm  and so additional stuff can be added there, like "I see something small moving"...  scripted responses are in the file "behavior.scm", so if something is seen, that is where you can script a response.

All of the above is "short term". In the long term, it really has to be learning.  For that, it has to be something completely different. This email is kind-of long already but ... the idea is to pattern-mine: "if 33% of the screen is red and X happened at the same time, this is important, remember and learn that!"  Except this never happens.  So instead, lets (randomly) try "if 33% of the screen is blue and X happened at the same time..." well, hey, that DOES happen, it means you went outside on a sunny day. So this should be remembered and recorded as an important filter-event, that converts visual stuff into knowledge.  The tricky part here is that this is ... CPU intensive, requires lots of training. Its a much much harder problem.  But.. enough.

Anyway, the upshot is: "there are no rules" -- we've done very little almost nothing with vision, so you can do anything you want.  

Re: python for opencog -- your best bet is to just poke atoms into the atomspace with netcat, for example, like here: 
called from here:

and uses netcat here:

Currently, this is probably the best way to use python to get data into and out of the atomspace.

--linas



--
You received this message because you are subscribed to the Google Groups "opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscribe@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Noah Bliss

unread,
Sep 16, 2016, 9:41:24 PM9/16/16
to opencog

Thank you for the info Linas,

I'll look at the current code and see if I can get a more complete implementation of OpenCV started. You mentioned another dev's overly simple integration which, while better than nothing, hardly fulfills our goal or utilizes the full potential of OpenCV.

With luck maybe I can get the visual end of opencog a bit more useful than a glorified motion detector. :P

Thanks again! I'll report back any major advances, otherwise check the pull requests and maybe my branch of you get curious.

As a side, if I am not mistaken, atomspace does most of its storage in sql right? Perhaps I could see about offloading visual processing to a dedicated machine along with whatever camera/sensor is being used, and get that set up with an "atomspace client" that could dump pre-formatted atoms straight into the db.

If there aren't any logistical restrictions to this method, it could provide a more modular design to opencog and also reduce unnecessary primary server strain.

Noah B.


To unsubscribe from this group and stop receiving emails from it, send an email to opencog+u...@googlegroups.com.

To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "opencog" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opencog+u...@googlegroups.com.

To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CAHrUA37v2zxE7nTbqrBtw65k539v_wW1JLX2%3D2jgC3bkDoyqsw%40mail.gmail.com.

Linas Vepstas

unread,
Sep 16, 2016, 9:54:40 PM9/16/16
to opencog
On Fri, Sep 16, 2016 at 8:41 PM, Noah Bliss <l3vi...@gmail.com> wrote:

Thank you for the info Linas,

I'll look at the current code and see if I can get a more complete implementation of OpenCV started. You mentioned another dev's overly simple integration which, while better than nothing, hardly fulfills our goal or utilizes the full potential of OpenCV.

With luck maybe I can get the visual end of opencog a bit more useful than a glorified motion detector. :P


I think the "saliency detector" code is buried somewhere in here:  https://github.com/hansonrobotics/HEAD -- building and running that is probably the easiest way to get a working end-to-end demo.

Thanks again! I'll report back any major advances, otherwise check the pull requests and maybe my branch of you get curious.

As a side, if I am not mistaken, atomspace does most of its storage in sql right?

Only if you actually turn that on. Otherwise everything is in RAM.
 

Perhaps I could see about offloading visual processing to a dedicated machine along with whatever camera/sensor is being used, and get that set up with an "atomspace client" that could dump pre-formatted atoms straight into the db.

netcat does that.  The python snippet with netcat was an example. 

For everything else, we use ROS. There's a bit of a learning curve for ROS, but its the ideal way for running multi-machine, distributed processing. 

--linas
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscribe@googlegroups.com.

To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "opencog" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opencog+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscribe@googlegroups.com.

To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.

Ben Goertzel

unread,
Sep 17, 2016, 1:06:37 AM9/17/16
to opencog
Noah,

OpenCV, as you know, is basically a toolkit, not an integrated system....

Right now, indeed, the vision processing we have hooked up to OpenCog
is restricted to

-- face detection, face tracking, face recognition
-- visual saliency identification
-- luminance-change detection

which is all pretty simple...

We have previously experimented with using kinfu to make a 3D map of a
robot's surroundings.... OpenCog's TimeSpaceMap is well-suited to
represent the output of kinfu (or similar) in a way that's integrated
with the Atomspace...

We have also played a bit with Region-CNNs, as a way of identifying
what objects are where in a visual scene (initially from a closed
class of objects)

So if I were going to integrate additional external vision tools with
OpenCog, I'd probably start with kinfu-or-similar, plus
(Region-CNN-with-trained-models)-or-similar...

Medium term, it's more interesting to integrate deep NN vision into
OpenCog, which Yenat is working on in our Addis office, but that's a
research project, whereas feeding output of kinfu and Region-CNN into
OpenCog is "just" complex software integration and training/tuning,
not really original research...

Anyway I am curious what specific visual functions you are thinking of adding?

-- Ben G
>>>> an email to opencog+u...@googlegroups.com.
>>>>
>>>>
>>>> To post to this group, send email to ope...@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/opencog.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "opencog" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> opencog+u...@googlegroups.com.
>>> To post to this group, send email to ope...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/opencog.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/opencog/CAHrUA37v2zxE7nTbqrBtw65k539v_wW1JLX2%3D2jgC3bkDoyqsw%40mail.gmail.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "opencog" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to opencog+u...@googlegroups.com.
>> To post to this group, send email to ope...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/CABpkOB-4HYkmtoqnBNpWaqdRKwou-w9CPevYOtNDYxGiJL9N%3Dg%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+u...@googlegroups.com.
> To post to this group, send email to ope...@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opencog/CAHrUA34J1i2qe-KTOUEZ%2B8gXXWhW1jmUoDWQt2H%3DTY7copfXRw%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
Ben Goertzel, PhD
http://goertzel.org

Super-benevolent super-intelligence is the thought the Global Brain is
currently struggling to form...

Noah Bliss

unread,
Sep 17, 2016, 11:18:15 AM9/17/16
to opencog

The concept of building a 3d map of the environment in atomspace certainly would be a better goal of there had to be a choice between the two. I'll definitely read up on kinfu before starting any sort of work.

A few simple examples of information gained through a more mature OpenCV implementation could consist of the following:

A simple place to start that would have little overhead and export atoms easily used could be:

Knowing environment. Consistent items not the focus.

Blob statistics, delimited by motion
Size of blob
Color of blob
Location on fov
Speed and direction
Acceleration

Advanced sampling:
Division of blob into sections, quarters horizontally,
Shape/size/color/edge flat or rounded statistics of that quadrant
Vertical division by the same standards.

Obviously this would not be the end. We could divide a blob into more slices, account for variation in background, etc. We would need a more advanced way to get more information from a visual feed to get it anywhere near human-like. But we could at least get more useful data than we currently are.

This kind of implementation could potentially augment other more "logical" representations of the environment by providing a more analog "eye-like" processing system. It also has the advantage of being potentially faster to implement and scale.

I don't see this implementation ever being a replacement for any sort of 3d map formation, but rather a low-overhead way of quickly making sense of raw visual input without pumping raw video into atomspace.


To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CACYTDBf1qsF21PyHU9V7t_nRPNtyiqn5FMjOtPeyFFrqMBzNhg%40mail.gmail.com.

Ben Goertzel

unread,
Sep 17, 2016, 11:22:34 AM9/17/16
to opencog
Ah, OK, I get it.... Yeah, having openCog know the size and color and
direction of blobs would be nice... I'm not quite sure what to do with
it immediately though...

I mean, for the "deep learning vision in OpenCog" direction we're
gonna feed fine-grained visual data (pixels, voxels) into neural nets
wrapped in Grounded SchemaNodes associated with RegionNodes in the
TimeSpaceMap...

OTOH, for hand-coding simple response rules for a robot or avatar
controlled by OpenCog, you kinda need to know what you're seeing not
just that you're seeing a red blob in a certain position, moving
toward a blue blob...

So one could argue that knowing where blobs of various colors are and
where they're moving is

-- too low-level for hand-coding response rules for an opencog agent

-- not low-level enough to fuel deep visual learning in opencog

... but maybe I'm misunderstanding something?

ben
> https://groups.google.com/d/msgid/opencog/CABpkOB_mL24sT40JvcL%3DkE6BnqHHMpWGURJdWoWmVJd-p%3D2G0A%40mail.gmail.com.

Linas Vepstas

unread,
Sep 17, 2016, 6:48:42 PM9/17/16
to opencog
On Sat, Sep 17, 2016 at 10:22 AM, Ben Goertzel <b...@goertzel.org> wrote:
Ah, OK, I get it....  Yeah, having openCog know the size and color and
direction of blobs would be nice... I'm not quite sure what to do with
it immediately though...

I do.  One could wave something around in front of the webcam, and ask "what do you see?" and the robot should be able to respond:  "I see something red moving around".

This is, in principle, a demo that is achievable in the short-term:  poke some moving-things atoms in the atomspace, and wire up the NL pipeline to be able to ask questions about moving things.  Getting the color right is certainly a nice touch.

I imagine that the NN crowd would be quite unimpressed by this, but what it does provide is an authentic demo of symbolic manipulation of sensory data.  I think this is an important milestone/demo for us.

Oh, I see that you implied this, already:
 -- too low-level for hand-coding response rules for an opencog agent

Nah. We need to at least prototype this stuff, hook it up into the pipeline. See how it works. You can get fancier, later if you can.  The sheer act of doing stuff like this, by hand, exposes the various weak areas, the various design flaws.  Its very easy to imagine something whizzy, but I think the baby steps of getting stuff like this functioning is a good educational experience.

If everyone actually pays attention to how it turns out, then the education is not wasted, and we don't end up repeating old mistakes, ... which we have a habit of doing.

 
> Blob statistics, delimited by motion
> Size of blob
> Color of blob
> Location on fov
> Speed and direction
> Acceleration

The above would be nice, as long as they are hooked up to language. The "advance" below are probably not useful, for getting the robot to talk about what it sees.
 
>
> Advanced sampling:
> Division of blob into sections, quarters horizontally,
> Shape/size/color/edge flat or rounded statistics of that quadrant
> Vertical division by the same standards.

>
> This kind of implementation could potentially augment other more "logical"
> representations of the environment by providing a more analog "eye-like"
> processing system. It also has the advantage of being potentially faster to
> implement and scale.
>
> I don't see this implementation ever being a replacement for any sort of 3d
> map formation, but rather a low-overhead way of quickly making sense of raw
> visual input without pumping raw video into atomspace.

Bingo!  Exactly.

--linas 

>> >>>>
>> >>>>
>> >>>> To post to this group, send email to ope...@googlegroups.com.
>> >>>> Visit this group at https://groups.google.com/group/opencog.
>> >>>> To view this discussion on the web visit
>> >>>>
>> >>>> https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com.
>> >>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to a topic in the
>> >>> Google Groups "opencog" group.
>> >>> To unsubscribe from this topic, visit
>> >>> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
>> >>> To unsubscribe from this group and all its topics, send an email to

>> >>> To post to this group, send email to ope...@googlegroups.com.
>> >>> Visit this group at https://groups.google.com/group/opencog.
>> >>> To view this discussion on the web visit
>> >>>
>> >>> https://groups.google.com/d/msgid/opencog/CAHrUA37v2zxE7nTbqrBtw65k539v_wW1JLX2%3D2jgC3bkDoyqsw%40mail.gmail.com.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "opencog" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an

>> >> To post to this group, send email to ope...@googlegroups.com.
>> >> Visit this group at https://groups.google.com/group/opencog.
>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/opencog/CABpkOB-4HYkmtoqnBNpWaqdRKwou-w9CPevYOtNDYxGiJL9N%3Dg%40mail.gmail.com.
>> >>
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "opencog" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > To post to this group, send email to ope...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/opencog.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/opencog/CAHrUA34J1i2qe-KTOUEZ%2B8gXXWhW1jmUoDWQt2H%3DTY7copfXRw%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Ben Goertzel, PhD
>> http://goertzel.org
>>
>> Super-benevolent super-intelligence is the thought the Global Brain is
>> currently struggling to form...
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "opencog" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> To post to this group, send email to ope...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/CACYTDBf1qsF21PyHU9V7t_nRPNtyiqn5FMjOtPeyFFrqMBzNhg%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> To post to this group, send email to ope...@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opencog/CABpkOB_mL24sT40JvcL%3DkE6BnqHHMpWGURJdWoWmVJd-p%3D2G0A%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
Ben Goertzel, PhD
http://goertzel.org

Super-benevolent super-intelligence is the thought the Global Brain is
currently struggling to form...

--
You received this message because you are subscribed to the Google Groups "opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscribe@googlegroups.com.

To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.

Andi

unread,
Sep 19, 2016, 6:02:10 PM9/19/16
to opencog, linasv...@gmail.com

i am just dreaming - reading this thread. But i want to share the dream :)

I love watching Snooker.
And I love watching RonnyOSullivan playing in his top form!

Just imagine Nao becoming a world champion of Snooker :)

Maybe 2030 we will remember this thread, where it begun.
--
>> >>>> an email to opencog+u...@googlegroups.com.

>> >>>>
>> >>>>
>> >>>> To post to this group, send email to ope...@googlegroups.com.
>> >>>> Visit this group at https://groups.google.com/group/opencog.
>> >>>> To view this discussion on the web visit
>> >>>>
>> >>>> https://groups.google.com/d/msgid/opencog/1baaeade-567a-4456-aaa3-85e2b003fc7b%40googlegroups.com.
>> >>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to a topic in the
>> >>> Google Groups "opencog" group.
>> >>> To unsubscribe from this topic, visit
>> >>> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
>> >>> To unsubscribe from this group and all its topics, send an email to

>> >>> To post to this group, send email to ope...@googlegroups.com.
>> >>> Visit this group at https://groups.google.com/group/opencog.
>> >>> To view this discussion on the web visit
>> >>>
>> >>> https://groups.google.com/d/msgid/opencog/CAHrUA37v2zxE7nTbqrBtw65k539v_wW1JLX2%3D2jgC3bkDoyqsw%40mail.gmail.com.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "opencog" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an

>> >> To post to this group, send email to ope...@googlegroups.com.
>> >> Visit this group at https://groups.google.com/group/opencog.
>> >> To view this discussion on the web visit
>> >>
>> >> https://groups.google.com/d/msgid/opencog/CABpkOB-4HYkmtoqnBNpWaqdRKwou-w9CPevYOtNDYxGiJL9N%3Dg%40mail.gmail.com.
>> >>
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "opencog" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > To post to this group, send email to ope...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/opencog.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/opencog/CAHrUA34J1i2qe-KTOUEZ%2B8gXXWhW1jmUoDWQt2H%3DTY7copfXRw%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Ben Goertzel, PhD
>> http://goertzel.org
>>
>> Super-benevolent super-intelligence is the thought the Global Brain is
>> currently struggling to form...
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "opencog" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> To post to this group, send email to ope...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/CACYTDBf1qsF21PyHU9V7t_nRPNtyiqn5FMjOtPeyFFrqMBzNhg%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> To post to this group, send email to ope...@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opencog/CABpkOB_mL24sT40JvcL%3DkE6BnqHHMpWGURJdWoWmVJd-p%3D2G0A%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
Ben Goertzel, PhD
http://goertzel.org

Super-benevolent super-intelligence is the thought the Global Brain is
currently struggling to form...

--
You received this message because you are subscribed to the Google Groups "opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+u...@googlegroups.com.

To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.

Noah Bliss

unread,
Sep 20, 2016, 11:07:37 PM9/20/16
to opencog
I was reflecting on your email Ben... I agree, arbitrarily segmenting a blob into predefined sections for processing may not be the best focus of this long-term. Perhaps for a small region this would be useful. (e.g. "What color is the object I am holding?" Then it would be able to set an arbitrary vicinity in the FOV inside which to search for objects, around your hand.) But perhaps a more useful feature would be vector creation using various algorithms on the blob. 

For example, if we were to hold up a solid white flash card, the system could distinguish it by its markedly distinct color. Using a degree of fuzziness it could then draw lines around the uniform regions. This would allow us to "wireframe" an image from raw video input and potentially even allow us to "imagine" or recreate a reasonably close representation of what is in the field of view simply by mapping out the resulting vectors. This would also be scalable since the precision of our edges could start nice and blocky (read: Nintendo level graphics) and as our code efficiency and hardware allow, we can refine the precision of these vectors into more complex and detailed renderings. 

This could also end up playing nicely with other sensors or systems that are used for 3d spacial construction. For example, we could have a second camera or "eye" and use it for distance measuring, providing us further data about the lines being drawn (depth and slope). This could also potentially be paired with sonar or radar to the same end. Both sensors could be feeding the same 3d construct to provide more data and better overall precision. 


I am getting ahead of myself with this though. I'll start with catching up to the abandonware first and report any progress beyond that. 

Best regards,
Noah Bliss

Noah Bliss

unread,
Sep 20, 2016, 11:15:49 PM9/20/16
to opencog
Afterthought:

Checked out Kinfu, looks to do something quite similar. I am somewhat concerned about the resolution currently offered though. I'll see if there is a way to scale it down to simpler objects for easier atomspace digging and verification. Otherwise I do understand the draw of Kinfu. Perhaps a hybrid-type system would be ideal. Off to do more research...


On Friday, September 16, 2016 at 11:37:31 AM UTC-4, Noah Bliss wrote:

Noah Bliss

unread,
Jan 18, 2017, 5:15:29 PM1/18/17
to opencog
College has kept me busy but I finally took the time to go through the pivision code on the hansonrobotics github. Correct me if I am wrong, but I saw no integration of visual information being fed into opencog, at least not directly. I don't know what kind of chewing ROS does to the information it gets from pi_vision, but it doesn't seem that is really the design philosophy we are going for based on the CogPrime guidelines: as little hand-holding as possible and let the system form its own rules based on patterned inputs right? Since There seems to be little meaningful integration of pi_vision into opencog and I have a personal dislike for the design philosophy of hansonrobotics (where opencog seems to be just a backend engine for one aspect of functionality rather than the core) I was looking to write a standalone visual processor that hooks straight into a CogPrime build. Obviously python would probably be best suited for this, but what would be the most desired way of getting information into the system? You want me to just use the python api to dump atoms into atomspace? Do they need to be tagged with timestamps/other forms of metadata or are those provided already through other CogPrime systems?

Any guidance is appreciated. I am not a neural networks/AI expert by any means and I'd like to be practically useful now rather than only after I finish reading the Bible that is the Opencog codebase.


Noah Bliss

Ben Goertzel

unread,
Jan 18, 2017, 11:21:28 PM1/18/17
to opencog
Ralf Mayet in HK is working on an approach such as you describe... help would be valued ... more later...

--
You received this message because you are subscribed to the Google Groups "opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscribe@googlegroups.com.

To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/ba2a5a62-ac97-4abe-ba60-5b69642ee4f5%40googlegroups.com.

Ben Goertzel

unread,
Jan 20, 2017, 12:09:30 PM1/20/17
to opencog, Ralf Mayet
Noah,

What Ralf is working on is making a "DeStin-like" visual processing
hierarchy in Tensorflow, probably using InfoGAN as a key ingredient
(within each "DESTIN-like node"), and then integrating this hierarchy
with OpenCog so that OpenCog can be used to recognize semantic
patterns in the state of the visual processing hierarchy, and these
semantic patterns can be fed back to the visual processing hierarchy
as additional features at various levels of the hierarchy

This is a lot of work, it's original research, and it will probably
take about 4-6 more months to lead to useful results.... If you would
like to get involved Ralf can help you get up to speed

thanks
ben
>> email to opencog+u...@googlegroups.com.
>> To post to this group, send email to ope...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/ba2a5a62-ac97-4abe-ba60-5b69642ee4f5%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.



--
Ben Goertzel, PhD
http://goertzel.org

“I tell my students, when you go to these meetings, see what direction
everyone is headed, so you can go in the opposite direction. Don’t
polish the brass on the bandwagon.” – V. S. Ramachandran

Noah Bliss

unread,
Jan 20, 2017, 3:17:55 PM1/20/17
to opencog, Ralf Mayet
Ben,

Sounds good, I would definitely be interested. Seems pretty ambitious but no one ever achieved great things by aiming low. I noticed Ralf was CC'd in this topic so if he could reach out I am available on all major platforms and while I may spend most of my initial time learning and "looking over his shoulder" anything I can contribute I will. 

Noah Bliss


You received this message because you are subscribed to a topic in the Google Groups "opencog" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opencog+u...@googlegroups.com.

To post to this group, send email to ope...@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CACYTDBfyv5NxMAYtj9G1PzzwUo1oiRYTuDNVPyVWdwFOABic6w%40mail.gmail.com.

Linas Vepstas

unread,
Jan 20, 2017, 8:53:45 PM1/20/17
to opencog, Ralf Mayet
I don't think I ever spoke very carefully to Ralf about this, nor am I
sure that the seed I tried to plant ever germinated in Ben's head, so
let me restart from scratch. Perhaps this is something Noah could
work on?

For simplicity, let me work with sound, because its 1D (as a time
series) not 2D like vision, so maybe simpler. Here's the idea:

step 0) get some audio framework, that tells you instantaneous
loudness (aka total power), frequencies (i.e. power in different
frequency bands).

step 1) randomly create a few dozen or a few hundred or thousand
simple sound filters, composed from above. Each filter outputs either
true or false -- it either triggered or it didn't

For example, one random filter might be "if there was a loud sharp
sound then event-true" where "loud sharp sound" was a rapid rise then
a rapid fall in total power in 0.1 seconds. In greater detail: "if
(current-loudness<0.31415 and loudness-at-time(t-0.2765)> 0.789 and
loudness-at-time(t-0.555) < 0.123) then send-event-to-cogserver"

The numbers above are picked randomly, as well as one or two or three
and-terms in the if-clause. You could add frequency bands, randomly,
too. For example "if pop then hiss" e.g. opening a soda
can-type-sound.

Again: there might be hundreds of these random filters running at once.

step 2) run these filters on one or more live microphones, for days/weeks

step 3) inside the cogserver, look for anything that might correlate
with the events. e.g. was the loud pop associated with a sudden
change in light? with a sudden movement in the visual field?
something/anything else going on at the same time?

If so, then mark the particular filter as "important" (increment it's
count-truth-value) Ben might call this filter as being "surprising"
-- it has high surprisingness.

step 4) after a few days discard the filters with low
importance/surprisingness. i.e. go back to step 1)

step 5) Use the high-surprisingness filters, and "genetically mutate"
them: try random variants of them. Try random combinations of them.

step 6) (optional) take the high-surprisingness filters and compile
them into high-efficiency GPU code, so that we don't waste CPU time in
step 2 -- running hundreds of filters for step 2 is probably very
cpu-intensive, so we need a way of compiling these into something
fast.

That's it. That's the meta-pseudo-code. Now, maybe if you are
clever, you can somehow replace steps 1 and 5 of the above with some
sort of tensorflow-ish deep NN or whatever. But the goal is to
correlate sounds with other significant events in the environment. ...
and to do it so that the system learns on it's own what ind of sounds
are important, and which ones are not.

The main problem here is getting enough data. Baby humans get tones
of audio data correlated with the environment, and I imagine that
forest creatures get even more, what with daybreak, birdsong, violent
predators, crazy forest shit. Its hard to imagine how to get that
kind of a sonically-rich environment for a robot, unless you put the
robot in a back-pack and went hiking around in the city or country or
wherever.

Or tapped into the mics on 1 million cellphones... you know, some nasty app ...

-- linas
> To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CACYTDBfyv5NxMAYtj9G1PzzwUo1oiRYTuDNVPyVWdwFOABic6w%40mail.gmail.com.

Ralf M.

unread,
Jan 21, 2017, 12:12:00 AM1/21/17
to Noah Bliss, opencog
Hey Noah,

>> On Friday, September 16, 2016 at 11:37:31 AM UTC-4, Noah Bliss wrote:
>> I understand an issue recently discussed with embodiment concerns
>> methods for processing visual input. It's well known that at this time
>> sending raw video into atomspace is a bad idea and that humans have built in
>> visual processors that assist our conscious minds in understanding what our
>> eyes see. (Obvious simple example being that the image is preflipped).

as Ben had mentioned, I am currently working towards creating a "vision processor" that should hopefully exactly do what you describe. The core goal of this system is to extract and generate states (patterns) from the raw vision input that will be easily digestible by the pattern miner in OpenCog. Later we want to have feedback from OC back into the system to improve cognition far beyond the state of the art. The current architecture is based on DeSTIN and implemented in Tensorflow.

The Visual Cortex in the human brain does many, many things, some are well understood, others are not and it is highly intertwined with other parts of the brain. It is my understanding that it should have an inspirational role in the design of the system, we do not want to build an exact replica. However, some of the features we will definitely need. 

These are some of the things we are looking at from a conceptual point of view:

  - scale and rotational invariance (the system should be reasonably able to deal with these things by itself and possibly learn the invariances it needs from data),
  - finding "disentangled representations" that will be much easier to parse by external processes (for this take a look at InfoGAN),
  - how to deal with visual clutter and do full-scene parsing (this will hopefully be addressed by having feedback from OpenCog back into the network).

And there are many more aspects on the horizon...

>> 2017-01-21 4:17 GMT+08:00 Noah Bliss <l3vi...@gmail.com>:
>> Seems pretty ambitious but no one ever achieved great things by aiming low.

Haha, yes while it does sound ambitious and far fetched at the moment, the more research I do around this topic the more I feel we are on the cusp of creating exactly such a system. Not only me, as a puny individual, but also the research community as a whole. If you look into current research around neural nets and vision processing you can tell people are figuring out ways to find disentangled representations and extract semantic meaning from images. We have made great strides in learning robust feature detectors and doing classification from unlabeled data in the past few years on low-dimensional toy-datasets, the next logical step is to understand these systems more, have them be able to do full scene parsing, and attach actual meaning to the things they recognize.


I'd be psyched if you want to get involved! While the implementation of the 'vision-daemon' is still in it's absolute infancy (it's not even a daemon yet) and I am making design decisions faster than I can document them, I am sure there is ample opportunity to collaborate. Take a look at the code at the link below, I am also hoping to provide more documentation and scientific references inside of the Github Wiki soon, for now please contact me directly if you have any questions.



Regards,
--
Ralf Mayet


>> To post to this group, send email to ope...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/opencog.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/opencog/ba2a5a62-ac97-4abe-ba60-5b69642ee4f5%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.



--
Ben Goertzel, PhD
http://goertzel.org

“I tell my students, when you go to these meetings, see what direction
everyone is headed, so you can go in the opposite direction. Don’t
polish the brass on the bandwagon.” – V. S. Ramachandran

--
You received this message because you are subscribed to a topic in the Google Groups "opencog" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opencog/31yT3osM_zI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opencog+unsubscribe@googlegroups.com.

Ben Goertzel

unread,
Jan 21, 2017, 10:21:12 AM1/21/17
to opencog, Ralf Mayet
Hmm, I'll think about this, thx...

In a way, I wonder if this is related to what InfoGAN does, with its
latent variables that need to have high mutual information with the
state of the NN modeling the data..

more later...
> To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CAHrUA34pb-jRc00kvauz%2BJtyCMJeHdjgW3yUaFWv5b6w%3DPHdhA%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages