how to train a CNN with feature vectors and not images?

[smajid...@gmail.com]

Hi

I have a dataset of mocap (motion caption) positions as vectors which I am going to train a DNN for this dataset.

the sample data would be like a 140-D dimension vector. 

Is it possible to train a CNN for this kind of data? I have

how to use convolution layers for this kind of data as kernels are e.g.5x5 while the data is  a vector?

If I make the data in a form of matrix, is it possible to train a pretrained CNN e.g. alexnet for this dataset?

Best

Majid

[antican...@gmail.com]

not to pretend to be an expert here,

But as I understand Opencog works very different from neural networks. To consult on how to use convolutional neuralnetwork you probably should
ask another forum

[Ben Goertzel]

Indeed this is not an OpenCoggy question, but some of us may be able
to help... is this dynamic data or instantaneous data you're trying to
classify?

> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+u...@googlegroups.com.
> To post to this group, send email to ope...@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opencog/1a04d763-3dfa-473f-a240-a0e452f6faba%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Ben Goertzel, PhD
http://goertzel.org

"I am Ubik. Before the universe was, I am. I made the suns. I made the
worlds. I created the lives and the places they inhabit; I move them
here, I put them there. They go as I say, then do as I tell them. I am
the word and my name is never spoken, the name which no one knows. I
am called Ubik, but that is not my name. I am. I shall always be.” --
Ubik

[Andi]

Hello All,
I do not want to disturb the ongoing work so an answer to this question is not urgent,
but it will help me during my investigations within the next month.

What in the hell could prevent me to look at the Atomspace as a certain kind of Neuronal Network?

Please don't tell me:"Because it is a hypergraph" haha....

One of my aims is, try to port the whole thing, or some parts, to hardware. Maybe a bunch of some GPU's, PLD's and a CPU can do it. It seems that they are designing some interesting machines for Deep Learning - so maybe even nothing new has to be invented..

For first steps I think it should at least be possible to use some GPU-power to do some work in parallel or is there really a theoretical barrier for paralleling some work that I cannot see in the moment?

Please don't be afraid,  I know what kind of challenging task this is and would carry it on my own back. But maybe it is not so much work if the right approach is found...
At least I want to investigate this - so any red lights blinking?

--Andi

[Gaurav Gautam]

I may be wrong, but as far as I understand one problem may be that neural networks are not really graphs or hypergraphs. Books show them as a set of layers and some connecting edges which looks a lot like a graph, but when they are implemented in code, they mostly are matrix operations. So, as far as I understand a program implementing a neural network will be doing matrix operations. If I am right about this, then I don't see how seeing atomspace as a neural network will help.

What I am saying is that I don't think the atoms and links can be connected to make a neural network straightforwardly. Of course, one could make atoms that represent the coefficients of the model that the CNN represents and then connect those with links that have weights and then make a function that can take such a hypergraph and tune the weights. But wouldn't that be very inefficient? Wouldn't you want to just represent a feature vector in atomese and then run CNN on it (through an external library perhaps) and get results in atomese that the other algorithms can pick up? But then again, I have very little idea what I am talking about, so I may be way off.

[Andi]

Hi Gaurav,
I think that it should be possible to take advantage of the massiv parallel computing power of a GPU also for the OpenCog system like it is used for NNs.

So, are there any NP-hard problems inside the box?

--Andi

[Ben Goertzel]

Guys,

The big problem with using GPUs for OpenCog is that most OpenCog
cognitive algorithms would be better suited for MIMD parallelism than
for SIMD parallelism

To put it simply, GPUs are SIMD parallel which means they are suited
for cases where one needs to repetitively do the same thing over and
over to multiple data items ... Neural net algorithms tend to be like
this. In OpenCog, ECAN is also like this (as it's basically a
special variant of an attractor neural net). But the other OpenCog
algorithms are generally not like this. They are tractably
parallelizable, but only on a MIMD parallel substrate...

Another issue is RAM access -- for OpenCog (or any system centered on
manipulation of large graphs) the biggest cost in terms of processing
time is RAM access for small, hard to predict RAM read/writes .... So
if the bulk of RAM is not on the GPU then all the savings realized by
the GPU will be eaten by GPU-CPU messaging

What you really want for OpenCog is a MIMD parallel chip, with a lot
of RAM, and special connects btw the processors' caches..... This
would let you put OpenCog on embedded devices in a useful way, and
also build OpenCog-tailored supercomputers.... These would be
customized for OpenCog in the same sense that the current crop of
"deep learning chips" are customized for hierarchical NNs.

Mandeep Bhatia and I have sketched some ideas about an "OpenCog chip"
along these lines but have been too busy with other stuff to refine
these ideas into a detailed design that can be given to an FPGA
programmer for prototyping... it will happen eventually ;)

For the present, GPUs could be used for certain special purposes
within OpenCog -- e.g.

-- ECAN importance spreading across an Atomspace whose structure does
not frequently change

-- maybe, with a lot of work, some sort of limited (but could still be
very useful) pattern matching against an Atomspace whose structure
does not frequently change


These could be quite valuable but wouldn't constitute "porting the
whole OpenCog to GPU"

-- Ben

> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+u...@googlegroups.com.
> To post to this group, send email to ope...@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/opencog/067f3e5c-493e-4f68-b7fb-1c2b46a05bb2%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.



--
Ben Goertzel, PhD
http://goertzel.org

Super-benevolent super-intelligence is the thought the Global Brain is
currently struggling to form...

[Linas Vepstas]

Hi Andi,

Ben has a good answer, and to emphasize, let me add this:   Think of the atomspace as being a collection of trees.  The atoms are the nodes in the tree.  Any one atom can appear in many trees, and so the whole thing is in fact tangled into a big matt, like a rhizome https://www.google.com/search?q=rhizome&tbm=isch 

The pattern matcher starts at one atom, and walks the rhizome, exploring nearest neighbors, until all the entire neighborhood is explored (and a match is found, or some other (local) computation is performed). 

The problem is that the atoms are scattered randomly through RAM, so when the nearest neighbor walk happens, random locations in RAM get visited.  I'm guessing that there is a lot of cache-miss going on two:  If you have, say, a CPU cache that is 8-way, 4-associative, then you could have maybe 32 atoms in the cache, but the chance that the 33rd atom will accidentally be in one of the existing cache lines is just about zero, and so the graph walk will have a 99.9% cache-miss rate.   (most graphs that get searched have more that 32 atoms in them. )

Hmm, I have an idea -- I guess the atomsapce *could* keep track of individual connected components  (create a bag of trees, which are connected by one or more atoms) -- any given search is guaranteed to stay in just one bag, and so maybe one could download the entire bag to the gpu before starting a search.   Could work if the bags are small enough to fit in GPU ram.

Maybe allocation could be changed to improve cache locality: allocate atoms so that they are more likely to be on the same cache line if they are also connected.  But this becomes a hard, fiddly computer-science problem...

--linas

To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/b3ca200e-c039-4417-96dc-5ef3f37f38ea%40googlegroups.com.

[Ben Goertzel]

A lot of people have tried to make graph manipulations work
effectively on GPU, and have also tried to make parallel backtracking
work effectively on GPU .. there are various MS and PhD theses in
these directions around ... there are logical-looking methods that
have been tried, but I've never seen anyone demonstrate or report
significant speedup on nontrivial problems..... In the end one is
trying to fix a square peg in a round hole (er, a graph-shaped peg in
a low-dimensional-array shaped hole..), and it's awkward and not so
efficient...

> https://groups.google.com/d/msgid/opencog/CAHrUA35TcD%3DK3b%3Dd0eOFMfv6n3qoS3Einn0pUjJpvN84VQ27AA%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.



--
Ben Goertzel, PhD
http://goertzel.org

[Linas Vepstas]

Here  this:

http://ci-journal.net/index.php/ciej/article/viewFile/327/315/2090

https://developingwritersdotorg.files.wordpress.com/2013/07/ee98c-topo_rhizome2.jpg

On Tue, Aug 2, 2016 at 11:05 AM, Linas Vepstas <linasv...@gmail.com> wrote:

[Andi]

Ty, Linas!
A picture tells more than 1000 words :)

[Andi]

Thank you Ben,
your answer helps me to start my investigations from the right perspective.
More between your lines..

Am Dienstag, 2. August 2016 15:19:49 UTC+2 schrieb Ben Goertzel:

Guys,

The big problem with using GPUs for OpenCog is that most OpenCog
cognitive algorithms would be better suited for MIMD parallelism than
for SIMD parallelism

 

O.K.

To put it simply, GPUs are SIMD parallel which means they are suited
for cases where one needs to repetitively do the same thing over and
over to multiple data items ...  Neural net algorithms tend to be like
this.   In OpenCog, ECAN is also like this (as it's basically a
special variant of an attractor neural net).   But the other OpenCog
algorithms are generally not like this.  They are tractably
parallelizable, but only on a MIMD parallel substrate...

 
O.K.

Another issue is RAM access -- for OpenCog (or any system centered on
manipulation of large graphs) the biggest cost in terms of processing
time is RAM access for small, hard to predict RAM read/writes ....  So
if the bulk of RAM is not on the GPU then all the savings realized by
the GPU will be eaten by GPU-CPU messaging

O.K.

What you really want for OpenCog is a MIMD parallel chip, with a lot
of RAM, and special connects btw the processors' caches.....   This
would let you put OpenCog on embedded devices in a useful way, and
also build OpenCog-tailored supercomputers....  These would be
customized for OpenCog in the same sense that the current crop of
"deep learning chips" are customized for hierarchical NNs.

This sounds like a SPARK-architecture could do it. SPARC T5

Mandeep Bhatia and I have sketched some ideas about an "OpenCog chip"
along these lines but have been too busy with other stuff to refine
these ideas into a detailed design that can be given to an FPGA
programmer for prototyping... it will happen eventually ;)

Maybe a partner like Oracle/Sun could be of value. They are very good doing things like this.

I am very sure that <Artificial Real World Cognition> will be the next big bussines and that it not can be done just with NNs.
It will happen here first.

To me it seems possible to get a partner like Oracle/Sun - they need the bussines and are always looking for new opportunities. For them it would be easy to spend some servers and help to port at least the core funktions of opencog and make it accessible via WWW. This would end up in some cogy cloud computing - why not?
 

For the present, GPUs could be used for certain special purposes
within OpenCog -- e.g.

-- ECAN importance spreading across an Atomspace whose structure does
not frequently change

-- maybe, with a lot of work, some sort of limited (but could still be
very useful) pattern matching against an Atomspace whose structure
does not frequently change

OK.

These could be quite valuable but wouldn't constitute "porting the
whole OpenCog to GPU"

 

of course...
sorry for being not precise.
I was just to entusiastic :)

[Linas Vepstas]

On Thu, Aug 4, 2016 at 5:29 PM, Andi <gabil...@gmail.com> wrote:

This sounds like a SPARK-architecture could do it. SPARC T5

Please re-read and re-think. The Sparc architecture is just another RISC superscalar architecture, not really or fundamentally different than MIPS or POWER, or even the risc superscalar core inside of the Intel x86 chips.  The primary difference is that the Sparc has a big register file and a register window, used for the C/C++/Java stack push and pop, whereas the other arches use those transistors register scoreboarding, coloring and so on. 

https://en.wikipedia.org/wiki/Superscalar_processor

https://en.wikipedia.org/wiki/Reduced_instruction_set_computing

https://en.wikipedia.org/wiki/SPARC

https://en.wikipedia.org/wiki/Scoreboarding

Again, the problem is how to traverse rhizomes quickly and efficiently, given the non-locality of memory access. 

--linas

 

[Andi]

Ty Linas!
->

Am Dienstag, 2. August 2016 18:05:28 UTC+2 schrieb linas:

Hi Andi,

Ben has a good answer, and to emphasize, let me add this:   Think of the atomspace as being a collection of trees.  The atoms are the nodes in the tree.  Any one atom can appear in many trees, and so the whole thing is in fact tangled into a big matt, like a rhizome https://www.google.com/search?q=rhizome&tbm=isch 

What I already understood is that it is related to PROLOG-trees but with real-truth values and attention values.
 

The pattern matcher starts at one atom, and walks the rhizome, exploring nearest neighbors, until all the entire neighborhood is explored (and a match is found, or some other (local) computation is performed). 

The problem is that the atoms are scattered randomly through RAM, so when the nearest neighbor walk happens, random locations in RAM get visited.  I'm guessing that there is a lot of cache-miss going on two:  If you have, say, a CPU cache that is 8-way, 4-associative, then you could have maybe 32 atoms in the cache, but the chance that the 33rd atom will accidentally be in one of the existing cache lines is just about zero, and so the graph walk will have a 99.9% cache-miss rate.   (most graphs that get searched have more that 32 atoms in them. )

Ty for this precise technical information it will help me much by thinking about performance. 

Hmm, I have an idea -- I guess the atomsapce *could* keep track of individual connected components  (create a bag of trees, which are connected by one or more atoms) -- any given search is guaranteed to stay in just one bag, and so maybe one could download the entire bag to the gpu before starting a search.   Could work if the bags are small enough to fit in GPU ram.

I had visions like that. Its a problem of mapping. this will be solved more in praxis than in theorie. We will see what can be done...
 

Maybe allocation could be changed to improve cache locality: allocate atoms so that they are more likely to be on the same cache line if they are also connected.  But this becomes a hard, fiddly computer-science problem...

 

In the momend I am not so much familiar with caching technics. In my answer to Ben I wrote that maybe a SPARK-architecture could do it. As fare as I know and understand there is some room for cachecontrol...

I think that after the current goals are achived - robots are doing well and good demos are there- than the project focus will come back to performance again. Maybe this will happen within about one year. If I get some help, from you and Ben , like I got here, from time to time - I hopefully will be there just in time!
--Andi
 

[Andi]

we do it not because it is easy, but because it is hard......  :)

[Linas Vepstas]

Hey,

If you want to port any algo to a GPU, or even make it run efficiently on a CPU, you had better understand what caches are, and how they work.  This is not a factoid about cpu design that you can just ignore: its fairly central, as it is a bottleneck for many classes of computational problems.  (It is also the reason why many other, most other algos run really really really well on modern cpus, such as your cellphone.  Without caches, your cellhone would be about as fast as a PC from about 20 years ago.)

--linas

[Andi]

O.K. ty linas!

[Andi]

O.K. !

[Andi]

Conclusion to my original question if it makes sense to think about parallel hardware architectures to enhance the performane of an OpenCog system.

1. There are no basic rejections.

2. OpenCog cognitive algorithms would be better suited for MIMD parallelism than for SIMD parallelism.

3. RAM acces, caching are the bottlenecks

4. What you really want for OpenCog is a MIMD parallel chip, with a lot

of RAM, and special connects btw the processors' caches.....   This
would let you put OpenCog on embedded devices in a useful way, and
also build OpenCog-tailored supercomputers....  These would be
customized for OpenCog in the same sense that the current crop of
"deep learning chips" are customized for hierarchical NNs.

5. Today the data structure of the atom space is like a rizome.
The atoms are scattered randomly through RAM. This can lead to a very high cache-miss rate.

6. It could be of advantage to put some associativity to the atomspace with the aim to make it possible to build clusters within which any given search is guaranteed to stay in just this cluster. This would maybe allow to load this cluster even into a GPU without causing to much cache-misses.


So in the moment, for my self, i guess that amoung existig designs a SPARK T5 could maybe do it better than a XENON PHI. Maybe even a chip with many original 8086 kernels without any caching and runing good old Borland turbo prolog on them could be the winner.

--Andi

[Andi]

Linas,
sleeping over this, I decided not to try to become a cache specialist but to continue with what I am really good - what is my special gift. This is to be able to look at very complex systems and to find small missing things to make them work better. During the most succesful period of my work life, I acted as a trouble shooter to minimise failures in huge fully automated production systems.

But you can be sure that if I will reach this point, I will remember what you told about the caching problem and to ask the experts to be conscious about it or to ask you to join the discussion.

And I can tell you that up to now, I never had the pleasure to see such a complex man made system like OpenCog and OpenCogPrime including the link grammar system. To me it seems that  just biological systems are more complex. So I feel like to be in paradise!
To me the OpenCog system is very beautiful! But also many of the more unconscious parts of my mind constantly sends messages that I can do many things to even make it more beautifull and that I should try to do it.

--Andi