Feature Request - Implement Local Winner Take All ( LWTA ) layer

[Prasanna S]

Ref: http://people.idsia.ch/~juergen/nips2013.pdf

This can be implemented with the current code of pooling layer.

I will give it a shot and update this thread with code if possible ( Am not a great coder, but will give it a try )

If any previous implementation is available, please let me know.

[Youssef Kashef]

Hey Prasanna,

would be very interested in an implementation of this for Caffe. Doing this as a pooling layer makes sense to me also.

I tried a quick hack ti implement WTA but in just a few iterations only one neuron would always win, so not very usable.

Do you know if there's an implementation of this in MATLAB or other, not specifically Caffe?

Please let me know if you'd like to collaborate on the implementation of this. Not too shabby when it comes to programming, but new to the Caffe codebase.

But no no worries if you prefer trying this out solo. Looking forward to a PR on this.

Good luck

Youssef

[Youssef Kashef]

This mentions a python implementation using Theano, but may need to contact the author for implementation details.

[Prasanna S]

Can you please explain a bit more when you said "Only one neuron would always win, so not very usable" ?

[Youssef Kashef]

My failed attempt at WTA in Caffe:

This was on the MNIST dataset. Unsupervised, so only using the images, and ignoring the ground truth info.

Given an image, the activations of the nodes in the output layer would not be identical due to random weight initialization. The WTA would pick the most active node. I want the winner to specialize in the pattern that let it win, while suppressing the others. The idea is that as this node specializes for one pattern it will have less activation for other patterns, where another node will win the competition. Eventually each node will specialize for a different pattern. You can think of it as a form of clustering.

What happened is that at first the winner of the WTA changed with different input. Makes sense. After some more iterations, node i would win more frequently and eventually always win. Whatever the pattern, only one node would always fire. It's as if that node learned the average of all patterns that led to its activation being higher than the activations of all other nodes. I checked the activations of the other nodes were pretty low.

This behavior didn't make much sense. There might have been an implementation error somewhere.

I also tried a soft-WTA. The activations would from a distribution from which the WTA would sample. This introduced stochastic noise into the WTA. But unfortunately, it didn't change the odd behavior.

I call this a hack because I didn't implement it as a pooling layer. I actually did this in the Conv layer. When initializing the label variable in the backward() and forward() methods, I ignored the actual label from the ground truth and set the label to be the argmax of the activations. This argmax was my hard WTA.

[Prasanna S]

I did very basic tests. Seems working. Also attached cu code

Kindly Do some more testing.

Merge the files of vision_layer.hpp and caffe.proto after seeing their diffs

[Prasanna S]

Did you change the caffe.proto file? I have added definitions to LWTAParameter in that file ( caffe_dir/src/caffe/proto/caffe.proto). Maybe I will clone caffe and just add these changes and maybe host it on github soon.

On Thu, 27 Aug 2015 at 20:10 Heechul Jung <jeongh...@gmail.com> wrote:

It occurs 'LWTAParameter' error as follows:

src/caffe/layers/lwta_layer.cpp:20:3: error: ‘LWTAParameter’ was not declared in this scope

   LWTAParameter lwta_param = this->layer_param_.lwta_param();

I think 'LWTAParameter' class is missing.

2015년 8월 22일 토요일 오전 1시 15분 26초 UTC+9, Prasanna S 님의 말:

I did very basic tests. Seems working. Also attached cu code

Kindly Do some more testing.

Merge the files of vision_layer.hpp and caffe.proto after seeing their diffs

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/A3F_NRlNxHM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/9c722df9-ab4b-466c-9976-12c0462f2033%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Дмитрий Мишкин]

Hi Prasanna,

I am going to use your implementation in my activations benchmark https://github.com/ducha-aiki/caffenet-benchmark/blob/master/Activations.md . How can I refer to it (your github profile, some repo, etc)?

четвер, 27 серпня 2015 р. 21:42:20 UTC+3 користувач Prasanna S написав:

[Prasanna S]

I havent put it on github. You can just use these files.
But I havent tested it completely. Do run your own tests.
And do lemme know if there were any errors.