Existing modules that induce sparsity?

[Hugh Perkins]

Hi,

I'm working my way through http://ufldl.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity .  Are there any existing torch modules that induce sparsity?  I want sparsity in weights, not just in activations, but I'm fine with starting with sparsity only in activations for now.

[Tushar N]

L1 regularization will give you activation sparsity. 

[Hugh Perkins]

Yes, that works great! Actually, I think that is weight sparsity,
right? Which is what I want in fact :-)

On Fri, Oct 2, 2015 at 4:55 PM, Tushar N via torch7
<torch7+APn2wQcFT9qhaE7xkO5UyQ6Z4...@googlegroups.com>
wrote:

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "torch7" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/torch7/2_qRc5Z6EYo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> torch7+un...@googlegroups.com.
> To post to this group, send email to tor...@googlegroups.com.
> Visit this group at http://groups.google.com/group/torch7.
> For more options, visit https://groups.google.com/d/optout.

[Hugh Perkins]

Oh, wait, I was looking at
https://github.com/torch/demos/blob/master/train-a-digit-classifier/train-on-mnist.lua#L221
, which is penalising non-sparse weights. And the link you gave me
penalises non-sparse activations, right? so, if I combine the two...

[Tushar N]

Yeah, the link in your post is general L1/L2 weight regularization.

(I still don't think of it as 'weight sparsity'-- it does keep weights small, but says nothing about sparsity)

If that's what you're looking for then that's great. Go ahead and utilize both penalties.

On Friday, 2 October 2015 16:09:32 UTC+5:30, Hugh Perkins wrote:

Oh, wait, I was looking at
https://github.com/torch/demos/blob/master/train-a-digit-classifier/train-on-mnist.lua#L221
, which is penalising non-sparse weights. And the link you gave me
penalises non-sparse activations, right?  so, if I combine the two...

On Fri, Oct 2, 2015 at 5:37 PM, Hugh Perkins <hughp...@gmail.com> wrote:
> Yes, that works great!  Actually, I think that is weight sparsity,
> right?  Which is what I want in fact :-)
>
> On Fri, Oct 2, 2015 at 4:55 PM, Tushar N via torch7

> <torch7+APn2wQcFT9qhaE7xkO5UyQ6Z4vs3tA1RzxaHVZIl_YTVF8DOiiZM70kbH@googlegroups.com>

[Florian STRUB]

I am aware of three kind of sparsity inside a network:
 - using L1 aka Lasso regularization -> minimize the number of weight
 - using ReLu that induce sparsity -> cancel the activation output
 - using dropout -> cancel the activation output for training

You can also use weigth matrix factorization. For instance, using  a k-rank weight matrix is equivalent of having a sparse weight matrix in the eighen space.

In the end, you are the only people that really know what tou need ;D

@Tushar L1 regularization does induce sparsity in convex problem. Minima correspond to 

Figure: ℓp ball. As the value of p decreases, the size of the corresponding ℓp space also decreases. This can be seen visually when comparing the the size of the spaces of signals, in three dimensions, for which the ℓp norm is less than or equal to one. The volume of these ℓp “balls” decreases with p. [2]

source: https://www.quora.com/What-is-the-difference-between-L1-and-L2-regularization

[Hugh Perkins]

On Tuesday, October 6, 2015 at 7:15:02 PM UTC+7, Florian STRUB wrote:

In the end, you are the only people that really know what tou need ;D

Thanks!  I kind of hit a wall for now ,since I re-remembered that most of the time in a convolutional network goes into the first few layers, where the number of planes is small.  Still, gradually moving forward.  Slowly, slowly :-P