torch.apply over rows, not just single elements.

[Daniel Galvez]

torch7 has a function called torch.apply which applies a function to every element in a tensor.

What I'd like is a similar function that applies a function to every row in a matrix. (Of course, it could be generalized so that the function could be applied along arbitrary dimensions in an n-tensor.) For example, if you input a 20 by 10 tensor, with the torch.norm function, this function would output a 20 by 1 tensor, with each element of the resulting tensor containing the norm of the corresponding row in the input tensor.

Basically, I'd like to avoid doing a for loop in lua code. theano has a function called scan for doing this sort of thing. I tried to look in the Convolution implementation in nn since I imagine it does some sort of batching to avoid a for loop in lua code, but I got lost quickly.

Does such a function exist in torch?

[Riddhiman Dasgupta]

So, basically, you're looking for the equivalent of MATLAB's bsxfun in Torch?

[Daniel Galvez]

No, not at all actually.

You an use torch.expand to get the same effect as matlab's bsxfun. That's one of the things I love about torch's concept of a tensor, since there's no memory copy to expand. Much cleaner than bsxfun.

In functional programming terms, I am curious if there is a function that can do "reductions" across dimensions so to speak. I'm realizing that the generality of what I'm asking for (I understand what I'm saying might be a bit vague) probably makes it unrealistic though.

[Riddhiman Dasgupta]

Sorry to hijack your thread like this, but how can you use torch.expand to get the same effect as bsxfun? :O

[Florian STRUB]

I do something like that:

      local applyRow = torch.Tensor.applyRow
      rawset(torch.Tensor,
             "applyRow",
             function(self, func)
                if self:nDimension() == 2 then
                   for i=1,self:size(1) do
                      local res = func((self[i]), i)
                      if res then
                         self[i] = res
                      end
                   end
                   return self
                else
                   error("Incorrect dimension")
                end
             end)
            
X = torch.Tensor(10, 5) 
          
torch.Tensor.applyRaw(X, function(x, i)
x:fill(i)
end ))

The method apply directly use the storage, so some optimization might be done.
Yet, it is better to think about this method as a tools rather than some optim.

PS the rawset enable to the method to use surrounding variables.

Kind regards,
STRUB Florian

[Daniel Galvez]

Thanks, Florian. I looked at the apply implementation and you're correct that it uses the storage. (Here, if anyone's curious.) I think you do get a speed up by using the storage because the storage has not concept of bounds checking. Anyway, I suppose what I wanted wasn't necessary since apply directly uses a lua for loop anyway. Fortunately, the for loop I wanted to avoid was not the limiting factor in my algorithm, as I have learned.

@Riddhiman: The best way to demonstrate is an example. Consider the forward propagation of a linear layer of a neural network. Your code would like this in torch:

local n    = 128 -- batch size                                                    
local f1 = 25 -- input feature size                                            
local f2 = 64 -- output (hidden) feature size                                  

local X    = torch.Tensor(f1, n) -- input features
local W    = torch.Tensor(f1, f2) -- weights
local b = torch.Tensor(f2) -- bias
-- initialize the above three tensors                                      
-- ...                                                                          

-- Here, you would use bsxfun in matlab to broadcast the addition of b
-- over the second dimension. (the batch dimension)                                          
local output = W:t() * X + torch.expand(b:resize(f2, 1), n)

[Florian STRUB]

Hi Daniel,
You are right about the storage/tensor access. It is also likely there are some Luajit optimizations in the apply method that we are not aware of.
However, it is still nice to have such helpers like applyRow. It increases code readability :)
If the loop becomes the limiting factor and the computation inside the loop is kind of easy (A lot of assumptions!!!), I would try the FFI library (with Tensor storage) : http://luajit.org/ext_ffi.html
To be honest, I have never use it myself since I have never needed it. Yet, it is still worth to have it it mind. This post provide some good insights https://groups.google.com/forum/#!searchin/torch7/ffi/torch7/eh7hbEIVJcI/UWH-v7VIcfEJ

Kind regards,
STRUB Florian