Let's define a new criterion: Weighted MSE loss function

[Rein Y. Wu]

Hi Everyone,

I hope to implement a criterion function that performing the weighted MSE. This criterion will be very useful when we solving some multi-output regression problems in deep learning. I know there are some existed functions such as MultiCriterion that used to compute the 'weighted' stuffs. However, MultiCriterion assign same weights to all the samples. I hope to assign different weights to different variables in different samples.  For example:

1. The Target of Sample 1 includes 3 variables: [a1, b1, c1]. The corresponding output of our DNN is: [a01, b01, c01]. The Weighted MSE should be: 

             1/3*((w11*(a1-a01)^2 + w12*(b1-b01)^2 + w13*(c1-c01)^2). 

2.  The Target of Sample 2 also have 3 variables: [a2, b2, c2]. The corresponding output of the DNN is: [a02, b02, c02]. The Weighted MSE should be: 

             1/3*((w21*(a2-a02)^2 + w22*(b2-b02)^2 + w23*(c2-c02)^2). 

We assigned different weights [w11,w12,w13];[w21,w22,w23] to the MSE of different samples (1 and 2). Is there a way to implement this criterion ?

I have an idea but I'm not sure whether it could work:

Could we do something like this ?

function Weighted_MSECriterion:updateOutput(input, target)

   self.output = 0

   for i=1,#self.criterions do

      self.output = self.output + self.weights[i]*(input[i] - target[i])^2

   end

   return self.output

end

But, I carefully read the existed files and not sure whether we could do something like 'input[i]' when define a criterion function. They always transfer the whole 'input' and 'target' into a *.c file to compute the criterion without explicitly indexing any elements in 'input' like 'input[i]'. Could we use input[i] and target[i] in Weighted_MSECriterion:updateOutput ? Why the original developers do not use this simple expression to compute MSE ?

Here is an example: https://groups.google.com/forum/#!topic/torch7/TliM2KO2ef0. I am not sure whether this guy success or not at last.

Thank you so much !

[Francisco Vitor Suzano Massa]

Yes, it's possible (and quite common) to implement modules/criterion in Lua only. Have a look at https://github.com/torch/nn/blob/master/BCECriterion.lua for example (which also has a weighting scheme)

Here is a simple implementation of what you are trying to achieve (not the most efficient, but you don't need to do a for loop in lua, just Tensor operations are enough)

function WeightedMSE:updateOutput(input,target)
  self.buffer = self.buffer or input.new()
  self.buffer:add(input,-1,target):pow(2) -- (input - target)^2
  self.buffer:cmul(self.weight)
  self.output = self.buffer:sum()
  return self.output
end

I just wrote the code on the fly, so there might be small errors. anyway, that's a more efficient way of doing what you want without for loops (much more efficient in GPU). But for loops work as well.

[Rein Y. Wu]

Thank you so much ! I just found this criterion has been defined here: 

https://github.com/torch/nn/blob/master/WeightedMSECriterion.lua