Handling Missing or Invalid Pixels in Imagery

[Guru Pradhan]

Hi All,

A warning: Noob to Caffe and machine learning in general. I am trying to extend the algorithm described here: http://mmlab.ie.cuhk.edu.hk/projects/SRCNN.html using satellite imagery and implement it in Python. As you know, clouds and instrument errors lead to missing pixels or pixels that have been filtered out (replaced with NaN or filler values). Can Caffe handle invalid pixels (NaN for some pixels) when carrying out convolutions or will it throw an error? I know the best method is to try it out for oneself but I figured I would ask first before diving in. Also, what are some common strategies to impute missing values in image data? 

Thanks! 

[Jan C Peters]

I am pretty sure that caffe uses ordinary additions and multiplications at the lowest level, and as such will propagate NaNs to the surrounding pixels during convolution. Which is probably not what you want. But what would you expect caffe to do? It is hard to come up with a good strategy for that case. Just leave those values out during convolution? Would run into all kinds of problems in the backprop pass and bias the result.

If those pixels are only few maybe you can fill them by interpolation. Or do the learning and prediction (of whatever value you are looking for) on (rectangular) patches of these images, cut out of regions with no missing pixels.

Jan

[Guru Pradhan]

Hey Jan, thanks for responding. I definitely do not want NaNs contaminating surrouding pixels at every convolutional layer. I was also thinking along the lines of cropping numerous sub-images or patches that contain only valid pixels. In this manner, I can:

1) Get rid of the NaN problem

2) Create a larger training set

Thanks, I will mark this question as complete.