Training on 3D Pointcloud or Voxel

[MSCderoe]

Hi,

is there a way to train a 3D CNN with pointclouds or voxels as input?
I`ve seen so much examples of how to train CNN on images but not a single one on pointclouds or voxels.
Or do I have to choose a different framework for that?

Thanks

[Przemek D]

The very idea of a convolution relies on a structure of data. In images, neighboring pixels form a grid over which you can iterate and which contains context information. In an XYZ point cloud there is no such relation - you just have a list of points with (generally) no structure in it. Just finding neighboring points in the cloud (something we take for granted when processing images) is a problem with dedicated algorithms to solve it.
Voxels are a different thing and sure you can do 3D convolution with it. Take a look at Maturana & Scherer (2015) for example - they convert a point cloud to voxel model via occupancy grids and convolve over it normally. 3D convolution was also used for computer tomography scan segmentation (as convolution over a 3D image) and pose estimation (3D "images" made of sequences of images - 3rd dimension is actually time). I'm sure you can find more examples.
Caffe supports 3D convolution - you can start reading how to accomplish it here.

[Przemek D]

Also, you might be interested in PointNet - although I don't know if this is achievable in Caffe.

W dniu niedziela, 29 października 2017 13:52:53 UTC+1 użytkownik MSCderoe napisał:

[MSCderoe]

Hi Przemek D,

thanks for your answers.
I read about VoxNet and PointNet and some other approaches but they are based on Theano and Tensorflow. So I wonder if it`s possible with caffe too or if I have to use one of the other frameworks.
I think there is a possibility to train on hdf5 files. So maybe this could be a way to do it.
The pointclouds I get from a Lidar so the biggest challenge is to tranform the raw lidar data into hdf5.
Do you think it makes sense to do it this way?