Adjusting camera pose by minimizing some photometric error function

[Xiaoqiang Zhang]

Hi,

I am a newbie for optimization and a student learning computer vision stuff. My current project is to estimate camera poses from images. 

I know that people usually minimize the reprojection error in Bundle Adjustment(B.A.). What I am trying to do is to use some photometric error function to estimated the poses. The error function I am trying now is the sum of color-variance of corresponding pixels of all images. 

For example, if the camera intrinsic parameters are known. With two camera poses, the pixel correspondence can be found when the 'depth' of the corresponding 3d points is given. If I have n different views and take one as a reference view, for one pixel on the reference image, i can find its corresponding pixel on each of the other views. The error function is defined as the color variance of these pixels.  So when the camera poses are wrong, the error would be high as those pixels are corresponded to different 3d points.

It is different from B.A. in two ways:

    1. The jacobian is hard to derive, because color information is used, it may be impossible to get the analytical jacobian.

    2. The jacobian is not sparse. If i want to calculate the error for one pixel on the reference image, i need to find all of its  corresponding pixel on each of the other views, compute their mean and then the variance.

From the paper '3D Reconstruction from Accidental Motion' in CVPR this year, I know ceres-solver for the first time. It is excellent! I have been reading its documents on the website and try to use it in my project. 

My question are 

1. Can the 'automatic derivatives' in ceres get the jacobian of my error function? 

2. In this dense case, is it doable to find the solution?

Thank you all for reading, any advice or comment is welcome.

Xiaoqiang

[Chris Sweeney]

Hi Xiaoqiang,

1) Optimizing a photometric error as you describe is a super hard problem and is almost certainly not feasible when you have more than a few cameras. You need extremely good initializations for your camera poses. In the context of SfM, we usually create a reconstruction from sparse image correspondences, optimize it with bundle adjustment, then use multiview stereo to obtain more photometrically-consistent results. To understand a bit more of why this problem is hard I would check out this paper: "Accurate, Dense, and Robust Multi-View Stereopsis" by Furukawa et al in PAMI 2008.

2) However, this idea will work when processing a video sequence for SLAM if the baseline between video frames is small enough. Check out the paper  "LSD-SLAM: Large-Scale Direct Monocular SLAM"  by Engel et al at ECCV this year, which describes exactly how to do that. They achieve excellent results with a method very close to what you describe.

If anybody on this list is aware of large scale SfM work that directly optimizes photometric error I would be really interested in hearing about that!

Chris

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/cc7ce46f-fa39-438c-b964-943c616ade50%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andreu Corominas Murtra]

Hi Chris and Xiaoqiang,

In case you don't know it, I think the following paper could be of interest to you:

Newcombe, R.A., Lovegrove, S.J., Davison, A.J. DTAM: Dense tracking and mapping in real-time. International Conference on Computer Vision (ICCV). Novmber 2011.

It's not "large-scale" in terms of mapping big environments, but is a successful example of "dense" SfM, not based on point features. They won the best Demo award in the ICCV'11, it was awesome !

best,

 

andreu

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CAB_h4O7yMM75B2HvRmxbhkKQ4KbW%3DHoFq0hBfUpG7ygvezUQdQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

-- 
--------------------------------------------------
 Andreu Corominas Murtra
 PhD, Research Engineer
 Mobile Robotics & Artificial Perception
 www.beta-robots.com
--------------------------------------------------

[Xiaoqiang Zhang]

Hi, Chris

Thank you for your suggestion. Right now i only have 10~20 frames. However the baseline between each two frame is not small, like 10~30 centimeters. Is this a reasonable situation to start with?

As Andreu mentioned, DTAM is a very good SfM work that directly optimizes photometric error. I am trying to learn something from their work.

By the way, I visited your homepage and found your recent paper 'SfM with known relative rotation'. That is exactly what i wanted to do.  

在 2014年11月25日星期二UTC-7下午10时47分44秒，Chris Sweeney写道：

[Chris Sweeney]

In my experience, 10-30cm is probably too large. If you know some order of the images (i.e. if they are from a video) Then you can try to iteratively add one at a time and optimize after adding each frame. But trying to optimize them all at once will almost certainly require a good initial guess for the poses.

DTAM is indeed a great paper. It does require a GPU though and I don't believe source code is available publicly. LSD-SLAM  optimizes photometric error and has their code online, though I have not gone through it myself. Both papers are great starting points for learning more in this area

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/0c7103ed-6ddf-4b9e-afe7-f9959b49ae91%40googlegroups.com.

[Rui Yu]

Hi Xiaoqiang,

I think it would be probably worth having a look at http://code.google.com/p/photoconsistency-visual-odometry/ and the discussions here https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!searchin/ceres-solver/photoconsistency/ceres-solver/O-jk48z6qvU/cqcDRlYO8LcJ

The project combines automatic differentiation and numerical differentiation to get the derivatives. They are doing RGBD registration of two views by minimizing photometric error.

Rui

在 2014年11月26日星期三UTC上午4时00分49秒，Xiaoqiang Zhang写道：