MVS Function with Ceres

[Chris Sweeney]

I recently saw THIS awesome upcoming work at ECCV for multiview stereo (based on THIS original paper) that utilizes a nice optimization framework. I immediately wondered if this couldn't be implemented with Ceres. Let's just consider the photometric consistency portion of the problem for simplicity.

Given images I and J, a pixel location x, a per-image projection function P_i(X) that projects a 3d point to a pixel location, and a per-image unprojection function that takes in a pixel and depth and outputs a 3D point Q_i(x, d). Let's also define x_ij to be the pixel location defined by P_j(P_i'(x, d)). The cost function for a given pixel and initial depth estimate is then:

C(x, d) = |J^t * G_j(x_ij) - G_i(x)|  where G_i is the gradient of image i at pixel x, and J is the jacobian of x_ij with respect to x. So the cost is the absolute difference of the gradients of the given pixel in image I and the pixel it reprojects to in image J for the given depth d, where the gradient of image J is warped by the jacobian to be in the coordinate system of image I.

Equations 3 and 4 from the "original" paper linked are the corresponding equations. 

Is it possible to write a cost function like this in Ceres? There are two hurdles I see:

1) Looking up and/or interpolating the image gradient values of the reprojected pixel. But maybe this is the use case for CostFunctionToFunctor? I haven't used it before.

2) Getting the 2x2 jacobian J. Is this possible in Ceres without explicitly computing it? It would be great to utilize the autodiff functionality of Ceres to obtain this jacobian somehow. That would allow for this MVS formulation to work for any type of camera projection without having to explicitly compute potentially cumbersome jacobians. Currently most stereo methods are only suited for perspective projection with all radial distortion effects removed. I would also be curious to see if this method can improve with various robust cost functions and other Ceres parameter tweaks.

-Chris

[Chris Sweeney]

Sorry, messed up my notation a bit. It should read:

x_ij =  P_j(Q_i(x, d))

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/257673be-5fc1-4ab9-97d3-d41e25ace22f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sameer Agarwal]

Comments in line:

On Thu, Sep 15, 2016 at 3:45 PM Chris Sweeney <kip...@gmail.com> wrote:

I recently saw THIS awesome upcoming work at ECCV for multiview stereo (based on THIS original paper) that utilizes a nice optimization framework. I immediately wondered if this couldn't be implemented with Ceres. Let's just consider the photometric consistency portion of the problem for simplicity.

Given images I and J, a pixel location x, a per-image projection function P_i(X) that projects a 3d point to a pixel location, and a per-image unprojection function that takes in a pixel and depth and outputs a 3D point Q_i(x, d). Let's also define x_ij to be the pixel location defined by P_j(P_i'(x, d)). The cost function for a given pixel and initial depth estimate is then:

C(x, d) = |J^t * G_j(x_ij) - G_i(x)|  where G_i is the gradient of image i at pixel x, and J is the jacobian of x_ij with respect to x. So the cost is the absolute difference of the gradients of the given pixel in image I and the pixel it reprojects to in image J for the given depth d, where the gradient of image J is warped by the jacobian to be in the coordinate system of image I.

Equations 3 and 4 from the "original" paper linked are the corresponding equations. 

Is it possible to write a cost function like this in Ceres? There are two hurdles I see:

1) Looking up and/or interpolating the image gradient values of the reprojected pixel. But maybe this is the use case for CostFunctionToFunctor? I haven't used it before.

This is what the BiCubicInterpolator class is for. 

2) Getting the 2x2 jacobian J. Is this possible in Ceres without explicitly computing it? It would be great to utilize the autodiff functionality of Ceres to obtain this jacobian somehow. That would allow for this MVS formulation to work for any type of camera projection without having to explicitly compute potentially cumbersome jacobians. Currently most stereo methods are only suited for perspective projection with all radial distortion effects removed. I would also be curious to see if this method can improve with various robust cost functions and other Ceres parameter tweaks.

If you have closed form expressions for P_i and Q_i, sure, if you don't, then inverse function theorem is your friend.

See answer 6 in the Modeling part of the FAQ

http://ceres-solver.org/faqs.html

for an example. 

Sameer

[Chris Sweeney]

Thanks, Sameer! Excited to try this out. I'll post back here if there are any "lessons learned" from implementing this.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CABqdRUDn6hRujgacgNCS%3DS_HmRTV1tbcdS9DfHkeNQBQQ%2BHqUw%40mail.gmail.com.

[Chris Sweeney]

Sameer's suggestion was to have a top-level analytic cost function and a separate cost function that defines the projections. The jacobians used in the cost function may then be retrieved through the cost function interface easily. This is super convenient!

However, there seems to be one more hurdle. In the cost function, the jacobian term J (the one inside the cost function) is actually a function of the depth d we want to optimize. So in order to provide the analytic jacobian of the entire cost function (w.r.t the optimization variable d), I would need the entries of the 2x2 matrix J to be functions in terms of the unknown depth d. It seems that the Evaluate() interface for retrieving the Jacobian from a cost function will only give the double values of the evaluated jacobian when really here I would need the entries of J as Jets. 

Is it possible to retrieve the Jacobian in terms of templated/Jet variables in this way? I guess an alternative way of thinking of this is to ask if it is possible to retrieve 2nd order partial derivatives in some way through Ceres?

[Sameer Agarwal]

not right now.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CAB_h4O6GzRKeH9n1-bFWa2ow6AigDGfL_mfDUVgmHBAuRLqqQA%40mail.gmail.com.

[Chris Sweeney]

Ok good to know.

Alternatively I could use numeric differentiation at the top level cost function. Would you expect this to work reasonably for this kind of problem?

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CABqdRUCv-5n_E1tiwpBg1YX6cmZV1GL7UkdGYZ_31XUszG1zLA%40mail.gmail.com.

[Sameer Agarwal]

you can use numeric differentiation, but I have never been a fan of numeric differentiation.

Good place to start though.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CAB_h4O5WzG3nznMQpn0aBoB-RoCj8vcWRCzCFCdD7eT%2BMTZJ3A%40mail.gmail.com.