Error in CostFunction Documentation?

[raf...@dotproduct3d.com]

Hello,

it appears that either there is an error in the docs for CostFunction or I am misunderstanding CostFunction.

From the cost_function.h header (wording in http://ceres-solver.org/nnls_modeling.html#costfunction is similar), important part in bold:

"Jacobian blocks are in the same order as parameter_block_sizes, i.e. jacobians[i], is an array that contains num_residuals_* parameter_block_sizes_[i] elements."

In my understanding this is only correct if a Manifold is used that has AmbientSize() == TangentSize(), such as the EuclideanManifold, or no manifold at all.

In other words, I would expect each jacobian block to be of size num_residuals * TangentSize() of the corresponding Manifold, which is generally different to parameter_block_sizes_[i].

I would further expect that if Manifolds are used the parameter_block_sizes_ member of CostFunction does not matter for the structure of the provided jacobians, except that its size() tells us how many jacobian blocks we have.

Is that understanding correct?
If no, what am I missing and, more importantly, how would I provide suitably sized jacobians inside CostFunction::Evaluate()?

Thanks!

[Sameer Agarwal]

The jacobians that cost function computer are always in the ambient space. The jacobians computed by the manifold is composed with the jacobian computed by the cost function by Ceres.

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/cc13435a-ce18-42a2-b4d4-f7196108cbd0n%40googlegroups.com.

[raf...@dotproduct3d.com]

OK.

Let's say I am using a manifold and I know how to linearize my manifold at a point X to get its Jacobian in tangent space (of size num_residuals * TangentSize()).
How do I plug that Jacobian into what CostFunction::Evaluate() provides?

[Sameer Agarwal]

Sorry but that's not possible.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/aba4f342-5795-4f47-a32e-40d062bc7dc3n%40googlegroups.com.

[raf...@dotproduct3d.com]

So, is it possible at all to use manifolds with AmbientSize() > TangentSize() without AutoDiffCostFunction?

Looking into the VINS-Mono code (which uses a quaternion-based manifold of tangent size == 6, ambient size == 7), it looks like they are simply setting the last column to zero:

https://github.com/HKUST-Aerial-Robotics/VINS-Mobile/blob/ab1c4ffb8d3ea1fcc01c4f5c651abc0c390807f9/VINS_ios/perspective_factor.cpp#L52

Are they doing it wrong?

Thanks

[Sameer Agarwal]

Yes they are doing it wrong.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/91e02ee4-8d8f-4a66-befd-138bad9e134bn%40googlegroups.com.

[raf...@dotproduct3d.com]

OK thanks.

I understand the purpose of using manifolds is to have better jacobians that live in minimal-dof tangent space. Yet there seems to be no way I can actually make use of those jacobians in Ceres. What am I missing?

Not trolling, just trying to figure out how I can make Ceres work for my use case.

[Sameer Agarwal]

have you read any of the example code? bundle_adjuster.cc shows how they are used. so do other bits of example code.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/4b612562-b67e-462f-b6d6-733ad2ab3a7en%40googlegroups.com.

[raf...@dotproduct3d.com]

Yes, I did.

bundle_adjuster.cc is using SnavelyReprojectionErrorWithQuaternions which is a AutoDiffCostFunction under the hood. It only implements the forward reprojection and doesn't deal with Jacobians at all. Hence my question about how it is possible without AutoDiffCostFunction.

[raf...@dotproduct3d.com]

Just checked all the other examples that use manifolds. All of them use AutoDiffCostFunction. So my question is still the same.

[Sameer Agarwal]

Manifolds in all interesting cases (except for Euclidean Manifold) have ambient dimension greater than tangent dimension.

The AutoDiffCostfunction is just a wrapper around CostFunction, it obeys the same contract. CostFunctions are responsible for computing the Jacobian by whatever means you like, automatic differentiation, numeric differentiation, or analytic derivatives that you compute yourself in the ambient space.

The Manifold gives the projection from the ambient space to tangent space and Ceres will internally compose the two. 

Does this answer your question?

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/58198679-3156-45d4-bd8b-72a4cc0d4902n%40googlegroups.com.

[raf...@dotproduct3d.com]

Thanks. We're getting closer.

What I'm trying to do:
The SE(3) group has a very simple and elegant jacobian for point motion w.r.t. pose motion. It's literally just a 3x6 matrix that looks like this
J_pose = (I_3x3 , -1 * p-hat)
where p-hat is the cross product matrix of the 3D point in camera coordinates.

SE(3) is also very stable under interpolation and offers better convergence vs. Quaternion-Euclidean or axis-angle-euclidean assemblys esp. with poor initial values for the problem at hand.

The jacobian is 3x6 so it is in tangent space.

My issue is that I'd like to make use of that fine jacobian but I don't see how.

As I see it, It would not make a lot sense for me (or an AutoDiffCostFunction) to compute Jacobians in ambient space since they would have none of those fine properties.

I also do not see how it would be possible for the Manifold to take my full ambient-space-Jacobian, perform some calculations on it and get an (elegant) tangent-space-y Jacobian as a result.
This is because J_pose may only be a part of the entire Jacobian chain. The whole reprojection function might look like this

p_repr = camIntr(proj(pose * p_3d))

so the full jacobian chain would be J_camIntr * J_proj * J_pose thruogh the chain rule.

Even if I (or an AutoDiffCostFunction) came up with a full ambient-space Jacobian of size 2x12 (12 because 3x3 rotation matrix + a 3vector for translation), it would be entangled with the jacobians of camIntr() (2x2) and proj() (2x3) through the chain rule. And Ceres would not know anything about camIntr() and proj(), so no way to disentangle either.

I guess my understanding of the purpose of ceres::Manifolds is wrong then. I thought the purpose was to leverage their elegant tangent space jacobians but it seems that's not the case, at least not directly (it would totally be possible if my original assumptions of Jacobian sizes in CostFunction::Evaluate() were true but that's not the case).

For a more concrete example, take SnavelyReprojectionErrorWithQuaternions:
The operator() is templated and a function of several T*s which means it'll be used with ceres::Jets for autodiff.
Inside operator() it is doing the entire camera transform and projection chain, which means that the Jets will diff through the entire thing and produce an ambient-space jacobian that will (by definition) not be as good as if the quaternion manifold's native tangent-space jacobian had been used for the pose transformation part.

So TLDR it looks like what you're saying is that Manifolds somehow take an ugly ambient-space jacobian, perform some tricks on it and output something better that's tangent-space-y.

And I don't see how that is possible or even desirable.

Sorry, long email. Does this make sense?

[raf...@dotproduct3d.com]

Actually, I think I get it now. This is where the manifold's PlusJacobian() comes into play.

It actually is possible to transform the ambient-space jacobian to a tangent-space jacobian. You just right-multiply the ambient-space jacobian with the manifold's PlusJacobian() since the "critical" jacobian to be transformed is always the right-most in the chain.

In the example:

J_camIntr * J_proj * J_pose * J_plus = J_tangent

(2x2)          * (2x3)  *    (3x12)  *  (12x6) =  (2x6)

OK, seems like that's it. Probably not super efficient though because the tangent-space matrix (J_pose * J_plus) often is much faster computed analytically instead of derived from the ambient-space jacobian.

Let me know if I got this correct.

Thanks again for your help.

[Sameer Agarwal]

Yes, thinking about the Jacobian of the Plus operator is the right way to go about this.

You will find that efficiency is generally speaking not a problem, the bulk of your time is spent in the linear solver.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/3c9802d0-2f09-4cd3-901e-a18ae66d5153n%40googlegroups.com.

[raf...@dotproduct3d.com]

OK.

This also explains why VINS-Mono gets away with their trick of just storing the tangent-space jacobian in the larger ambient jacobinan matrix:

They use a simple identity matrix (padded with zeros) as their PlusJacobian().

[raf...@dotproduct3d.com]

For future readers:

I was made aware of this thread on github, which nails it:

https://github.com/ceres-solver/ceres-solver/issues/615