Analytic Jacobians of state w.r.t. Quaternion global vs. local size

[Simon Lynen]

Hi Sameer,

I use Quaternions in JPL convention[1] and provide an analytic Jacobian to Ceres. I have implemented the local parametrization for this convention, and when using Autodiff I get the correct results with my Jacobian/Residuals. However for the analytic Jacobians it seems I am doing something wrong, as the solver takes unsuccessful steps. We use the same analytic Jacobian in an EKF framework and I double checked it's correctness.

I pass the global size (4) of this parameter block to SizedCostFunction when deriving from it. As a consequence Ceres gives me a 2x4 Jacobian for this state. Since the Jacobian is only of size 3x2 () I set the fourth column to zero. Apparently Ceres is evaluating the fourth value, since if left unset it leads Ceres to emit a "numerical failure".

What am I missing, and what is the correct way of filling this Jacobian?

Thank you, Simon

[1] N. Trawny and S. I. Roumeliotis. Indirect Kalman Filter for 3D Attitude Estimation. University of Minnesota, Dept. of Comp. Sci. & Eng., Tech. Rep. 2005-002, March 2005.

[Sameer Agarwal]

Hi Simon,

Let me try and understand the problem better.

You have a residual function r(q) where q is your quarternion, where the quaternion is in JPl convention.

Now you are going to need two things here. The jacobian of r w.r.t q and the local parameterization of q.

Each of which can be implemented using autodiff or analytically. Which of these jacobians are you having trouble with? can you also show me the code fragment that you are using?

Thanks,

Sameer

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/787d1fdb-6ded-401a-afb4-a5a65ebebcfc%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Simon Lynen]

Hi Sameer,

thanks for the instant reply.

Yes we have a high dimensional state where the residuals are a function of (multiple) quaternions. However for the sake of simplicity we can just use an example from this paper: http://www-users.cs.umn.edu/~trawny/Publications/Quaternions_3D.pdf

The paper is about attitude estimation using an EKF and a sun sensor as measurement.

So lets assume our residual is formed by eq. 222: Where we have a direction measurement from a sun-sensor and our current orientation estimate to form the residual (eq. 228). The Jacobian of the measurement is the measurement Matrix H (eq. 232) This Matrix has 3 rows and 6 columns. (the state to be updated is the 3d orientation, where the full state in this example also includes an 3 element gyro bias).

Similarly I want to update an orientation estimate in Ceres using a 2D point measurement (simple BundleAdjustment). The Jacobian of this residual w.r.t. the state thus has 3x2 elements. 

Because I templated SizedCostFunction with 4 (as the global size) I get this "too large" (4x2) Jacobian to fill in. This is the actual question which I was interested in: What does Ceres expect in the Jacobian entries along the null directions of the cost?

Simon 

[Simon Lynen]

Sorry, forgot to answer the other point:

- The Jacobian for O-Plus (in the local parameterization) w.r.t. q is fine as it works fine when used by an Autodiff cost function.

- The Jacobian of the residual w.r.t. the state's quaternion is the one that the question is about.

It is hard for me to use Autodiff for the latter, since the residual calculation involves IMU constraints and functions which I can't template for Jet.

Simon

[Sameer Agarwal]

So lets assume our residual is formed by eq. 222: Where we have a direction measurement from a sun-sensor and our current orientation estimate to form the residual (eq. 228). The Jacobian of the measurement is the measurement Matrix H (eq. 232) This Matrix has 3 rows and 6 columns. (the state to be updated is the 3d orientation, where the full state in this example also includes an 3 element gyro bias).

Similarly I want to update an orientation estimate in Ceres using a 2D point measurement (simple BundleAdjustment). The Jacobian of this residual w.r.t. the state thus has 3x2 elements. 

Do you mean a 2x3 jacobian?

 

Because I templated SizedCostFunction with 4 (as the global size) I get this "too large" (4x2) Jacobian to fill in. This is the actual question which I was interested in: What does Ceres expect in the Jacobian entries along the null directions of the cost?

I see the issue here.

Lets assume you have a residual function r(q) where r is a vector of size 2. Then the cost function should compute a 2 x 4 jacobian, i.e. the derivative of each residual w.r.t all four of the components of the quaternion. It is the job of the local parameterization to deal with the local tangent space. It will convert this jacobian from 2x4 to 2x3 by post multiplying it by a 4x3 matrix which is the jacobian of the Plus operation for your quaternion parameterization.

Hope this answers your question.

Sameer

 

Simon 

On Thursday, August 1, 2013 10:13:18 PM UTC-7, Sameer Agarwal wrote:

Hi Simon,

Let me try and understand the problem better.

You have a residual function r(q) where q is your quarternion, where the quaternion is in JPl convention.

Now you are going to need two things here. The jacobian of r w.r.t q and the local parameterization of q.

Each of which can be implemented using autodiff or analytically. Which of these jacobians are you having trouble with? can you also show me the code fragment that you are using?

Thanks,

Sameer

On Thu, Aug 1, 2013 at 9:25 PM, Simon Lynen <simon...@lynen.eu> wrote:

Hi Sameer,

I use Quaternions in JPL convention[1] and provide an analytic Jacobian to Ceres. I have implemented the local parametrization for this convention, and when using Autodiff I get the correct results with my Jacobian/Residuals. However for the analytic Jacobians it seems I am doing something wrong, as the solver takes unsuccessful steps. We use the same analytic Jacobian in an EKF framework and I double checked it's correctness.

I pass the global size (4) of this parameter block to SizedCostFunction when deriving from it. As a consequence Ceres gives me a 2x4 Jacobian for this state. Since the Jacobian is only of size 3x2 () I set the fourth column to zero. Apparently Ceres is evaluating the fourth value, since if left unset it leads Ceres to emit a "numerical failure".

What am I missing, and what is the correct way of filling this Jacobian?

Thank you, Simon

[1] N. Trawny and S. I. Roumeliotis. Indirect Kalman Filter for 3D Attitude Estimation. University of Minnesota, Dept. of Comp. Sci. & Eng., Tech. Rep. 2005-002, March 2005.

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/787d1fdb-6ded-401a-afb4-a5a65ebebcfc%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/e93b0e7a-9094-4182-9c7f-32595e4cfab0%40googlegroups.com.

[Simon Lynen]

Hi Sameer,

OK, the insight on how Ceres applies the local parameterization helps to understand why I get a 2x4 passed.

Thanks

Simon

[Keir Mierle]

Zombie thread strikes--

On Thu, Aug 1, 2013 at 10:40 PM, Simon Lynen <simon...@lynen.eu> wrote:

Sorry, forgot to answer the other point:

- The Jacobian for O-Plus (in the local parameterization) w.r.t. q is fine as it works fine when used by an Autodiff cost function.

- The Jacobian of the residual w.r.t. the state's quaternion is the one that the question is about.

It is hard for me to use Autodiff for the latter, since the residual calculation involves IMU constraints and functions which I can't template for Jet.

Note that there is a nice way to mix numeric and autodiff:

https://ceres-solver.googlesource.com/ceres-solver/+/master/include/ceres/cost_function_to_functor.h

All you have to do is write a CostFunction that implements analytic or numeric diff for the operation; then you wrap that with a CostFunctionToFunctor, and call that inside an autodiff context.

Thanks,

Keir

 

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/78ee8fc8-4132-4af7-bdd6-bdf31c5e728b%40googlegroups.com.

[Nhon Trinh]

Hi Sameer,

I am new to ceres and have similar questions that Simon had here. It makes sense now from your answer. However, it sounds a bit a redundant that user needs to provide two large Jacobian matrices (2 x 4 and 4 x 3) while the solver only needs their product (2x3). In many cases, computing the final Jacobian matrix (2x3) directly is much simpler than computing the original two Jacobians that produce it, especially when the dimension of the tangent space is much smaller than the ambience space.

Is there a way that user only needs to provide the final Jacobian matrix and not the original two?

-- Nhon

[Sameer Agarwal]

Nhon,

This is the tradeoff between modularity and efficiency. We want the user to be able to optionally add a local parameterization to a parameter block and to be able to write CostFunction objects without being aware of the local parameterization. This means that ceres has to be responsible for combining the information between the cost function jacobians and the local parameterization jacobians.

I am curious as to why this is a concern? Is it a matter of performance or just elegance?

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/7c685a5d-4c6c-47aa-b332-6f953adc83bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nhon Trinh]

Sameer,

I haven't done any testing so I don't know how much this affects performance., but I suspect it will. In my particular case, I have existing codes that compute jacobians of the cost function with respect to the tangent space, not the ambient space, so I was trying to re-use those existing codes to set up the problem for ceres to solve. However, it looks like I'll need to rewrite my cost function (and make use of autodiff as much as possible).

The framework I was developing was that each object (a parameter block in ceres), e.g., rotation, position, is considered an element in a Lie group. As such, all operations are always in the tangent space. This is similar to the setup used in gtsam (http://borg.cc.gatech.edu/) developed by Frank Dellaert's group.

-- Nhon

[Sameer Agarwal]

Nhon,

I understand your pain about porting the code.

That said, is your concern mostly about porting or actual execution time? 

If it is about time, Ceres is careful about caching the jacobian computation. So the time penalty beyond a small matrix multiply should not be significant. If it is, I would like to know.

Ceres does not have the notion of typed parameter blocks, which prohibits us from combining the tangent space with the derivative computation. 

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/d63c54b2-2830-4385-b11d-a09f6cb25a92%40googlegroups.com.

[Nhon Trinh]

Sameer,

Both, actually. Porting is the immediate concern. However, with the help of autodiff, hopefully it won't be too bad.

As for execution, my concern is not so much on the need to multiply the two Jacobian matrices, but rather on the computation of the Jacobians themselves. For some of my cost functions, the Jacobian needs to computed numerically since they call third-party libraries and cannot be templatized. Increasing the dimension of the parameter space (tangent space --> ambient space) means that there will be more calls to the cost evaluation function, which could lead to increased computation time.

[Sameer Agarwal]

Nhon,

I am aware of the impedance  mismatch between local parameterizations and numerical differentiation. Unfortunately I do not have a good solution for this, I will think about this and see what I can come up with.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/3a87f930-ce7d-4160-a091-b172b484bb2e%40googlegroups.com.

[Nhon Trinh]

Hi Sameer,

Here's a thought I have: let J1 be the cost function jacobian, and J2 be the local param jacobian. As we discussed earlier, the solver is only interested in the product J3 = J1 * J2. I already have J3 available. Instead of computing J1 and J2 as they are defined, we can set J1 = [J3, 0] and J2 = [Identity, 0], then we preserve the product J1*J2, which is what the solver wants.

Do you think it will work? I don't know the ceres code enough to know whether this will lead to other problems.

-- Nhon

[Sameer Agarwal]

Nhon,

I think this will work if you have the Plus operation working correctly if you implement your own CostFunction object.

The hard part is to do this right using say NumericDiffCostFunction. The reason is that the NumericDiffCostFunction object is unaware of the local parameterization associated with the parameters and computes the derivatives (like the autodiffcostfunction) in the ambient space.

Making NumericDiffCostFunction be aware of the local parameterization inside ceres would require changing the interface of the basic CostFunction object.  This is non-trivial work and has implications for the API as well as the internals.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/80452d74-500d-401c-a15e-3302668ed267%40googlegroups.com.

[Jon Zubizarreta Gorostidi]

Hi,

Did the trick with jacobians work?

Thanks in advance,

Jon

[Matias N.]

Hi,

sorry to resurrect old thread but I'm facing exactly the same issue: I'm implementing equations for jacobians of cost function, which are already defined w.r.t. the tangent space. So, since ceres expects the intermediate Jacobian I need to compute the jacobian w.r.t. the ambient space, which is quite tedious and feels redundant since I already have the solution which in the end ceres expects (the product of both jacobians).

I'm considering going for the option suggested by Nhon Trinh above. However, I wanted to check with you if this is reasonable considering how ceres is implemented or would this be a "hack" that may break. In other words, by "faking" the parametrization jacobian this way, what else could be affected? Since this is for a SE(3) pose, where I would always request covariance in local space, this "fake" jacobian would not be used there either, right?

Best,

Matias

[Sameer Agarwal]

as long as the definition of your plus operator is consistent with your local parameter jacobian you should be fine.

sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/4170ed76-6ff9-442f-88cb-32fe5cf5b076%40googlegroups.com.

[Matias N.]

Thanks for the response.

By the way, I added a comment to the conversation in https://github.com/ceres-solver/ceres-solver/issues/387

I wanted to know if it would be possible to add an option to directly use the jacobians computed by the cost function, assuming they are computed w.r.t. local space.

Best,

Matias

[Sameer Agarwal]

The CostFunction API the way its structured right now does not allow it.

Doing something like this would require a signifiant change to how the evaluator inside Ceres is structured and API changes too.

Happy to consider a proposal for it though.

Sameer

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/19067bc1-de1a-443b-9ef3-20615dc81886%40googlegroups.com.