Analytical Derivtives

[Alan Buchanan]

Hello Sameer,

I am trying to implement Analytical Derivatives (mainly to educate myself).

My example is using 3DVectors (from GPS static processing).

I notice that autodifferentiative while it produces a good solution doesn't generate a Jacobian that I was expecting which means I don't end up with a fully populated variance/covariance matrix.

i.e

I expected the jacobian to be made up of rows of    1 and -1  rather than just a row of 1  [Coordinate difference corrections rather than a single coordinate correction.

The covariance matrix ends up as a vector of variances without a y covariances.

The 3D vector example will also (at a later date) need to have extra parameters to solve for scale and the 3 rotations,

This is the case where the vector is in WGS84 and the Geoetic datum is in another reference frame.

My example cost function is below. My problem is the Jacobian is NULL when the Evaluate function is called. Where does the Jacobian get allocated its memory ?

using ceres::SizedCostFunction;

class GPSVectorCostFunction

: public SizedCostFunction<3,3,3> /* number of residuals  size of first parameter */ 

{

public:

virtual ~GPSVectorCostFunction() {}

virtual bool Evaluate(double const* const* parameters,

double* residuals,

double** jacobians) const {

double dx = parameters[1][0] - parameters[0][0];

double dy = parameters[1][1] - parameters[0][1];

double dz = parameters[1][2] - parameters[0][2];

residuals[0] = ((dx) - gpsvector[0]) *(pinv[0] + pinv[1] + pinv[2]);

residuals[1] = ((dy) - gpsvector[1]) *(pinv[3] + pinv[1] + pinv[4]);

residuals[2] = ((dz) - gpsvector[2]) *(pinv[5] + pinv[2] + pinv[4]);

// f(x) = 10 - x.

if (jacobians != NULL && jacobians[0] != NULL)

{

jacobians[0][0] = 1;

jacobians[0][3] = -1;

jacobians[1][1] = 1;

jacobians[1][4] = -1;

jacobians[2][2] = 1;

jacobians[2][5] = -1;

}

return true;

}

void AddObs(Vec3 gpsvect, double hii, double htt, Vec6 sdd, Vec6 pin)

{

gpsvector = gpsvect;

hi = hii;

ht = htt;

sd = sdd;

pinv = pin;

}

Vec3 gpsvector;

double hi;

double ht;

Vec6 sd;

Vec6 pinv;

};

[Sameer Agarwal]

Alan,

What is the code that you use to compute the automatically differentiated version of the above cost function?

Your Jacobian computation is wrong. You are ignoring the multiplication

residuals[0] = ((dx) - gpsvector[0]) *(pinv[0] + pinv[1] + pinv[2]); 

Sameer

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/b5d031df-bc20-4ee3-8816-d538ae3d9483%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alan Buchanan]

Sameer,

I don't understand the bug you identified . (pinv[0] + pinv[1] + pinv[2]) is a row from the variance/covariance matrix from the gps post processing. I am trying to weight my residuals.

This is my code for auto-differentiation. 

bool operator()(const T* const ip, const T* const fs, T* residuals) const

{

residuals[0] = ((fs[0] - ip[0]) - gpsvector[0]) *(pinv[0] + pinv[1] + pinv[2]);

residuals[1] = ((fs[1] - ip[1]) - gpsvector[1]) *(pinv[3] + pinv[4] + pinv[5]);

residuals[2] = ((fs[2] - ip[2]) - gpsvector[2]) *(pinv[6] + pinv[7] + pinv[8]);

return true;

}

 

Just to explain

1. gpsvector is the 3Dvector computed between ECEF points ip and fs.

2. pinv is a 3x3 variance/covariance that is generated by the static gps processing.

3. I am not worried about implementing rotations and scale between the gpsvector (in the satellite coordinate system) and the local geodetic network (ip &fs).

    This is not that important these days as most geodetic networks are derived from satellite datum observations (WGS84 or ITRF flavours).

I would still like to understand how to use Analytical derivatives as part of my education on Ceres and in case I need to use it later on.

- Explanation why my Analytical cost function doesn't have memory allocated for the Jacobians.

Thanks again for all your help.

Alan

[Sameer Agarwal]

Alan,

On Fri, Mar 31, 2017 at 2:37 AM Alan Buchanan <abuc...@gmail.com> wrote:

Sameer,

I don't understand the bug you identified . (pinv[0] + pinv[1] + pinv[2]) is a row from the variance/covariance matrix from the gps post processing. I am trying to weight my residuals.

Yes I understand that. But when you increase the weight of a residual, you also cause its derivative to be scaled by the same amount.

suppose you are minimizing 

1/2 * r(x)^T C^{-1} r(x) 

here r(x) is your unweighted residual and C is your covariance matrix.

The way you end up actually implementing this in ceres is by implementing a cost functor which computes

C^{-1/2} r(x)

in your cost functor the vector

fs[0] - ip[0]

fs[1] - ip[1]

fs[2] - ip[2]

is r(x)

and actually you may be using the inverse of the covariance matrix incorrectly too. You need the square root of the inverse not the inverse of the pseudo inverse.

I think your confusion stems from the fact that you think that the Jacobian should be of of the unweighted residual. This is not true. 

The residual that optimization algorithm sees is the weighted residual (as it should) so the derivative/jacobian has to account for the weighting too.

Which when you use automatic differentiation gets automatically accounted for.

This is my code for auto-differentiation. 

bool operator()(const T* const ip, const T* const fs, T* residuals) const

{

residuals[0] = ((fs[0] - ip[0]) - gpsvector[0]) *(pinv[0] + pinv[1] + pinv[2]);

residuals[1] = ((fs[1] - ip[1]) - gpsvector[1]) *(pinv[3] + pinv[4] + pinv[5]);

residuals[2] = ((fs[2] - ip[2]) - gpsvector[2]) *(pinv[6] + pinv[7] + pinv[8]);

return true;

}

 

Just to explain

1. gpsvector is the 3Dvector computed between ECEF points ip and fs.

2. pinv is a 3x3 variance/covariance that is generated by the static gps processing.

3. I am not worried about implementing rotations and scale between the gpsvector (in the satellite coordinate system) and the local geodetic network (ip &fs).

    This is not that important these days as most geodetic networks are derived from satellite datum observations (WGS84 or ITRF flavours).

I would still like to understand how to use Analytical derivatives as part of my education on Ceres and in case I need to use it later on.

- Explanation why my Analytical cost function doesn't have memory allocated for the Jacobians.

Because the cost function is only involved in evaluation, the part of ceres that calls the cost function will allocate the memory and call Evaluate on it.

Sameer

 

Thanks again for all your help.

Alan

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/5edacdc1-104b-465f-9921-2a00b1ae84d5%40googlegroups.com.

[Alan Buchanan]

Thanks For your reply.

So the reason my AnalyticalDerivative has not had the Jacobian array allocated any memory when I call evaluate is that the Cost Function has not been called earlier on in the Ceres pipeline ?

The only change I made was to the AddResidualBlock function.

Am I missing something (again) ?

void AddResidualBlock(ceres::Problem &prob)

{

// Analytical derivitive cost function

GPSVectorCostFunction *cf;

cf = new GPSVectorCostFunction;

cf->AddObs(gpsvector, hi,ht,sd, pinv);

ceres::CostFunction* cost_function = cf;

//  auto diff cost function

// ceres::CostFunction* cost_function = GPSVectorError::Create(gpsvector, sd,pinv,hi,ht);

prob.AddResidualBlock(cost_function, NULL, ip, fs);

}

[Sameer Agarwal]

On Fri, Mar 31, 2017 at 10:07 PM Alan Buchanan <abuc...@gmail.com> wrote:

Thanks For your reply.

So the reason my AnalyticalDerivative has not had the Jacobian array allocated any memory when I call evaluate is that the Cost Function has not been called earlier on in the Ceres pipeline ?

Ceres allocates the memory and calls Evaluate on it at various points of time in the execution of the optimization algorithm.

 

The only change I made was to the AddResidualBlock function.

Am I missing something (again) ?

void AddResidualBlock(ceres::Problem &prob)

{

// Analytical derivitive cost function

GPSVectorCostFunction *cf;

cf = new GPSVectorCostFunction;

cf->AddObs(gpsvector, hi,ht,sd, pinv);

ceres::CostFunction* cost_function = cf;

//  auto diff cost function

// ceres::CostFunction* cost_function = GPSVectorError::Create(gpsvector, sd,pinv,hi,ht);

prob.AddResidualBlock(cost_function, NULL, ip, fs);

}

A residual block is a combination of a cost function, loss function and parameter blocks. The act of adding it just adds the cost function to an indexing structure. It does nothing else. It is when you call solve, that ceres allocates and asks the CostFunction to put the jacobian in it.

Sameer

[Alan Buchanan]

Sameer,

Thanks for your assistance.

I haven't been able to get Analytical Derivatives to work but it isn't a priority for as I was just trying to see how they work.

I probably have a simple bug some where.

I will close this thread and revisit later on when my understanding is a bit better.  

[Srikanth S]

Hello,

I am facing a similar issue. I am solving a BA problem optimizing both structure and motion. The auto differentiation works well. But for the analytical differentiation, the jacobians pointer is always Null(in the Evaluate function) after the first iteration and hence I cannot supply the jacobians. Therefore it just computes the cost.

Wanted to check your ways to debug this issue. Whether the jacobians computed is itself wrong or is it due to something else.

Sameer,

Highly appreciate your assistance.

[Keir Mierle]

Hi Srikanth,

You may want to read the cost_function.h header to understand the contract between Ceres and your cost function implementation. If the jacobian pointer is NULL, then Ceres is requesting a cost-only evaluation without derivatives. Typically this means a cheaper calculation can be done (since no derivatives are needed).

Thanks,

Keir

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/b9dee06f-ff92-43e7-b0c5-9145de72c1c6%40googlegroups.com.

[Srikanth S]

Hello Keir,

Thanks for the reply.

I was wondering how does Ceres determine whether to request for derivatives or not(Determine whether a cheaper calculation can be done). Does it take the gradient calculated after the first iteration into consideration before deciding whether to request for derivatives.

For my case, when using the auto differentation, the gradient is computed and the cost decreases. But for the same dataset for the analytical case shown below. The gradient is not computed after the first iteration. So, I was wondering whether I can the relax the criteria for ceres to always request jacobians or is there something wrong in my implementation.

  iter      cost          cost_change     |gradient|   |step|    tr_ratio     tr_radius      ls_iter  iter_time  total_time

   0  4.039097e+06    0.00e+00    4.82e+06   0.00e+00   0.00e+00  1.00e+04        0    2.38e-03    3.16e-03

   1  4.294325e+06   -2.55e+05    0.00e+00   8.88e+01  -6.32e-02  5.00e+03        1    2.60e-02    2.92e-02

   2  4.294318e+06   -2.55e+05    0.00e+00   8.88e+01  -6.32e-02  1.25e+03        1    4.13e-02    7.06e-02

   3  4.294280e+06   -2.55e+05    0.00e+00   8.88e+01  -6.32e-02  1.56e+02        1    4.16e-02    1.12e-01

   4  4.293922e+06   -2.55e+05    0.00e+00   8.86e+01  -6.31e-02  9.77e+00        1    3.15e-02    1.44e-01

   5  4.287972e+06   -2.49e+05    0.00e+00   8.53e+01  -6.17e-02  3.05e-01        1    2.24e-02    1.66e-01

Thank you for your time.

Regards,

Srikanth

On Tuesday, October 10, 2017 at 3:26:23 AM UTC+5:30, Keir Mierle wrote:

Hi Srikanth,

You may want to read the cost_function.h header to understand the contract between Ceres and your cost function implementation. If the jacobian pointer is NULL, then Ceres is requesting a cost-only evaluation without derivatives. Typically this means a cheaper calculation can be done (since no derivatives are needed).

Thanks,

Keir

On Mon, Oct 9, 2017 at 11:15 AM, Srikanth S <sri...@gmail.com> wrote:

Hello,

I am facing a similar issue. I am solving a BA problem optimizing both structure and motion. The auto differentiation works well. But for the analytical differentiation, the jacobians pointer is always Null(in the Evaluate function) after the first iteration and hence I cannot supply the jacobians. Therefore it just computes the cost.

Wanted to check your ways to debug this issue. Whether the jacobians computed is itself wrong or is it due to something else.

Sameer,

Highly appreciate your assistance.

On Sunday, April 2, 2017 at 4:44:24 PM UTC+5:30, Alan Buchanan wrote:

Sameer,

Thanks for your assistance.

I haven't been able to get Analytical Derivatives to work but it isn't a priority for as I was just trying to see how they work.

I probably have a simple bug some where.

I will close this thread and revisit later on when my understanding is a bit better.  

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

[Sameer Agarwal]

If ceres computes an unsuccessful step - i.e. LM step does not make any progress, it can re-compute a new step without re-computing the derivative. In the above version you will notice none of the steps are making progress, so in every iteration it is trying to compute a new step for decreasing values of the LM parameter.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/5dfcfdcc-856e-46c1-9921-9802e9832e5b%40googlegroups.com.

[Keir Mierle]

If the autodiff version makes progress, but your analytical one does not, then it is probable that your derivatives are wrong. Take a look at the check_gradients option:

Solver::Options::check_gradients

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/5dfcfdcc-856e-46c1-9921-9802e9832e5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CABqdRUBz95UgJ9cTyjKC-_XSjZg_2Qbvj0PsW7Ui2zAF_umGKw%40mail.gmail.com.