camera pose gradient = 0, termination: CONVERGENCE

[Sundaresh R]

Hello list,

I am new to ceres. I am feeling very happy to work with this wonderful library. I am using ceres-solver to optimize camera pose in my project. I have a 2D image and corresponding 3D CAD model. I know the ground truth of the camera pose when the 2D image was acquired. I want the algorithm to converge to give back ground truth values.

I have single image and single calibrated camera. I am keeping all the parameters of the pose constant and just varying the z component of pose. I use some functions from OpenCV library. Could anyone of you please let me know if this definition is correct?

The error function terminates in the first iteration with gradient = 0 and cost change =0, termination: CONVERGENCE

Thanks in advance

//cost function
/Pose = [x y z alpha beta gamma]
struct MyCostFunctor{
    MyCostFunctor(cv::Mat _image, Model _model, cv::Mat _cameraMatrix, Pose _pose){

    m_image = _image;//2d image
    m_model = _model;//3d model
    m_cameraMatrix = _cameraMatrix;//intrinsic camera parameters
    m_Pose =_pose;//ground truth pose 

    }

         bool operator()(const double* const z, double* residual) const{

        double score{};
        cv::Mat _shiftedModel = rigidTransform(m_model.m_vertices, m_Pose, *z);//moves the 3d model in image space
        cv::Mat _projectedMat = projectTo2d(_shiftedModel, m_cameraMatrix);//projects 3d model onto image space
        cv::Mat _imagePatch = generateImagePatch(_shiftedModel, _projectedMat, m_model, drawing);//function to plot the projected model
        score = scoreFunction(m_image, _imagePatch);//compares the pixels in the projected image and 2d ground truth image

        residual[0] = 11191 - score;//11191 is the number of pixels belonging to the ground truth image, score is the number of pixels belonging to the projected image
        return true;

    }

    cv::Mat m_image;
    Model m_model;
    cv::Mat m_cameraMatrix;
    Pose m_Pose;
};

//Problem definition
// X0 = Pose = [x y z alpha beta gamma]
    ceres::Problem problem;
    ceres::CostFunction* costFunction = new NumericDiffCostFunction<MyCostFunctor, ceres::CENTRAL,1,1>(new MyCostFunctor(image, model, cameraMatrix, X0));

    double pose_z = 9.0;//9.79241;

    problem.AddResidualBlock(costFunction, NULL, &pose_z);

[Sameer Agarwal]

My guess is that the perturbation that NumericDiffCostFunction is using does not create a large enough difference, causing you to get a zero residual and gradient.

If I were to guess, your generateImagePatch function is generating an integer valued image, in which case it is likely truncating its values thereby destroying the perturbation.

Sameer

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/9fcc8eac-e3c8-4ba7-a614-6fc9296f3410%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sundaresh R]

Sameer,

Thanks a lot for the reply. Could you please shed some more light on this issue? Is there any other way around to deal with this?

[Sameer Agarwal]

Yes, do not truncate your image after projection onto integers. 

Basically for numeric differentiation to work, your function needs to be sensitive to small changes.

you can try this yourself, just evaluate the function for small varying values of z and see what happens.

if it does not change, that indicates a flat gradient to ceres and it thinks it has converged.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/900d3f7f-1a3d-4294-911a-fab8db8434f2%40googlegroups.com.

[Sundaresh R]

But, the function "generateImagePatch" is returning values as expected. It returns the model projection along with label to each pixel which is integer value. I intend to use this information in the scoreFunction to match the label of pixels in ground truth and projected image. This is my score function. Could you also please let me know if the problem definiton and cost function definition is correct or not?

Regards,

Sundaresh

[Sameer Agarwal]

Sundresh,

I cannot tell from reading this bit of code if the underlying implementation is correct or not. Also the issue is not just correctness. As integer valued function I am sure its fine. My point is, integer valued functions are not differentiable and appear like flat functions to an optimizer that depends on gradients.

To give you a simple example

let f(x)  = 255 * x.

and then you apply the floor function to it. floor(f(x)) is now an integer valued function but between its integer points it has a constant value, it has a derivative of zero everywhere. Thats roughly what you are doing.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/39caabed-45af-4843-98b6-dd9f5dffbdeb%40googlegroups.com.

[Sundaresh R]

Hi Sameer,

Thanks a lot for the quick reply. I am really sorry for bothering you with so many questions. I get what you said. But is there a way I can make this a differentiable function. After residual computation, the updated parameter value is used by the function "rigidTransform" to shift the model to new position. I notice that the change in residual is almost 0. I am not able to do anything because of this. It would be of great help if you can suggest what can be done in this situation.

Sundaresh

[Sameer Agarwal]

Sunderash,

Your problem is not the rigid transform function. It is most likely in generateImagePatch. The simplest thing to do is, is to not project onto integer valued pixel values.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/d80faf0f-961a-4850-b880-e81e94009e48%40googlegroups.com.

[Sundaresh R]

HI Sameer,

Thank you for the inputs earlier. I set the following options in Solver::Options:

options.gradient_check_numeric_derivative_relative_step_size = 1e-2;
    options.initial_trust_region_radius = 1e-1;
    options.min_relative_decrease = 1e-2;
    options.check_gradients = true;
    options.max_num_iterations = 100;
    options.linear_solver_type = ceres::SPARSE_NORMAL_CHOLESKY;

The correction for integer valued pixel values was taken care.

I now get the following ouptut with initial value of z = 8

Z : 8
Score : 296
Z : 8.00001
Score : 296
Z : 7.99999
Score : 296
Z : 8
Score : 296
Z : 8.08
Score : 469
Z : 7.92
Score : 185
iter      cost      cost_change  |gradient|   |step|    tr_ratio  tr_radius  ls_iter  iter_time  total_time
   0  5.935051e+07    0.00e+00    0.00e+00   0.00e+00   0.00e+00  1.00e-01        0    1.21e+00    1.21e+00
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0918 08:23:09.148187 25776 gradient_checking_cost_function.cc:144] Gradient error detected. Terminating solver.

Solver Summary (v 2.0.0-eigen-(3.2.92)-lapack-suitesparse-(4.4.6)-cxsparse-(3.1.4)-eigensparse-no_openmp)

                                     Original                  Reduced
Parameter blocks                            1                        1
Parameters                                  1                        1
Residual blocks                             1                        1
Residuals                                   1                        1

Minimizer                        TRUST_REGION

Sparse linear algebra library    SUITE_SPARSE
Trust region strategy     LEVENBERG_MARQUARDT

                                        Given                     Used
Linear solver          SPARSE_NORMAL_CHOLESKY   SPARSE_NORMAL_CHOLESKY
Threads                                     1                        1
Linear solver ordering              AUTOMATIC                        1

Cost:
Initial                          5.935051e+07

Minimizer iterations                        1
Successful steps                            1
Unsuccessful steps                          0

Time (in seconds):
Preprocessor                         0.000090

  Residual only evaluation           0.000000 (0)
  Jacobian & residual evaluation     1.208781 (1)
  Linear solver                      0.000000 (0)
Minimizer                            1.208880

Postprocessor                        0.000004
Total                                1.208974

Termination:                          FAILURE (
Gradient Error detected!
Extra info for this residual: Residual block id 0; depends on parameters [0x7ffc60b854f0]

Detected 1 bad Jacobian component(s). Worst relative error was 1775.

========== Jacobian for block 0: (1 by 1)) ==========
 block  row  col        user dx/dy    num diff dx/dy         abs error    relative error         parameter          residual
     0    0    0                 0             -1775              1775              1775                 8             10895 ------ (0,0,0) Relative error worse than 1e-08
)


Process finished with exit code 0.

Can this be ovecome  by any other alternate methods? I am currently using NumericDiffCostFunction

Regards,

Sundaresh

[Sameer Agarwal]

Sundaresh,

It looks like whatever you did to fix your gradients did not work.
Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/4eea266d-be12-439b-9c93-b9a980c3b7b6%40googlegroups.com.

[Sundaresh R]

Hi Sameer,

ceres::CostFunction* costFunction = new NumericDiffCostFunction<MyCostFunctor, ceres::RIDDERS,1,1>(new MyCostFunctor(image, model, cameraMatrix, X0), ceres::TAKE_OWNERSHIP);

This works. However, it does not work for all values of z.

Sundaresh

[Sameer Agarwal]

Sundaresh,

That just means your code compiles. It does not mean that the way you function behaves, it is smooth enough to be optimized. 

What the solver is complaining about is that it seems that the derivatives it is computing don't seem to be match the numeric differentiation.

you can just disable check_gradients and see how well the optimization does.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/7979b113-2d4c-4946-91f3-d23412157214%40googlegroups.com.

[Sundaresh R]

I disabled check_gradients.

Z : [8 ; 1]

iter      cost      cost_change  |gradient|   |step|    tr_ratio  tr_radius  ls_iter  iter_time  total_time

   0  5.935051e+07    0.00e+00    2.24e+07   0.00e+00   0.00e+00  1.00e-02        0    2.55e-01    2.55e-01
Z : 8.05249
Z : [8.05249 ; 1]
   1  5.813654e+07    1.21e+06    2.33e+07   5.25e-02   1.04e+00  3.00e-02        1    2.86e-01    5.41e-01
Z : 8.19754
Z : [8.19754 ; 1]
   2  5.296263e+07    5.17e+06    2.71e+07   1.45e-01   1.55e+00  9.00e-02        1    2.87e-01    8.28e-01
Z : 8.52084
Z : [8.52084 ; 1]
   3  3.848276e+07    1.45e+07    3.86e+07   3.23e-01   1.73e+00  2.70e-01        1    3.64e-01    1.19e+00
Z : 8.94448
Z : [8.94448 ; 1]
   4  2.239812e+07    1.61e+07    1.64e+07   4.24e-01   1.10e+00  8.10e-01        1    2.87e-01    1.48e+00
Z : 10.1704
   5  1.006658e+07    1.23e+07    0.00e+00   1.23e+00   7.92e-01  4.05e-01        1    3.56e-02    1.51e+00
Z : 9.73414
Z : [9.73414 ; 1]
   6  2.606420e+05    2.21e+07    1.36e+05   7.90e-01   2.00e+00  1.22e+00        1    4.32e-01    1.95e+00
Z : 7.62656
   7  6.015948e+07   -5.99e+07    0.00e+00   2.11e+00  -2.89e+02  6.08e-01        1    3.61e-02    1.98e+00
Z : 8.28211
   8  4.914370e+07   -4.89e+07    0.00e+00   1.45e+00  -3.06e+02  1.52e-01        1    3.67e-02    2.02e+00
Z : 9.22755
   9  1.316358e+07   -1.29e+07    0.00e+00   5.07e-01  -2.01e+02  1.90e-02        1    3.53e-02    2.05e+00
Z : 9.66256
Z : [9.66256 ; 1]
  10  1.230080e+05    1.38e+05    3.63e+05   7.16e-02   1.43e+01  5.70e-02        1    3.56e-01    2.41e+00
Z : 9.62605
Z : [9.62605 ; 1]
  11  0.000000e+00    1.23e+05    0.00e+00   3.65e-02   9.54e+00  1.71e-01        1    3.53e-01    2.76e+00

Solver Summary (v 2.0.0-eigen-(3.2.92)-lapack-suitesparse-(4.4.6)-cxsparse-(3.1.4)-eigensparse-no_openmp)

                                     Original                  Reduced
Parameter blocks                            1                        1
Parameters                                  1                        1
Residual blocks                             1                        1
Residuals                                   1                        1

Minimizer                        TRUST_REGION

Sparse linear algebra library    SUITE_SPARSE
Trust region strategy     LEVENBERG_MARQUARDT

                                        Given                     Used
Linear solver          SPARSE_NORMAL_CHOLESKY   SPARSE_NORMAL_CHOLESKY
Threads                                     1                        1
Linear solver ordering              AUTOMATIC                        1

Cost:
Initial                          5.935051e+07

Final                            0.000000e+00
Change                           5.935051e+07

Minimizer iterations                       12
Successful steps                            8
Unsuccessful steps                          4

Time (in seconds):
Preprocessor                         0.000076

  Residual only evaluation           0.393043 (11)
  Jacobian & residual evaluation     2.369687 (8)
  Linear solver                      0.000189 (11)
Minimizer                            2.763345

Postprocessor                        0.000008
Total                                2.763429

Termination:                      CONVERGENCE (Gradient tolerance reached. Gradient max norm: 0.000000e+00 <= 1.000000e-10)

Is solution usable: 1
Final x3: 9.62605

Process finished with exit code 0

But actual x3 should be 9.79241

[Sameer Agarwal]

your gradient is zero, solver cannot do any better. zero gradient indicates to the solver that you have reached a local minimum.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/82af249f-2990-4e16-aacd-4d2470f3d6ee%40googlegroups.com.

[Sundaresh R]

Sameer,

Thanks a lot for all your replies. Does this mean this solution is the best I can get?

How exactly to set the solver options :

*initial_trust_region_radius

*gradient_check_numeric_derivative_relative_step_size

*min_relative_decrease

Varying these parameters gives me different final solutions

Sundaresh

[Sameer Agarwal]

Sundaresh,

The default values of these parameters generally speaking should not require any change. if your objective function has large flat regions of zero derivative, these are just local minima that you should expect any solver to get stuck in. You maybe able to change the path of the solver a little bit by changing these parameters, but that fundamentally does not change the fact that if you have found a point with zero gradient, there is no getting out of it.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/4e3e7d79-d27d-4c14-82fe-98e4a5f1d81d%40googlegroups.com.

[Sundaresh R]

Sameer,

Thanks again for everything. Got a better overview.

Sundaresh