Under determined system with cost function less sensitive to parameter changes

[Ajay Karthik]

Hello Ceres Team,

I recently started using ceres for solving the optimization problems. 

Unfortunately I have an under determined system, which I can imagine to be bad and hence I am planning to add more residual function, which could be of help.

Nevertheless I have one more issue. During the code debugging, I realized that my parameters have less impact on the cost function.

f(Param + d_param) has small impact only when d_param is chosen to be a higher value. Yes I realize that my starting value is not good. But I would like some kind of guidance on what could be done in order to achieve sensitivity of parameters to the respective residual computation.

Any suggestions are welcome.

Thank you,

Ajay Karthik.

[Sameer Agarwal]

Ajay,

I am not sure I understand your question.

Could you perhaps explain with an example what you want to achieve?

Sameer 

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/9a825c1e-8b57-4c57-86a3-77636ef9dc38n%40googlegroups.com.

[Ajay Karthik]

Hi Sameer,

 Problem Setup : 

Input/Parameter estimation :

    A 2D discretized grid, at each center point of the grid, I am estimating the parameter value (e.g. fluid velocity). I am ensuring that the parameters are normalized by setting the lower and upper bounds to be 0. and 1. respectively.

Structurally some of the grid points could have zero-velocity, meaning that it is not hollow at that point. So basically I am trying to cut a pattern on to a thin film structure.

Output/Cost Function : 

    I have a 1D sensor grid at the outlet for measurements. My measurement distribution mostly resembles a distribution, lets say an gaussian distribution. Nevertheless I am more interested in the slope of this distribution than the values itself. Hence I have a scalar output/cost function. Hence I said my system is under determined in this case. In the meantime, I am exploring what kind of other information, I could give the solver.

My results with this basic setup:

    Linear solver ordering              AUTOMATIC                        1

Cost:
Initial                          5.000000e-01
Final                            5.000000e-01
Change                           0.000000e+00

Minimizer iterations                        1
Successful steps                            1
Unsuccessful steps                          0
Line search steps                           0

Time (in seconds):
Preprocessor                         0.000359

  Residual only evaluation           0.000000 (0)
    Line search cost evaluation      0.000000
  Jacobian & residual evaluation     0.029389 (1)
    Line search gradient evaluation   0.000000
  Linear solver                      0.000000 (0)
  Line search polynomial minimization  0.000000
Minimizer                            0.031867

    Unfortunately I didn't understand the result.

     Initial Cost : 0.5
     Residual only evaluation : 0.0

Debugging : 

    I realised that, during debugging a small change is being added to the parameters. But this small change is not having a perceivable impact on the cost function. But when I manually increase this change value(d_parameter), then I can see a small impact on the cost function. Hence I mentioned in my previous post, that the parameters values might be little insensitive to the cost function. Hence I expected the jacobian value to be too low.

My solution technique : 

     I am trying to evaluate the continuous parameter values on the discretized grid. At the end have a threshold value, inorder have patterns on my 2D structure.

I am hoping that I was able to give at least a little bit insight onto the problem.

Any ideas and suggestions are welcome.

[Sameer Agarwal]

Are you using the cubic interpolator?

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/5582f6a4-ebf2-423e-b448-4701743ff160n%40googlegroups.com.

[Ajay Karthik]

Unfortunately I am unaware of a cubic interpolator. Why do you ask ? Just curious.

Thank you,

Ajay.

[Sameer Agarwal]

because if you are using bilinear interpolation (which I suspect you are) you are constructing a non-differentiable interpolation function, which will lead to the kind of problems you are seeing.

Have a look at http://www.ceres-solver.org/nnls_modeling.html?highlight=evaluate

and

https://github.com/ceres-solver/ceres-solver/blob/master/include/ceres/cubic_interpolation.h

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/181acaaf-d94c-41c4-9fbd-4c7f005a4018n%40googlegroups.com.

[Ajay Karthik]

Hi Sameer,

Thank you for the insights. Unfortunately I am not using any kind of interpolation in the slope estimation(cost function). But nevertheless may be my cost function itself is not differentiable since I am using the DynamicNumericDiffCostFunction.

I will have an in-depth look into my problem setup and will get back to you in case some more help is required.

Thank you,

Ajay.

[Ajay Karthik]

Hi Sameer,

Here I am trying to log the parameter and respective residual values.
parameters : 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
residual : 0.852419
parameters : 0.5 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
residual : 0.873536
parameters : 0.0999999 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
residual : 0.852419
parameters : 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
residual : 0.852419
parameters : 0.1 0.0999999 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
residual : 0.852419
parameters : 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
residual : 0.852419

 Terminating: Function tolerance reached. |cost_change|/cost: 0.000000e+00 <= 1.000000e-06

From these values, I personally understand the residual change is less sensitive to the change infinitesimal changes of parameter values. The parameter value in red, is changed manually from my side and hence I also saw a change in the residual. I think, since ceres is only adjusting my parameter values with an  infinitesimal amount,  and  hence my gradient computation is 0.0 during the first step itself. 

Any remarks on my understanding and may be how I can improve the correlation between the parameter values and the cost function are welcome.

Sorry for all the trouble, I am giving you and thank you in advance.

Thank you,

Ajay Karthik.

[Sameer Agarwal]

Ajay,

I suspect your function is piecewise constant. Can you describe a simple small mathematical version of what you are trying to do and first debug there?

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/f49db4ba-98f1-4528-b110-0de5a669252bn%40googlegroups.com.

[Ajay Karthik]

Hi Sameer,

Yes it might be piecewise constant function. I need to check about it, since my cost function is a complete black box for me. As of now, I am working with this scenario:

Honestly I don’t know if I could say it easily if the function is piecewise constant or not. Nevertheless I am looking into exact gradient calculation as one of the other cost functions.

Any insights are welcome.

Thank you,

Ajay.

[Sameer Agarwal]

if you have a function f(x) defined on a grid say [0,...,n-1].

If when evaluating it on a real value x, you truncate/round to an integer and look up its value in the grid then its piecewise constant.

if you take two adjacent value by taking its floor and ceil of x and then linearly interpolate these values of the interval then you have a piecewise linear function and so on.

you should just be able to check this by evaluating the function on the interior of a cell and see its behaviour.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/4f08251d-d676-4ca1-8229-881657cb3349n%40googlegroups.com.

[Ajay Karthik]

Hello Sameer,

Sorry for the super late response. But I was on a vacation and wasn't able to really check on it.

I found that my cost functor isn't a piecewise constant function. It is just that the variation in the parameter values is too small, that the cost function value is not changing.

Just to reiterate, the problem setup diagram is just posted in my last post on this thread.

Please have a look at the problem formulation, and any suggestions are welcome.

void Solve()
{
   using namespace ceres;
   Problem problem;
   
   auto pixelCount = GetGUIParam("Pixel Count");
   auto cost_function = new DynamicNumericDiffCostFunction<SlopeResidual, ceres::CENTRAL>(new SlopeResidual());
   for (unsigned int parameterBlockIndex = 0; parameterBlockIndex < pixelCount; ++parameterBlockIndex)
   {
      cost_function->AddParameterBlock(1);
   }
   cost_function->SetNumResiduals(1);

   std::vector<double> pixelValues = InitPixelValues(pixelCount); //-as of now random initialization  
   std::vector<double*> parameter_blocks;
   parameter_blocks.reserve(pixelCount);
   for (auto& pixelValue : pixelValues)
   {
      parameter_blocks.push_back(&pixelValue);
   }
   problem.AddResidualBlock(cost_function, nullptr, parameter_blocks);
   for (auto blockIndex = 0; blockIndex < pixelCount; ++blockIndex)
   {
      problem.SetParameterLowerBound(parameter_blocks[blockIndex], 0, 0.);
      problem.SetParameterUpperBound(parameter_blocks[blockIndex], 0, 1.);
   }

   Solver::Options options;
   options.max_num_iterations = 100;
   options.linear_solver_type = DENSE_QR;
   options.minimizer_progress_to_stdout = true;
   Solver::Summary summary;
   Solve(options, &problem, &summary);
   std::cout << summary.FullReport() << '\n';
   sourceOptimizer_p.AcceptBestResult();
}


class SlopeResidual
{
public:
   SlopeResidual()
      :m_simulator(AccessSimulator())
   {}

   bool operator()(double const* const* pixelValues,
                   double* residual) const
   {
      Simulator.UpdatePixelValues(pixelValues);
      Simulator.Calc();
      residual[0] = Simulator.GetSlope();
      return true;
   }

private:
   Simulator m_simulator;
}

Solver Summary :
parameters : 1 1 1 1 1 1 1 1 1 1 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 0.999999 1 1 1 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 1 1 1 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 0.999999 1 1 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 1 1 1 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 1 0.999999 1 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 1 1 1 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 1 1 0.999999 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 1 1 1 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 1 1 1 0.999999 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 1 1 1 1 1
residual : 0.666643
parameters : 1 1 1 1 1 1 1 1 1 1 1 0.999999
residual : 0.666643
iter      cost      cost_change  |gradient|   |step|    tr_ratio  tr_radius  ls_iter  iter_time  total_time
   0  2.222064e-01    0.00e+00    0.00e+00   0.00e+00   0.00e+00  1.00e+04        0    1.12e+01    1.12e+01

Solver Summary (v 1.14.0-eigen-(3.3.5)-no_lapack-eigensparse-openmp-no_tbb)

                                     Original                  Reduced
Parameter blocks                           12                       12
Parameters                                 12                       12
Residual blocks                             1                        1
Residuals                                   1                        1

Minimizer                        TRUST_REGION

Dense linear algebra library            EIGEN
Trust region strategy     LEVENBERG_MARQUARDT

                                        Given                     Used
Linear solver                        DENSE_QR                 DENSE_QR
Threads                                     1                        1
Linear solver ordering              AUTOMATIC                       12

Question :

 When I change the parameter values while  debugging, lets say instead of 0.99999 to 0.998. It actually results in a change in the residual value. Hence I need some kind of mechanism, where a considerable change is introduced into the parameter values. 

May be my problem formulation is wrong. Any suggestions are welcome. Thank you in advance.

P.S. I actually tried with a single paramter block containing all the pixel values. Unfortunately ran into same solution.

Thank you,

Ajay.

[Sameer Agarwal]

https://github.com/ceres-solver/ceres-solver/blob/master/include/ceres/numeric_diff_options.h#L46

is your friend.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/9637fecc-5fbd-4151-a0e2-fba3d6f2d70bn%40googlegroups.com.

[Ajay Karthik]

Hi Sameer,

Thanks a lot. That worked like a charm.

Now I just need to work on faster convergence strategies.

Thank you,

Ajay.