autodiffcostfunction Cost Function calculation for 2 images

[erdem hayır]

I am calculating cost function by subtracting every pixels from real image and virtual image. Virtual image is generated by the data rotation, translation and focal length which I want to optimize with Autodiffcostfunction. Normally I tried to do that with numericalcostfunction by evaluation_callback which calculates virtual image before cost function. However I learned evaluation_callback does not work with numericalcostfunction. My question is how can i use autodiffcostfunction to calculate costfunction bw real image and pre-computed virtual image. Of course virtual image should be calculated for every new rotation, translation and focal length values. Thanks for your help

[Sameer Agarwal]

Erdem,

I do not really understand your question. Could you describe what you are trying to do mathematically/fragment of code and where the problem is?

Also have you had a look at cubic_interpolation.h ? It is designed for working with images in scenarios like yours.

Sameer

On Mon, May 27, 2019 at 5:53 AM erdem hayır <erdem...@gmail.com> wrote:

I am calculating cost function by subtracting every pixels from real image and virtual image. Virtual image is generated by the data rotation, translation and focal length which I want to optimize with Autodiffcostfunction. Normally I tried to do that with numericalcostfunction by evaluation_callback which calculates virtual image before cost function. However I learned evaluation_callback does not work with numericalcostfunction. My question is how can i use autodiffcostfunction to calculate costfunction bw real image and pre-computed virtual image. Of course virtual image should be calculated for every new rotation, translation and focal length values. Thanks for your help

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/29a6bcc0-f5b1-496d-8db5-dd5b8e95a630%40googlegroups.com.

[erdem hayır]

Hay Sameer, 

I could not respond it because I tried to implement bicupicinterpolation. I have real image I, dot pattern reconstruction S, and virtual image V. 

 My cost function calculates difference bw normalized real image and normalized virtual image. Virtual image is dependent on S and in every iteration 

S should be updated. 

this is my grid and bicubic interpolator in main function

mGrid.reset(
        new ceres::Grid2D<uchar, 1>(S.ptr<uchar>(0), 0, S.rows, 0, S.cols));
mInterpolator.reset(
        new ceres::BiCubicInterpolator<ceres::Grid2D<uchar, 1>>(*mGrid));

inside functot i called it 

ImageCost(cv::Mat &S, cv::Mat &V,  const cv::Mat I, int i,int j,double width, double height, map<int, vector<Point3d>> planeNormalAndDistance,
          std::shared_ptr<ceres::BiCubicInterpolator<ceres::Grid2D<uchar, 1>>> mInterpolator_, int imageIdx_)
:S_(S), V_(V), I_(I), i_(i),j_(j), width_(width), height_(height), planeNormalAndDistance_(planeNormalAndDistance),
mInterpolator(mInterpolator_), imageIdx(imageIdx_){

down is my changing values rotation, focal length and translation.

template <typename T>
bool operator()(const T* const fp, const T* const rotation, const T* const translation, T* residuals) const {

In every iteration I want to  get updated focal length, rotation and translation. Use them in evaluation_callbacks function to calculate new S. 

Then I want to use S in next iteration of cost function

below is evaluation_callback function

evaluation_callback_functor(cv::Mat & S, cv::Mat realImg,  int width, int height, map<int, vector<Point3d>> planeNormalAndDistance,
                            double* translation, double* rotArray,  double* fp_
)
        : S_(S), realI(realImg), width_(width), height_(height), planeNormalAndDistance_(planeNormalAndDistance),

translation_(translation), rotArray_(rotArray), fp(fp_) 

I am calling it in main like following

options.evaluation_callback = (new evaluation_callback_functor(S, im1,width, height,
        mapNormalAndDist, translation,rotationArray, fp));

My further question: 

1- Can I use evaluation_callback with AutodiffCostfunction?

2- Will aforementioned method bicubicinterpolator get my new S value or not?

Best Regards

On Tuesday, May 28, 2019 at 7:48:07 PM UTC+2, Sameer Agarwal wrote:

Erdem,

I do not really understand your question. Could you describe what you are trying to do mathematically/fragment of code and where the problem is?

Also have you had a look at cubic_interpolation.h ? It is designed for working with images in scenarios like yours.

Sameer

On Mon, May 27, 2019 at 5:53 AM erdem hayır <erdem...@gmail.com> wrote:

I am calculating cost function by subtracting every pixels from real image and virtual image. Virtual image is generated by the data rotation, translation and focal length which I want to optimize with Autodiffcostfunction. Normally I tried to do that with numericalcostfunction by evaluation_callback which calculates virtual image before cost function. However I learned evaluation_callback does not work with numericalcostfunction. My question is how can i use autodiffcostfunction to calculate costfunction bw real image and pre-computed virtual image. Of course virtual image should be calculated for every new rotation, translation and focal length values. Thanks for your help

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-...@googlegroups.com.

[Sameer Agarwal]

I am sorry I still do not understand. But let me try and answer to the best of my ability.

On Mon, Jun 10, 2019 at 7:07 AM erdem hayır <erdem...@gmail.com> wrote:

Hay Sameer, 

I could not respond it because I tried to implement bicupicinterpolation. I have real image I, dot pattern reconstruction S, and virtual image V. 

 My cost function calculates difference bw normalized real image and normalized virtual image. Virtual image is dependent on S and in every iteration 

S should be updated. 

this is my grid and bicubic interpolator in main function

mGrid.reset(
        new ceres::Grid2D<uchar, 1>(S.ptr<uchar>(0), 0, S.rows, 0, S.cols));
mInterpolator.reset(
        new ceres::BiCubicInterpolator<ceres::Grid2D<uchar, 1>>(*mGrid));

why are you using uchar here? use float or double here. I should make it clearer in the documentation.

inside functot i called it 

ImageCost(cv::Mat &S, cv::Mat &V,  const cv::Mat I, int i,int j,double width, double height, map<int, vector<Point3d>> planeNormalAndDistance,
          std::shared_ptr<ceres::BiCubicInterpolator<ceres::Grid2D<uchar, 1>>> mInterpolator_, int imageIdx_)
:S_(S), V_(V), I_(I), i_(i),j_(j), width_(width), height_(height), planeNormalAndDistance_(planeNormalAndDistance),
mInterpolator(mInterpolator_), imageIdx(imageIdx_){

down is my changing values rotation, focal length and translation.

template <typename T>
bool operator()(const T* const fp, const T* const rotation, const T* const translation, T* residuals) const {

In every iteration I want to  get updated focal length, rotation and translation. Use them in evaluation_callbacks function to calculate new S. 

Then I want to use S in next iteration of cost function

I do not understand your calling sequence here.  Why are you using EvaluationCallback at all? you should only need to use it in circumstances where you really need to share compute across cost functions and it is very expensive to re-do the computation. Since you are still just getting started with ceres,  I recommend not using evaluationcallback at all.

Sameer

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/867ff512-948f-4a42-93e8-bacfe8ba4991%40googlegroups.com.

[erdem hayır]

Let me be more try do be more clear

I have to use S to calculate V. Cost function dependent on I and V.  My block variables focal length, rotation and translation. 

In next iteration I will get the new focal length, rotation and translation. By using them I want to calculate new S. This is supposed to be use in next iteration of cost function. I thought evaluation_callbacks could be best options and My thesis supervisor also recommended me to use it. 

What should i use otherwise?

Am i wrong about working principle of AutoCost function and evaluation callback. In every iteration Cost function gives me new block values and I can use them in evaluation callback by the help of pointers. Can't I?

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/867ff512-948f-4a42-93e8-bacfe8ba4991%40googlegroups.com.

[Sameer Agarwal]

On Mon, Jun 10, 2019 at 7:26 AM erdem hayır <erdem...@gmail.com> wrote:

Let me be more try do be more clear

I have to use S to calculate V. Cost function dependent on I and V.  My block variables focal length, rotation and translation. 

In next iteration I will get the new focal length, rotation and translation. By using them I want to calculate new S. This is supposed to be use in next iteration of cost function. I thought evaluation_callbacks could be best options and My thesis supervisor also recommended me to use it. 

What should i use otherwise?

Am i wrong about working principle of AutoCost function and evaluation callback. In every iteration Cost function gives me new block values and I can use them in evaluation callback by the help of pointers. Can't I?

Why are you trying to share the compute across iterations? there is no guarantee what value of parameter blocks is used across iterations.

Why not compute S and V every time the cost function is evaluated?

Sameer

 

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/102833a2-602e-4c36-8f9c-584963b0fcd7%40googlegroups.com.

[erdem hayır]

Reason: 

When I need to compute V pixel by pixel, I need whole S not just related pixels. Cuz there is no 1 to 1 corresponce. For example 

if I calculates pixel (10, 20) in V, that could be pixel(25, 34) in S. 

And I am calculation cost function for every pixels

for (int i = 1; i < width - 1 - 3; ++i) {
    for (int j = 1; j < height - 1 - 3; ++j) {
        /*ceres::CostFunction* cost_function =
                 new ceres::NumericDiffCostFunction<RcpAndFpOptimizer, ceres::CENTRAL, 1, 1, 3, 3>(
                         new RcpAndFpOptimizer(S,normalizedVirtualImg, im1, i, j, width, height, mapNormalAndDist));

         problem.AddResidualBlock(cost_function, NULL, fp, rotationArray, translation);*/

        ceres::CostFunction *cost_function =
                new ceres::AutoDiffCostFunction<ImageCost, 1, 1, 3, 3>(
                        new ImageCost(S, normalizedVirtualImg, realImg, i, j, width, height, mapNormalAndDist,
                                      mInterpolator, imageIdx));

        problem.AddResidualBlock(cost_function, NULL, fp, rotationArray, translation);


    }
}

I need whole S. that's why I can not calculate S in cost function

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/102833a2-602e-4c36-8f9c-584963b0fcd7%40googlegroups.com.

[Sameer Agarwal]

okay. so autodiff or numeric diff have nothing to do with evaluationcallback. evaluation callback is called once everytime the problem residuals and/or the jacobian is evaluated.  So as long as you take care of type conversions and various Jacobians you should be fine.

I recommend trying this out on a toy problem and getting your intuition about how it works right.

Sameer

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/afc9de4b-3354-4c75-94bc-e939d36b480e%40googlegroups.com.

[Andy Keeling]

Hi - sorry to jump in - I have a very similar task and wanted to check I am using Ceres correctly for it

I render a 3D image in VTK using camera parameters (rotation/translation) that are to be optimised.

I have a second 2D photo to which I will be aligning the 3D render.

My plan is to use some cost function in OpenCV, to grade the correlation between 2 images:

float correlation(cv::Mat &image1, cv::Mat &image2) 
{
    cv::Mat corr;
    cv::matchTemplate(image1, image2, corr, cv::TM_CCORR_NORMED);
    return corr.at<float>(0,0);  // corr only has one pixel
}

So my workflow is:

1) Render 3D image using current camera R|T and grab 2D image

2) Apply image correlation as shown above using OpenCV (maybe return this as 1/cost because the OpenCV cost increases with better correlation)

3) Adjust R|T and return to 1)

I previously used a library called nlopt for this but want to port to Ceres.

Is this really just something I should implement using NumericalDiff() (I have only ever used AutoDiff but my understanding is I need templated functions for this, which is not possible in this case).

Or should I be looking at something like the  Ceres 2D grid, or cubic interpolation?

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/afc9de4b-3354-4c75-94bc-e939d36b480e%40googlegroups.com.

[Sameer Agarwal]

This is precisely the case the bicubic interpolation based grid is for.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/1baeb040-e626-4803-be67-5ef592d76e7f%40googlegroups.com.

[Patrik Huber]

Hi Sameer,

I'd like to pick up one of your earlier comments in this thread:

 

mGrid.reset(new ceres::Grid2D<uchar, 1>(S.ptr<uchar>(0), 0, S.rows, 0, S.cols));
mInterpolator.reset(new ceres::BiCubicInterpolator<ceres::Grid2D<uchar, 1>>(*mGrid));

why are you using uchar here? use float or double here. I should make it clearer in the documentation.

I've been using unsigned char as well. Is there any particular problem with it? Why would you recommend to use float/double instead? Would you recommend converting the whole image from unsigned char to double at the start of the optimisation?

It would be great to know a bit more about this and also have it mentioned in the documentation (I read through all the documentation and code of Grid2D and BiCubicInterpolator and don't remember seeing anything like that mentioned).

Thanks a lot!

Patrik

[Sameer Agarwal]

Patrik thanks for catching this. I screwed up.

we explicitly cast to double 

https://github.com/ceres-solver/ceres-solver/blob/master/include/ceres/cubic_interpolation.h#L201

so please ignore what I said about this. uchar is just fine.

Sameer

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/05b5e126-9fca-48c0-ab91-9652155315f3%40googlegroups.com.

[Patrik Huber]

Ah, cool! Thanks a lot for the clarification Sameer! :-)

Patrik

On Monday, 17 June 2019 19:07:44 UTC+1, Sameer Agarwal wrote:

Patrik thanks for catching this. I screwed up.

we explicitly cast to double 

https://github.com/ceres-solver/ceres-solver/blob/master/include/ceres/cubic_interpolation.h#L201

so please ignore what I said about this. uchar is just fine.

Sameer

On Mon, Jun 17, 2019 at 11:02 AM Patrik Huber <patri...@gmail.com> wrote:

Hi Sameer,

I'd like to pick up one of your earlier comments in this thread:

 

mGrid.reset(new ceres::Grid2D<uchar, 1>(S.ptr<uchar>(0), 0, S.rows, 0, S.cols));
mInterpolator.reset(new ceres::BiCubicInterpolator<ceres::Grid2D<uchar, 1>>(*mGrid));

why are you using uchar here? use float or double here. I should make it clearer in the documentation.

I've been using unsigned char as well. Is there any particular problem with it? Why would you recommend to use float/double instead? Would you recommend converting the whole image from unsigned char to double at the start of the optimisation?

It would be great to know a bit more about this and also have it mentioned in the documentation (I read through all the documentation and code of Grid2D and BiCubicInterpolator and don't remember seeing anything like that mentioned).

Thanks a lot!

Patrik

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-...@googlegroups.com.