breaking into multiple residual blocks because of different residual value range

[Ajay Karthik]

Hello team,

My scenario : 

• 1 parameter block (can't change it easily because initially it is a dense problem, but nevertheless the optimizer will reduce some of the parameter values to zero). Don't know how to use this fact ? (can be a different discussion altogether)
• single residual block of size 4, but with different value ranges of individual residuals
  • Takes a lot of computation time
• Multiple residual blocks, ensuring that in each block individual residual value ranges are matched.
  • Computation time improved.

I would like to hear your suggestions, to further improve the performance. Parallelization is also an option, but not sure if it can be done with only a single parameter block.

Any suggestions are welcome.

Thank you,

Ajay.

[Sameer Agarwal]

I do not understand what you mean by individual residual blocks with range matching of residuals.

If you have a parallelizable expensive compute inside your cost function yo in can use threading to parallelize it yourself also.

That way to you do not depend on residual level parallelization in Ceres.

Sameer 

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/bed123c6-b70d-4845-a3fc-17e3313471f8n%40googlegroups.com.

[Ajay Karthik]

Hi Sameer,

I personally feel it is beneficially to have the range of residual values to be in the same scale.

For example we have residual block of size 4 : 

Residual block current values : [0.6, 55, 47, 0.07]

Hence I made some reshuffling with respect to residual blocks

Residual block 1 : [0.6] Residual block 2 : [55, 47] block 3 : [0.07]

After making this split, I was able to see a slight improvement in the performance. Nevertheless I am unable to understand the reason behind it.

Thanks,

Ajay.

[Sameer Agarwal]

Unless you are using a loss function, this kind of splitting should have no effect in exact arithmetic. 

However it may cause some small numerical differences which my get amplified due to your problem's conditioning.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/b7306eae-1e8a-45be-bc44-5ed6a5339500n%40googlegroups.com.

[Ajay Karthik]

Hi Sameer,

Thank you for the reply. So could it be the case that splitting the problem into multiple residual blocks helped me with the problem's conditioning. 

Thank you,

Ajay.

[Sameer Agarwal]

No, the conditioning remains the same, but breaking the residual into multiple pieces introduced some small numerical changes which get amplified by the condition number.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/1bb32b0b-1a59-43c9-825e-d3b6aa9d1413n%40googlegroups.com.

[Ajay Karthik]

Hi Sameer,

Thank you for the tip about the system conditioning. I will just make sure that I have a single residual block.

Is it possible to check the condition number of particular matrix / some linearized system ?

Thank you,

Ajay.

[Sameer Agarwal]

You can just check the condition number of the Jacobian you are competing by dumping it out from your cost function.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/c8344997-2fb1-4648-bcc6-44f5f9484710n%40googlegroups.com.

[Ajay Karthik]

Ok got it thank you.

[Ajay Karthik]

Hello Ceres Team,

I need a suggestion on optimizing for all the residual blocks. Because I perceive that the optimizer is optimizing only on few blocks, may be because these values are more sensitive to the changes in the parameter block.

Initial residual values

[0.52, 4., 4.1, 0.2, ....]

Final residual values

[0.203, 3.37, 4.4, 0.09,...]

As you can see, smaller range of values [0.5, 0.2] ->[0.203, 0.09] are being minimized quite a bit comparatively. Could you please let me know, suggestions for the same. I already tried scaling the larger range of values [4, 4.1,...], but without any luck.

Any suggestions are a great help. I was thinking more in terms of, in case if the relative change in the individual residual could be controlled until all the individual residuals relative change could come to the same level. I know,the way I am formulating is pretty vague, but I was just wondering in case something like that will help.

Thank you,

Ajay Karthik.

[Sameer Agarwal]

Ajay,

it would help to understand what you are trying to do mathematically rather than talking in terms of residuals.

the fact that some residuals are not being reduced is a property of your mathematical model. 

consider for example a problem 

f(x) = g^2(x) + h^2(x)

with two residuals.

and the second residual is a constant, the magnitude of the residual then has nothing to do with it, it will never be reduced.

also why does it matter whether some residuals are being minimized or not, the overall objective function value is what matters.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/d867bff4-3b5f-4aef-b8db-506e7dff579an%40googlegroups.com.

[Ajay Karthik]

Hi Sameer,

Here is a brief overview of, how my residuals are modelled.

The simple curve might be the result of a simulation. Moreover I know from the experiments that for my particular case, the output is symmetric. Hence I believe that there should be solution, which minimizes all the individual residuals.

May be I am converging myself into a local minimum, since one of the residuals is really sensitive to the parameter block. Could that be possible ?

Thanks,

Ajay

[Sameer Agarwal]

On Wed, Jul 27, 2022 at 11:32 AM Ajay Karthik <ajayka...@gmail.com> wrote:

Hi Sameer,

Here is a brief overview of, how my residuals are modelled.

The simple curve might be the result of a simulation. Moreover I know from the experiments that for my particular case, the output is symmetric. Hence I believe that there should be solution, which minimizes all the individual residuals.

you mean that you believe that at the global minimum the objective function will be zero?

Sameer

 

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/a89cda14-fbf0-4678-9575-e3d09d20a512n%40googlegroups.com.

[Ajay Karthik]

Hi Sameer,

No I don't think it can be zero. But I expect individual residuals to decrease, instead in one of my cases there was an slight increase or it stayed the same.

Thank you,

Ajay.

[Sameer Agarwal]

On Wed, Jul 27, 2022 at 11:42 AM Ajay Karthik <ajayka...@gmail.com> wrote:

Hi Sameer,

No I don't think it can be zero. But I expect individual residuals to decrease, instead in one of my cases there was an slight increase or it stayed the same.

no thats not necessarily going to be the case. the only thing you can expect to happen is for the net objective function to decrease.

it is very easy to construct examples where the optimum is such that one residual decreases and the other one increases for a net decrease in the total objective function.

consider the problem x^2 + ((x - 1)/100)^2  and starting at x = 1. 

one residual will be 1 and the other zero. but as you move towards zero, the first residual will decrease and the second will increase.

Sameer

 

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/b69da957-1db8-4823-8ade-5867facc4092n%40googlegroups.com.

[Ajay Karthik]

Actually I see your point. May be I am optimising towards something like this : 

I guess I have introduce a new residual, which holds the equality of residuals between the r_1 and r_2. Then may be everything will be fine. I probably need to do something like 100. * (abs(r_1) - abs(r_2)).

Thank you for your time.

Best regards,

Ajay

[Ajay Karthik]

Hi Sameer,

Unfortunately I ran into similar kind of issue again. That it is running into a local minimum, because I don't have intelligent scaling factor.

Now my solution maintains the symmetry but at the expense that it doesn't achieve optimal values for the other residuals.

It is almost like, it doesn't allow the temporary increase in the objective function, which might result in the significant decrease later in the solution. Yes I am using the monotonic steps flag, in order to allow this behavior. I guess the only way for me, is to have a smart scaling factor of the individual residual terms.

In case you have come across something like this, let me know or point me in that direction.

Thank you

Ajay