Normalization of losses

[Anton Zhilin]

I'm trying to use a LossFunction to robustify my problem. But I wonder whether resuduals are normalized before application of LossFunction. In Matlab, it looked like:

residuals = rho(residuals / (median(residuals) * tuningConstant))

If residuals are not normalized, then it's a catastrophe. How can I do this?

[Sameer Agarwal]

no the residuals are not normalized in any way before application of the loss function.

I am not entirely sure what you mean by it being a catastrophe. 

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/dff72d0f-9fff-4cac-8dba-784b4c2b358c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anton Zhilin]

residuals = rho(residuals / (median(residuals) * tuningConstant))

How can I do what is described by the formula? In this notation, Ceres currently does just this:

residuals = rho(residuals)

Before passing them on. Rho is CauchyLoss, for example.

[Sameer Agarwal]

Anton, you can do this by having a single residual and computing the median yourself and dividing the residuals by it, but by doing so you will make the problem completely dense. 

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/e8753e60-e14e-4229-8529-591f4ab00674%40googlegroups.com.

[Anton Zhilin]

So there is no way do do this with sparse problems? Do LossFunctions take residuals from a single CostFunction only?

I don't mind losing a little bit of performance on this normalization.

понедельник, 15 августа 2016 г., 16:58:13 UTC+3 пользователь Sameer Agarwal написал:

[Sameer Agarwal]

no, because in calculating the median, you are making all the residuals depend on each other.

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/9d1c59c1-4701-41f5-ac6e-a8b081303e0d%40googlegroups.com.

[Anton Zhilin]

How about this: before each step I manually compute all residuals, take the median and send it to LossFunctions. Can I somehow do this? If so, how?

понедельник, 15 августа 2016 г., 18:05:11 UTC+3 пользователь Sameer Agarwal написал:

[William Rucklidge]

You may be able to do this with an iteration callback (to get the median) and using LossFunctionWrapper, wrapping ScaledLoss which you adjust/replace as needed.

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/5401ec50-c677-4057-ae3b-38fa1a1efe49%40googlegroups.com.

[Sameer Agarwal]

IterationCallback right now does not return a vector of residuals and unless you are willing to evaluate cost functions yourself (you cannot call Problem::Evaluate from an iteration call back), this won't quite work.

On Mon, Aug 15, 2016 at 8:30 AM 'William Rucklidge' via Ceres Solver <ceres-...@googlegroups.com> wrote:

You may be able to do this with an iteration callback (to get the median) and using LossFunctionWrapper, wrapping ScaledLoss which you adjust/replace as needed.

On Mon, Aug 15, 2016 at 8:15 AM, Anton Zhilin <antony...@gmail.com> wrote:

How about this: before each step I manually compute all residuals, take the median and send it to LossFunctions. Can I somehow do this? If so, how?

понедельник, 15 августа 2016 г., 18:05:11 UTC+3 пользователь Sameer Agarwal написал:

no, because in calculating the median, you are making all the residuals depend on each other.

On Mon, Aug 15, 2016 at 8:02 AM Anton Zhilin <antony...@gmail.com> wrote:

So there is no way do do this with sparse problems? Do LossFunctions take residuals from a single CostFunction only?

I don't mind losing a little bit of performance on this normalization.

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/5401ec50-c677-4057-ae3b-38fa1a1efe49%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CABsh3u6a%2BNCXMKD14zJGS7KRf_-RM43QKGjsG9Z%3DNRq-dgngNQ%40mail.gmail.com.

[Anton Zhilin]

Ok, I'll look into these, thanks!

понедельник, 15 августа 2016 г., 18:30:29 UTC+3 пользователь wjr написал:

[Alex Stewart]

You might be able to do this via something like this:

- Setup Problem with CostFunctions using Problem::Options::cost_function_ownership = DO_NOT_TAKE_OWNERSHIP, and retain externally the CostFunction object pointers (so that you can set the current scale factor on each) - given that you want to scale them all by the same value, the ordering isn’t important.

— Your other options would be either: rebuilding the Problem at each iteration of the loop below (safe, but probably slow / a bit unwieldy) - or (bad) using GetCostFunctionForResidualBlock() & cost_cast().

for (i=0; i<num_iterations; ++i) {

- Solve() with max_num_iterations = 1.

- Evaluate residuals and compute median.

- Update the CostFunctions with the new scale factor via the externally retained pointers.

}

-Alex

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/fda659e3-8c15-42dc-b4ab-5cdb1df30039%40googlegroups.com.

[Sameer Agarwal]

do what alex says. except that you will not get any trust region sizing out of this.

so run the solver for a few iterations instead of 1.

also you do not need to deal with ownership issues, as long as you have the ptr to the cost function and can call a method on it to update the scaling.

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/FCB229F1-1BEC-45DC-97D4-5CA6C8E02B70%40gmail.com.

[Keir Mierle]

Hi Anton, Unfortunately Sameer is right; the cost as you have written it becomes completely dense. Alex's suggestion is a good one, though I would suggest instead seeing if you could take the median cost as a constant that you compute before you start the optimization.

Potentially you could track the median residual value vs an initial median residual estimate, then restart the optimization if there is too much difference between the initial estimate and the current median residual value. However I suspect this is not provable convergent.

Note: do NOT change the cost function via callbacks or through other means while the optimizer is running; it will cause bad things to happen (and violates a fundamental assumption of Ceres).

On Mon, Aug 15, 2016 at 10:05 AM, 'Sameer Agarwal' via Ceres Solver <ceres-...@googlegroups.com> wrote:

do what alex says. except that you will not get any trust region sizing out of this.

so run the solver for a few iterations instead of 1.

also you do not need to deal with ownership issues, as long as you have the ptr to the cost function and can call a method on it to update the scaling.

Sameer

On Mon, Aug 15, 2016 at 10:03 AM Alex Stewart <alex...@gmail.com> wrote:

You might be able to do this via something like this:

- Setup Problem with CostFunctions using Problem::Options::cost_function_ownership = DO_NOT_TAKE_OWNERSHIP, and retain externally the CostFunction object pointers (so that you can set the current scale factor on each) - given that you want to scale them all by the same value, the ordering isn’t important.

— Your other options would be either: rebuilding the Problem at each iteration of the loop below (safe, but probably slow / a bit unwieldy) - or (bad) using GetCostFunctionForResidualBlock() & cost_cast().

for (i=0; i<num_iterations; ++i) {

- Solve() with max_num_iterations = 1.

- Evaluate residuals and compute median.

- Update the CostFunctions with the new scale factor via the externally retained pointers.

}

-Alex

On Aug 15, 2016, at 5:50 PM, Anton Zhilin <antony...@gmail.com> wrote:

Ok, I'll look into these, thanks!

понедельник, 15 августа 2016 г., 18:30:29 UTC+3 пользователь wjr написал:

You may be able to do this with an iteration callback (to get the median) and using LossFunctionWrapper, wrapping ScaledLoss which you adjust/replace as needed.

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/fda659e3-8c15-42dc-b4ab-5cdb1df30039%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/FCB229F1-1BEC-45DC-97D4-5CA6C8E02B70%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CABqdRUCnczjgF_uCihnQB2UNyixoFi8aJ%2BxmUnq_jvrXSRUqGg%40mail.gmail.com.