Constrained Euclidean optimisation

[hau...@gmail.com]

Hi

I'm using manopt (with great success! thanks a bunch!) to optimise a function

f: R x Sym+(D) --> R

which takes a Euclidean scalar and a DxD positive definete matrix as input. I just use a product manifold to represent the input domain, and it works great.

However, in reality the Euclidean scalar should be constrained to an interval [a, b], with a > 0. Is this possible with the manopt framework?

I can see that I can optimise over [0, infinity) by replacing the Euclidean domain with a 1x1 positive definite matrix, but it is unclear to me if I can enforce an upper bound.

Any thoughts?

Thanks
Søren

[BM]

Hello Soren, 

Thank you for the question. Manopt cannot handle an interval constraint directly. However, as you said, we could optimize over (0, infinity). 

A naive question: what about a penalty term in the objective function for enforcing an upper bound? For example, if 

f : R+ x Sym+(D) --> R : (\alpha, X)  ---> f(\alpha, X), \alpha is assumed to be positive, then we could optimize over f(\alpha, X) + \gamma \alpha, \gamma is a tunable regularization parameter. Does this work in your case?

A second thought: depending on the cost function and the behavior of \alpha, could we have a change of variables that maps (a, b) to (0, infinity)?

Regards,

BM

[Nicolas Boumal]

Hello Søren,

Thanks for the nice feedback!

In response to Bamdev's last remark: a classical trick to work on an interval [a, b] is to do a change of variable. For example, you can do this:

Let x be a scalar, unconstrained, and

Let y = (a+b)/2 + abs(b-a)/2 * sin(x)

In this fashion, y is always in the correct interval. Of course, this does not exclude the bounds (a and b) and this induces some complications in the formulas for the cost and the gradient, but nothing too horrible.

I have no experience using this trick myself, so if you end up trying it, I'd love to hear about your experience.

Let us know if you have more questions / if this doesn't help.

Cheers,

Nicolas

[BM]

Interesting.

A band pass could be an alternative as well. There are smooth mathematical approximations. As Nicolas said, the gradient could be a bit more complicated, but its not deadly. 

Here is link http://en.wikipedia.org/wiki/Band-pass_filter

There is a rich history behind filters in general. 

Cheers,

BM 

[hau...@gmail.com]

Thanks guys!

I've been playing a bit more with things, and it doesn't seem like I need the upper bound in practice, so I think I'll just stay with a restriction to [0, infinity).

In the past, I have used the type of reparametrisations you both suggest with mixed success. It seems like the implied changes in the gradient slows down convergence (incidentally, the Deep Learning people seems to have similar experiences, which is one reason why they use rectified linear units rather than old-school sigmoid units).

Thanks for the help and a lovely piece of software!

Søren