Introducing the MixEst toolbox

[mrma...@gmail.com]

Dear Nicolas and Bamdev

Hi,

We have developed a Matlab toolbox for mixture models, and we use Manopt to employ gradient-based optimization methods on manifolds to do parameter estimation for the models. Basically we define product manifolds for the parameters of the models and then optimize the log-likelihood. Various design aspects of our code have been inspired from your excellent toolbox. We also include some manifolds and an LBFGS solver for Manopt. Unfortunately we still haven't had time to make them compatible with Manopt 2. You can find the toolbox here: http://visionlab.ut.ac.ir/mixest. We will be happy to learn of your thoughts on our library.

Here is some suggestions about your awesome toolbox that came to my mind while developing MixEst (sorry if I'm being concise too much):
1. Support for mini-batch optimization
2. Support for discarding the optimization history after each mini-batch (as in Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Le, Q. V., & Ng, A. Y. (2011). On optimization methods for deep learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-11) (pp. 265-272).)
3. Letting custom stop messages in user's stopfun (adding a reason output for stopfun prototype)
4. Adding a stopping criterion that checks for a tolerance on cost difference, not the value of the cost.

Thank you

[Nicolas Boumal]

Hello Mohamadreza,

Thank you for your message and your useful feedback.

Could you tell us a bit more about what mini-batch optimization is, and its purpose?

For stopfun, this is a very good suggestion, and was added to the TODO list. (To be honest, I was surprised this was not already that way in the toolbox.)

About adding the specific stopping criterion you mention, it seems to me that it is a rather non-standard one (although I may be mistaken on this), because it is rather difficult to get right (the cost may stagnate for a number of reasons, far away from a critical point).  Nevertheless, it is readily doable in Manopt. In fact, this is the example given for usage of stopfun (as I am sure you have seen yourself): If we feel more interest in this becoming a standard criterion, we will happily include it, of course.

The return value is a Boolean. If stopnow is returned as true, the solver will terminate.

Example:

options.stopfun = @mystopfun;
function stopnow = mystopfun(problem, x, info, last)
    stopnow = (last >= 3 && info(last-2).cost - info(last).cost < 1e-3);
end

This will tell the solver to exit as soon as two successive iterations combined have decreased the cost by less than 10-3.

Btw, in time, your BFGS implementation could be of significant interest to other Manopt users as well. 

Cheers,

Nicolas

PS: if you experience difficulties switching to Manopt 2.0, we would be happy to hear about your experience. Many things changed indeed. For "normal" use of the toolbox, code should be backward compatible, but for more heavy, almost hack-level of the toolbox as is probably your case, it may indeed be more difficult to make the transition.

[mrma...@gmail.com]

Oops, about mini-batching I guess that was a mix up. It's about taking a small number of data points at each iteration for large datasets, but I see Manopt doesn't deal with data directly.

The stopping criterion is not so important, but I thought setting an options.tolcostdiff would be easier than using stopfun, and when we have no idea of the range of values that the cost can take, it is a common practice to stop optimization when the cost improvement becomes too small (beside checking the norm of the gradient). Local minima can be avoided by some higher-level method.

Thanks for your consideration

All Best

[Nicolas Boumal]

Hello again,

How data is handled is indeed not so much the focus of Manopt, but, on a related note, it would make sense to have a type of Riemannian stochastic gradient descent algorithm in Manopt (as for example appeared in some work by Silvere Bonnabel: http://arxiv.org/abs/1111.5280).

Best,

Nicolas