Preconditioning LBFGS

[Oussama Berqi]

Hello,

When I seed the LBFGS with a vector very close to the optimal solution, the first pass move away from the optimal, then returns after few passes. Is there any proper way to use a custom preconditional (like the stadard Jacobi preconditioner) with LBFGS?

Thanks

[David Hall]

There's not, sorry. Could you open an issue? I'm not entirely sure of the best API for providing that functionality. Thoughts?

-- David

--
You received this message because you are subscribed to the Google Groups "Scala Breeze" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-breeze...@googlegroups.com.
To post to this group, send email to scala-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scala-breeze/bca343a2-a93d-437c-bad8-f383afb78278%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oussama Berqi]

Thanks four your answer David, I have opened an issue.

I guess a good API would be choosing a preconditioner while creating an LBFGS object: 

val lbfgs = new LBFGS[T]( maxNumIterations, numCorrections,convergenceTol)

So that when you minimize using an initial vector, the preconditioner adds the Hessian's matrix approximation automatically to the history?

lbfgs.minimize(cachedDiffFunction, initialWeights)

Oussama

[David Hall]

The problem is, DiffFunction's interface only exposes the gradient and objective value. Not the second derivative. It's not always possible to compute the second moments easily, so LBFGS needs to support the current kinds of functions. I guess we could just do a pattern match to check if the function supports a second derivative, but that's kind of icky. 

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-breeze/409304fd-fea1-44c0-89db-0583a33a96cd%40googlegroups.com.

[xuepeng sun]

Hi David,

   I have a follow-up question on this thread.  There is a LBFGS.ApproximateInverseHessian class provided in the LBFGS class. Do you have a usecase with it?

   I'd like to perform 4~5 evaluations of the objective/gradient and obtain the ApproximateInverseHessian.  I would then compute the max eigen-value 

 via power method. Do you have any suggestions or pointers?  

   Thanks a lot. 

Xuepeng

[Debasish Das]

BFGS normally should be used for cases where Hessian is difficult to compute analytically...

If you have a good pre-conditioner from structure of the problem, nonlinear CG should work much nicer...TRON is an example of that...it is also implemented in breeze...