Gradient and Hessian Check

[lineya...@gmail.com]

Hi,

Recently I've tried a lot of manifold factories in Manopt. But I find that   "checkgradient" and "checkhessian" don't always work. For example, in the fixedrank_2factors(3factors and preconditioned),  the result (slope) seems wrong, if I don't compute gradient and hessian for mistaken. And the tutorial says we cannot get the right answers for fixedrank_2factors factory. In addition, I find that it doesn't work for symfixedrankYYcomplexfactory, too. Would you please give me more infos about this? So when can we use the two functions to check our code? Thanks very much.

Kai

[BM]

Hello Kai, 

Could you give a specific example, where you found incorrect results. It would help us to understand the problem better. 

Regards,

BM

[lineya...@gmail.com]

Hi BM,

Actually I tried the case in your paper "Low-rank optimization for distance matrix completion".

I found that it's not because checkgradient doen't work, but because of a typo in equation (11), where the coefficient should be 4 instead of 2. Then so far I only know that checkgradient won't work in 2factors and 3factors as referred in the tutorial.

[BM]

Hello Kai, 

Thanks for pointing out the typo in the paper Low-rank optimization for distance matrix completion. 

Then so far I only know that checkgradient won't work in 2factors and 3factors as referred in the tutorial.

Checkgradient should always work. Checkhessian, on the other hand, gives a different slope. Could you check your gradient computations again?

Regards,

Bamdev

[lineya...@gmail.com]

Hi BM,

Yes you're right. It's checkhessian which yields a slope 2 instead of 3. My bad.

Kai

[lineya...@gmail.com]

Hello BM,

About the gradient I have another question. Would you please give me some explanation on it? 

Riemannian metric g(xi,eta) to the tangent space is essential. Then Riemannian gradient  grad(f(X)), Riemannian hessian Hess f(X) [X0] are defined in order to develop the gradient descent or trust region algorithm.  In the book optimization algorithms on the manifold, (3.37) says grad(f(X)) should be the horizontal lift of the "gradient" deriving from  g(xi_X,grad(f_bar(X)))=Df_bar(X)[xi_X] (which means the "inner product" of Riemannian gradient and direction should be equal to the Euclidean gradient). 

Actually, I'm confused about that's the difference of f and f_bar. What does it mean that f denotes the restriction of f_bar to a summanifold? Why should we do the horizontal lifting instead of directly using grad(f_bar(X))?  Could you please show me with a simple example?  Thanks very much.

Kai Yang

[BM]

Hello Kai,

Thanks for your questions.

Actually, I'm confused about that's the difference of f and f_bar. What does it mean that f denotes the restriction of f_bar to a summanifold? Why should we do the horizontal lifting instead of directly using grad(f_bar(X))? Could you please show me with a simple example?

I guess you are referring to the case of quotient manifolds. In this case, f_bar and f have different domains. f_bar is from the total space (which could be R^n or any of its submanifold) to the real line and f is from the quotient space to the real line. Our interest is to optimize f. 

It should be noted that the gradient of f is an abstract object, but its horizontal lift has a unique representation, and hence, the interest in computing the horizontal life of gradient of f. 

An important observation is that the "horizontal lift of the gradient of f" is equal to the "gradient of f_bar," provided we endow the space with a Riemannian metric. If we have a different metric, this equality no longer holds. The Riemannian metric, for quotient manifolds, needs to satisfy some particular constraints.

Finally, as you have pointed out correctly, we do use "grad(f_bar(X))". But, this choice is with respect to a Riemannian metric in the space of f_bar. 

Chapter 3 of http://press.princeton.edu/chapters/absil/Absil_Chap3.pdf should be a good resource. It takes a while to get around the notations but has all the information there. Another very good resource is the paper at https://arxiv.org/pdf/1407.5965v1.pdf. 

Regards,

BM

 

[lineya...@gmail.com]

Hello BM,

Thanks very much for your time. Actually, I used the notation in Chapter 3 of optimization algorithms on the manifold, which says that M is a submanifold or quotient manifold generated from M_bar. I'll read and learn further following your very helpful suggestions. Thanks very much.

Kai