Question about the checkhessian function

[Zhao Xingyu]

I have carefully verified my ehess derivation, but I still get this result when I run checkhessian(problem). Is there a problem with my derivation or code?

result:
# Hessian check
The slope should be 3. It appears to be: 2.00143.
If it is far from 3, then the gradient or the Hessian might be erroneous.
Hess f(x)[d] must be a tangent vector at x.
If so, the following number is zero up to machine precision: 2.52699e-16.
If it is far from 0, the Hessian returns non-tangent vectors.
The Hessian at x must be linear on the tangent space at x.
If so, ||a*H[d1] + b*H[d2] - H[a*d1+b*d2]|| is zero up to machine precision.
Value: 1.90364e-15 (norm of H[a*d1+b*d2]: 8.5303)
If it is far from 0, then the Hessian is not linear.
The Hessian at x must be symmetric on the tangent space at x.
If so, <d1, H[d2]> - <H[d1], d2> is zero up to machine precision.
Value: 2.07333 - 2.07333 = 4.44089e-16.
If it is far from 0, then the Hessian is not symmetric.

[Zhao Xingyu]

This is my checkgradient(problem)

[Nicolas Boumal]

Hello,

Indeed, these plots and text outputs do strongly suggest that (a) your gradient is correct, but (b) the Hessian is not.

We may be able to help more if you provide more information, e.g., what you put in problem.M and also in problem.cost, problem.egrad and problem.ehess.

Best,

Nicolas

[Zhao Xingyu]

Okay, thank you very much for your reply. My specific situation is as follows:

1.This is the product manifold I defined:

%%%

elems.x = spherecomplexfactory(1+para.M,1);

elems.y = euclideanfactory(para.M);

manifold = productmanifold(elems);

problem.M = manifold; %%%

Then I ran

%%%  
checkgradient(problem);
checkhessian(problem); 

%%%

The results are as shown in Figure 1

2.I tested the gradient and Hessian for each manifold separately, meaning I fixed one variable and optimized the other. When the manifold of the optimized variable is spherecomplexfactory, the results of running checkgradient(problem); checkhessian(problem); are as shown in Figure 2.

3.I again fixed one variable and optimized the other. When the manifold of the optimized variable is euclideanfactory, the results of running checkgradient(problem);and checkhessian(problem);are as shown in Figure 3.

For the results in Figure 1, it seems that the gradient of the product manifold is correct, but the Hessian is wrong. However, when I test the gradient and Hessian for each variable separately (the results in Figures 2 and 3), both seem to be correct. I can be absolutely certain that when I test each variable individually, the code is not modified at all—it's all copied directly—so there's no possibility of writing errors.

How can I explain this discrepancy? Is my Hessian really wrong?

[Nicolas Boumal]

These are useful plots. And indeed, I still believe the Hessian is incorrect, though we have more information now:

Your cost function is defined on a product manifold. For simplicity, I'll write this as f(x, y).

It appears you have the correct gradient for f.

Also, it appears you have the correct diagonal blocks for the Hessian of f, that is, d²f/dx² and d²f/dy² seem to be correct since checkhessian is valid when you fix either of the variables to be constant.

Thus, it seems that the issue is with the mixed derivative, that is, d²f / dx dy.

Since part of your manifold is complex, the issues might come from a missing "conjugate" or other operation, or missing real(.) part.

(Just in case, and as you likely know, in Matlab, A' means "conjugate transpose", whereas A.' means transpose (without conjugation).)

I hope this helps push in the right direction.