Hessian matrix calculation

73 views
Skip to first unread message

邱知道

unread,
Apr 7, 2021, 1:19:54 PMApr 7
to Manopt
Hello, Nicolas
I want to minimize a cost function: f(s) = s'*C*s*s'*C'*s. 
Its first-order derivative is: df(s) = 2*(s'*C*s*C'*s + s'*C'*s*C*s) and it could be used as 
Problem.egrad = @(s) 2*(s'*C*s*C'*s + s'*C'*s*C*s);
Could you please tell me how to calculate and use its Hessian matrix? I realized there is a '‘H’ in the given examples, like Problem.ehess = @(s, H)...... and I don't know why there is a ’H‘ in the variables.

Jayadev Naram

unread,
Apr 7, 2021, 2:33:56 PMApr 7
to Manopt
Hello,
     To my understanding ehess function is expected to compute the directional derivative of egrad at a point 's' along a vector 'v'. 
I have written a small code using euclideanfactory manifold. I have assumed that 's' is an n-dimensional real vector. The gradient and hessian checks confirm that my calculations are correct. 

Please correct me if I am wrong.

function test()
    
    n = 100;
    C = magic(n);
    A = C + C';

    manifold = euclideanfactory(n,1);
    problem.M = manifold;
    
    % Note that for a vector s, s'*C*s = s'*C'*s
    problem.cost  = @(s)     1/2*(s'*C*s)^2;
    problem.egrad = @(s)    (s'*C*s)*(A*s);
    problem.ehess = @(s,v) (s'*C*s)*(A*v) + (v'*A*s)*A*s;

    checkgradient(problem); pause;
    checkhessian(problem); pause;

end

Best Regards,
Jayadev Naram.
Message has been deleted

邱知道

unread,
Apr 7, 2021, 8:09:57 PMApr 7
to Manopt

Hello,
Yes, your code runs correctly. But when I change   'manifold = euclideanfactory(n,1) ' to 'euclideancomplexfactory(n,1); ', the 'checkhessian(problem)' output a wrong result:
The slope should be 3. It appears to be: 1.
If it is far from 3, then directional derivatives,....
Best,
Xiangfeng

Jayadev Naram

unread,
Apr 7, 2021, 11:38:47 PMApr 7
to Manopt
Hello Xiangfeng,
     I have corrected the code to work with euclideancomplexfactory. I am attaching the code below. The changes I made are that to treat s'*C*s and s'*C'*s as different entities and carefully evaluate the gradient and the directional derivative of gradient. For numerical reasons, there is a small residue of imaginary part left in computing (s'*C*s)*(s'*C'*s), so I had to return the real part of it. Now the gradient and hessians check pass! 

function test()
    
    n = 100;
    C = rand(n)+rand(1)*1i*rand(n);
    
    manifold = euclideancomplexfactory(n);
    problem.M = manifold;
    
    % Note that s'*C*s is the complex conjugate of s'*C'*s.
    % Yet (s'*C*s)*(s'*C'*s) might have some residual imaginary
    % part, so we have return real part of the cost function.
    problem.cost  = @(s)   real(1/2*(s'*C*s)*(s'*C'*s));
    problem.egrad = @(s)   (s'*C'*s)*(C*s) + (s'*C*s)*(C'*s);
    problem.ehess = @(s,v) (s'*C'*s)*(C*v) + (s'*C'*v)*(C*s) + ...
                           (v'*C'*s)*(C*s) + (s'*C*s)*(C'*v)+ ...
                           (s'*C*v)*(C'*s)+ (v'*C*s)*(C'*s);

    checkgradient(problem); pause;
    checkhessian(problem); pause;

end

Regards,
Jayadev.

Nicolas Boumal

unread,
Apr 8, 2021, 4:42:35 AMApr 8
to Manopt
Hello everyone,

Jayadev, thanks for weighing in. You can also write the cost function as 1/2*abs(s'*C*s)^2, in which case there is no need to take the real part.

Best,
Nicolas

邱知道

unread,
Apr 8, 2021, 10:21:12 AMApr 8
to Manopt
Hello, Jayadev.
Thank you very much for your help.
Best,
Xiangfeng
Message has been deleted

Robert Qiu

unread,
Jul 2, 2021, 7:32:26 AMJul 2
to Manopt
Hello, Jayadev
Thank you very much for your previous help. But I am still not very clear about the calculation process of the Riemann Hessian matrix. I can only calculate the Euclidean gradient and Euclidean Hessian of the objective function, but I don't know how to calculate the directional derivative of the Euclidean gradient along a certain direction, nor can I calculate the Riemann Hessian matrix. Can you give me a detailed calculation process. thank you very much.
Best,
Xiangfeng

在2021年4月8日星期四 UTC+8 上午11:38:47<jayade...@gmail.com> 写道:

Robert Qiu

unread,
Jul 2, 2021, 7:43:01 AMJul 2
to Manopt

Can you clarify the similarities and differences between Euclidean Hessian and  the direction derivatives of  Euclidean gradient ?

Nicolas Boumal

unread,
Jul 2, 2021, 8:43:09 AMJul 2
to Manopt
Hello,

This question is really about linear algebra and multivariate calculus: the Hessian is a "matrix", sure, but more importantly it is a linear map. That linear map is precisely the directional derivative of the gradient vector field (in the Euclidean case). For the Riemannian case, we use a special notion of derivative for vector fields, but it's the same idea.

My recommendation would be first to make sure you are comfortable with the 'linear map' interpretation of Hessians in the Euclidean case, then that you read up on Riemannian Hessians in Chapter 5 of my book (freely available on my webpage).

Best wishes,
Nicolas
Reply all
Reply to author
Forward
0 new messages