Hessian matrix calculation

[邱知道]

Hello, Nicolas

I want to minimize a cost function: f(s) = s'*C*s*s'*C'*s. 

Its first-order derivative is： df(s) = 2*(s'*C*s*C'*s + s'*C'*s*C*s) and it could be used as 

Problem.egrad = @(s) 2*(s'*C*s*C'*s + s'*C'*s*C*s);

Could you please tell me how to calculate and use its Hessian matrix? I realized there is a '‘H’ in the given examples, like Problem.ehess = @(s, H)...... and I don't know why there is a ’H‘ in the variables.

[Jayadev Naram]

Hello,

     To my understanding ehess function is expected to compute the directional derivative of egrad at a point 's' along a vector 'v'. 

I have written a small code using euclideanfactory manifold. I have assumed that 's' is an n-dimensional real vector. The gradient and hessian checks confirm that my calculations are correct. 

Please correct me if I am wrong.

function test()

    

    n = 100;

    C = magic(n);

    A = C + C';

    manifold = euclideanfactory(n,1);

    problem.M = manifold;

    

    % Note that for a vector s, s'*C*s = s'*C'*s

    problem.cost  = @(s)     1/2*(s'*C*s)^2;

    problem.egrad = @(s)    (s'*C*s)*(A*s);

    problem.ehess = @(s,v) (s'*C*s)*(A*v) + (v'*A*s)*A*s;

    checkgradient(problem); pause;

    checkhessian(problem); pause;

end

Best Regards,

Jayadev Naram.

[邱知道]

Hello,

Yes, your code runs correctly. But when I change   'manifold = euclideanfactory(n,1) ' to 'euclideancomplexfactory(n,1); ', the 'checkhessian(problem)' output a wrong result:

The slope should be 3. It appears to be: 1.

If it is far from 3, then directional derivatives,....

Best,

Xiangfeng

[Jayadev Naram]

Hello Xiangfeng,

     I have corrected the code to work with euclideancomplexfactory. I am attaching the code below. The changes I made are that to treat s'*C*s and s'*C'*s as different entities and carefully evaluate the gradient and the directional derivative of gradient. For numerical reasons, there is a small residue of imaginary part left in computing (s'*C*s)*(s'*C'*s), so I had to return the real part of it. Now the gradient and hessians check pass! 

function test()

    

    n = 100;

    C = rand(n)+rand(1)*1i*rand(n);

    

    manifold = euclideancomplexfactory(n);

    problem.M = manifold;

    

    % Note that s'*C*s is the complex conjugate of s'*C'*s.

    % Yet (s'*C*s)*(s'*C'*s) might have some residual imaginary

    % part, so we have return real part of the cost function.

    problem.cost  = @(s)   real(1/2*(s'*C*s)*(s'*C'*s));

    problem.egrad = @(s)   (s'*C'*s)*(C*s) + (s'*C*s)*(C'*s);

    problem.ehess = @(s,v) (s'*C'*s)*(C*v) + (s'*C'*v)*(C*s) + ...

                           (v'*C'*s)*(C*s) + (s'*C*s)*(C'*v)+ ...

                           (s'*C*v)*(C'*s)+ (v'*C*s)*(C'*s);

    checkgradient(problem); pause;

    checkhessian(problem); pause;

end

Regards,

Jayadev.

[Nicolas Boumal]

Hello everyone,

Jayadev, thanks for weighing in. You can also write the cost function as 1/2*abs(s'*C*s)^2, in which case there is no need to take the real part.

Best,

Nicolas

[邱知道]

Hello, Jayadev.
Thank you very much for your help.

Best,

Xiangfeng

[Robert Qiu]

Hello, Jayadev

Thank you very much for your previous help. But I am still not very clear about the calculation process of the Riemann Hessian matrix. I can only calculate the Euclidean gradient and Euclidean Hessian of the objective function, but I don't know how to calculate the directional derivative of the Euclidean gradient along a certain direction, nor can I calculate the Riemann Hessian matrix. Can you give me a detailed calculation process. thank you very much.

Best,

Xiangfeng

在2021年4月8日星期四 UTC+8 上午11:38:47<jayade...@gmail.com> 写道：

[Robert Qiu]

Can you clarify the similarities and differences between Euclidean Hessian and  the direction derivatives of  Euclidean gradient ?

[Nicolas Boumal]

Hello,

This question is really about linear algebra and multivariate calculus: the Hessian is a "matrix", sure, but more importantly it is a linear map. That linear map is precisely the directional derivative of the gradient vector field (in the Euclidean case). For the Riemannian case, we use a special notion of derivative for vector fields, but it's the same idea.

My recommendation would be first to make sure you are comfortable with the 'linear map' interpretation of Hessians in the Euclidean case, then that you read up on Riemannian Hessians in Chapter 5 of my book (freely available on my webpage).

Best wishes,

Nicolas