First, there's a typo in the problem. The problem is (there should be a transpose on the last H_i):
+ sum_i H_i * X * X^T *H_i^T
Thanks for the suggestions. Actually, I've tried to prove its geodesic convexity but failed. What I've done is the following:
In the beginning, I doubt if it is convexity on the manifold, so I looked for some books and papers about that:
Absil, P-A., Robert Mahony, and Rodolphe Sepulchre. "Optimization algorithms on matrix manifolds." Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2009.
Vishnoi, Nisheeth K. "Geodesic convex optimization: Differentiation on manifolds, geodesics, and convexity." arXiv preprint arXiv:1806.06373 (2018).
Ferreira, O. P., A. N. Iusem, and S. Z. Németh. "Concepts and techniques of optimization on the sphere." Top 22 (2014): 1148-1170.
Then I tried to prove its convexity by plugging in the formula of the geodesic on the sphere. But I was not able to show it after struggling with the complicated and messy formula for several days. Then I noticed that if X is a solution to the problem, then XQ is also a solution to it when Q is an arbitrary orthogonal matrix, so I turned to the quotient manifold, spectrahedron in our case, i.e., sphere manifold with the orthogonal group. After getting familiar with the basic knowledge of quotient manifolds such as horizontal space, I started to prove its geodesic convexity on the quotient manifold. But again I failed because the proof became much more complicated than before.
After that I came up with this paper:
Journée, Michel, et al. "Low-rank optimization on the cone of positive semidefinite matrices." SIAM Journal on Optimization 20.5 (2010): 2327-2351.
And it gives me the following insight:
If X of size MxS (S<M) is the solution to the problem on spectrahedron(M, S), i.e., the Riemannian gradient vanishes at X when we apply gradient descent methods, then the matrix X'=[X,0] (of size M x S+1) by appending a column of all 0 to X will also result in a vanishing Riemannian gradient on spectrahedron(M, S+1), but X' is usually not a solution to that problem. That means, the problem has other points at which the Riemannian gradient is zero but they might not be optimum. This brings me to the conjecture that the problem is not geodesic convex. But the simulation always converges to the same result instead of those stationary points, which makes me so curious about the properties of the problem.
Then I thought of the pseudoconvexity and wondered if it can be generalized to Riemannian manifolds. But I must admit that it is beyond my knowledge level and might have no time to dive into it because I spent one month on this single problem and I have to return to my research project....
If you could provide further insights about it, I'll appreciate it so much! If not, it doesn't matter and thank you all the same, and I'll come back to this problem after finishing my current project. I'll read your suggested book for a deeper understanding of Riemannian geometry. Thank you so much!