Hello Bala,
Thank you for your interest.
This very special problem happens to have a closed-form solution. Indeed, consider expanding the Frobenius norm:
||A - YR||^2 = ||A||^2 + ||YR||^2 - 2 Trace( A^T Y R)
Since ||YR||^2 = ||Y||^2 for all orthogonal matrices R, the two first terms are merely constants, and we are left with the problem of
max Trace( A^T Y R)
subject to R is a rotation.
let A^T Y = USV^T be the SVD decomposition of A^T Y (U and V are orthogonal, S is diagonal, with nonnegative entries).
If we let R = VU^T, then the attained cost is Trace(S). It can be shown that this is indeed maximal.
If VU^T happens to have determinant -1 rather than +1, you can use VJU^T instead, where J = diag(1, 1, ..., 1, -1). This will set the determinant to +1, while minimizing the impact on the cost (since the last diagonal entry of S is also the smallest).
This being said, if you wish to use Manopt to solve this problem (one good reason being that it would give you more freedom in experimenting with other norms, adding extra terms in the cost, having more than one variable, etc.), then please see this example (toward the end of the page):
I believe this is exactly the problem you mentioned. You could have found it from the tutorial page, in the Manifolds section, clicking on the "Rotation group" link ; but admittedly, it is not very visible, and we need to improve on that.
Best,
Nicolas