About the grad norm in process of optimazation by CG on Grassmann manifold？

[grandowife]

I recently read a clip of open source which used ManOpt to optimize a problem defined on Grassmann, and the problem is:

manifold.U = stiefelfactory(d, p);

manifold.E = euclideanfactory(1, n(n-1)/2);

problem.M = productmanifold(manifold);

problem.cost = @objective;

problem.egrad = @grad;

When I customed a specified objective function and gradient function, and chose to use CG optimization, 

I got the information like: 

------------------------------------------------------------------------------

iter                              costVal                             grad,norm

0                            +1.122343e+03               2.6575468e+01

1                            +8.998734e+02               4.0290268e+02

2                            +8.722229e+02               4.0000018e+02

3                            +8.722229e+02               4.0000018e+02

last stepsize smaller than minimum aloowed, options.minstepsize = 1e-10

-------------------------------------------------------------------------------------

And it seems like converged.  But I wanna ask why is the "grad.norm" increase?

Thank you very much

[Nicolas Boumal]

Hello,

Thank you for your message.

A first thing to check is whether the gradient is correct. Could you try to run the following command on your problem structure:

checkgradient(problem);

This will run a test on the gradient. It will be useful if you can copy-paste the output of this command here, so we can see if the gradient is correct.

Also, two comments just in case:

1. Manopt always minimizes. In case you want to maximize, you can replace the cost function with -f(x), and also change the sign of the gradient.
2. You mentioned wanting to optimize over Grassmann, but the code you showed is for a product of Stiefel and Euclidean. In case you meant to use Grassmann instead of Stiefel, you can use grassmannfactory(..).

Best wishes,

Nicolas

[grandowife]

Dear Nicolas,

Happy new year and best wishes!

As your suggestion, I tried checkgradient(problem); and found the responce like:

-----------------------------------------------------

The slope shoud be 2. It appears to be : 0.999995

If it is far from 2, the direction derivatives might be erroneous.

The residual should be 0, or very  close. Residual: 7.97448e-17.

If it is far from 0, the the gradient is not in the tangent space.

---------------------------------------------------

It seems like the direction derivative is calculated wrong? May I ask for some suggestions?

And may I ask what is the meaning of defining a problem equals to the product of two manifolds?

Best wishes

Qiuying Shi

在 2018年12月29日星期六 UTC+8下午6:29:29，Nicolas Boumal写道：

[Nicolas Boumal]

Hello Qiuying,

Happy new year to you too!

Indeed, the slope test did not work out (we find a slope of about 1, whereas we should get about 2). This indicates the gradient is incorrect.

I recommend you go through your derivations again to check the gradient, and if you cannot fix it, feel free to post your code here so we can have a look.

About products: if x lives on the manifold M and y lives on the manifold N, then (x, y) lives on the manifold M x N. To optimize on the manifold M x N in Manopt, you can use the tool MN = productmanifold(elements), where elements.M = ... and elements.N = ... are obtained from factories.

Best,

Nicolas

[grandowife]

Hello Nicolas

I have write you an email and sent it to your gmail box

Look forward for your reply

Best Wishes

QiuyingShi

在 2019年1月2日星期三 UTC+8下午8:10:12，Nicolas Boumal写道：

[grandowife]

Hello Nicolas，

I have rewirte my code for calculating the gradient and objective value, it seems the gradient is successfullty passed the 'checkgradient' function.

Thank you for your help, sincerely.

By the way, I have tried to use 'positivefactory' to define my variable, but Matlab pop an error "undefined function or variable 'postivefactory' ".

Is that my Manopt edition doesn't support this manifold?

Look forward for your reply

Best.

Qiuying Shi

在 2019年1月3日星期四 UTC+8下午3:12:27，grandowife写道：

[Nicolas Boumal]

Hello again,

Good to hear about the gradient being fixed.

For positive factory, notice that you are missing the first 'i' in the name of the factory you call (just a typo).

Best,
Nicolas

[grandowife]

Hi，

Yes, as you say, I make a typo in the last message, but in my code, there is no typo. And I found it leads to an error "undefined function or variable 'postivefactory' ".

I don't know why. Is there any thing else I can make sure the element in a matrix all positive?

Best

在 2019年1月5日星期六 UTC+8上午11:47:58，Nicolas Boumal写道：

[Nicolas Boumal]

Hello, 

Can you try to call importmanopt again, to make sure all folders are on your Matlab path? 

Best, 

Nicolas 

--
http://www.manopt.org
---
You received this message because you are subscribed to the Google Groups "Manopt" group.
To unsubscribe from this group and stop receiving emails from it, send an email to manopttoolbo...@googlegroups.com.
Visit this group at https://groups.google.com/group/manopttoolbox.
To view this discussion on the web visit https://groups.google.com/d/msgid/manopttoolbox/46d217d4-9b7e-4d6b-aaf7-44531e940f43%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[grandowife]

Hello

Follow your  suggestion, I found the reason. It is because I haven't import all the subfolders in Manopt to Matlab and I found this all because the  Manopt  was not been fully unzipped.

Thank you for your suggestions.

By the way, I want to accelerate the process of optimization when using Manopt. I found there is an example named 'usingGPU'. May I please ask some questions?

If there exists many viarables in my task, is it neccessary to turn all these variables to gupArry?If not, what kind of  variables should I turn?

Looking forward for your answers.

Best

Qiuying Shi

在 2019年1月5日星期六 UTC+8下午8:59:37，Nicolas Boumal写道：

[Nicolas Boumal]

Hello Qiuying,

Before turning to GPU, I would recommend spending some time working on the 'normal' version of the code.

In particular, performance can be improved significantly by avoiding loops (for, while) and branching statements (if-else). Matlab offers a lot of functions that can help you avoid these: it is a process called "vectorization".

Another aspect to improve then is: avoiding redundant computations. For example, it is often the case that to compute the gradient, we need a lot of computations that have already been executed to compute the value of the cost function. The Manopt tutorial has some information about caching that can help you reduce redundancy. See also some of the examples.

GPU is a new feature: you are welcome to try it, and do let us know if you run into trouble with it.

Best of luck,

Nicolas

To view this discussion on the web visit https://groups.google.com/d/msgid/manopttoolbox/e3b13024-604f-41bf-b024-ed3fe67b14e7%40googlegroups.com.

[grandowife]

Hi Nicolas

Thank you for your advice, I will try these and keep you posted if I have any questions about using GPU.

Sincerely

Qiuying Shi

 

在 2019年1月6日星期日 UTC+8上午11:01:21，Nicolas Boumal写道：