Yalmip is not working anymore after Matlab Update!

[Nabil]

Hello Everybody!
I has been happily using Yalmip (Such a good piece of SW) for a while with no problems in MATLAB Version 7.9.0.529 (R2009b). However, I took the "brillant" idea to update it to MATLAB Version: 7.14.0.739 (R2012a). The IT guy who did the dirty job of updating the SW misteriously left operative both MATLAB versions. Anyway, stop the drama, the fact is that now YALMIP does not work properly in any of the versions anymore. I have the following output for yalmiptest command:

Test| Solution| Solver message|
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++| sdpvar/set methods| N/A| Successfully solved (YALMIP)| LP| N/A| Unknown problem in solver (try using 'debug'-flag in sdpsettings) (Undefined function or method 'linprog' for input arguments of type 'struct'.)|
| LP| N/A| Unknown problem in solver (try using 'debug'-flag in sdpsettings) (Undefined function or method 'linprog' for input arguments of type 'struct'.)|
| QP| N/A| Unknown problem in solver (try using 'debug'-flag in sdpsettings) (Undefined function or method 'quadprog' for input arguments of type 'struct'.)|
| QP| N/A| Unknown problem in solver (try using 'debug'-flag in sdpsettings) (Undefined function or method 'quadprog' for input arguments of type 'struct'.)|
| SOCP| Correct| Successfully solved (SeDuMi-1.3)|
| SOCP| Correct| Successfully solved (SeDuMi-1.3)|
| SOCP| Correct| Successfully solved (SeDuMi-1.3)|
| SDP| Correct| Successfully solved (SeDuMi-1.3)|
| SDP| Correct| Successfully solved (SeDuMi-1.3)|
| SDP| Correct| Successfully solved (SeDuMi-1.3)|
| SDP| Correct| Successfully solved (SeDuMi-1.3)|
| MAXDET| Correct| Successfully solved (SeDuMi-1.3)|
| MAXDET| Correct| Successfully solved (SeDuMi-1.3)|
| Infeasible LP| N/A| Unknown problem in solver (try using 'debug'-flag in sdpsettings) (Undefined function or method 'linprog' for input arguments of type 'struct'.)|
| Infeasible QP| N/A|Unknown problem in solver (try using 'debug'-flag in sdpsettings) (Undefined function or method 'quadprog' for input arguments of type 'struct'.)|
| Infeasible SDP| N/A| Infeasible problem (SeDuMi-1.3)|
| Moment relaxation| Correct| Successfully solved (SeDuMi-1.3)|
| Sum-of-squares| Correct| Successfully solved (SeDuMi-1.3)|
| Bilinear SDP| N/A| No suitable solver|

When I use my codes (they used to work perfectly using fmincom as solver) i get the solver output (on the info field) error message:
'Unknown problem in solver (try using 'debug'-flag in sdpsettings) (Reference to non-existent field 'Algorithm'.)'
Running the same problem with the debug flag enable as suggested, I get:
Sol=solvesdp(Const,Obj,opts)
??? Reference to non-existent field 'Algorithm'.

Error in ==> callfmincon at 39
if strcmp(model.options.fmincon.Algorithm,'trust-region-reflective')

Error in ==> solvesdp at 337
eval(['output = ' solver.call '(interfacedata);']);

Any idea to overcome this problem>. Thank you so much for your support.

Cheers,
Nabil

[Johan Löfberg]

Somehow it looks like YALMIP thinks you have the Optimization toolbox, but in some way it doesn't work

What happens if you run

which linprog
linprog(1,-1,1)
O = optimset('fmincon');O.Algorithm

[Nabil]

Interesting,
The output is:

linprog(1,-1,1)
O = optimset('fmincon');O.Algorithm

C:\MATLAB\R2009b\toolbox\optim\optim\linprog.m  % Has no license available
??? Undefined function or method 'linprog' for input arguments of type 'double'.
 
It seems that after the update the function is not available anymore due to a license restriction.  Is this possible?

 

More interesting info... if I just type fmincom i get:

 

??? License checkout failed.
License Manager Error -18
Make sure the license file on the MATLAB client matches the license file of the license server and that the license server has been started properly.

Troubleshoot this issue by visiting:
http://www.mathworks.com/support/lme/R2009b/18

Diagnostic Information:
Feature: Optimization_Toolbox
License path: 17...@eensun5.XXXXX.ac.uk;C:\Documents and Settings\leonabil\Application
Data\MathWorks\MATLAB\R2009b_licenses;C:\MATLAB\R2009b\licenses\license.dat;C:\MATLAB\R2009b\licenses\network.lic
FLEXnet Licensing error: -18,147.

[Johan Löfberg]

Yes, you've lost your license.

An even worse problem is the fact that you have code from R2009 in your path (assuming you did this in R2012a). If that is the case, you have to clean up your system. Things will be very unpredicable if you are mixing versions.

[Nabil]

This is really getting messy.  The last result was obtanind in R2009b version.  Now I have run the same in the R2012b version and I get this:

>> which linprog

linprog(1,-1,1)
O = optimset('fmincon');O.Algorithm

C:\MATLAB_R2012a\toolbox\optim\optim\linprog.m
Undefined function 'optimget' for input arguments of type 'struct'.

Error in linprog (line 167)
largescale = strcmpi(optimget(options,'LargeScale',defaultopt,'fast'),'on');
 
>> fmincon
Error using fmincon (line 214)
FMINCON requires at least four input arguments.

 

In this version the license is working, (that makes sense cause is the updated version) but still cannot solve the problem and returns an error!!!

[Johan Löfberg]

>> which optimget
C:\Program Files\MATLAB\R2012a\toolbox\matlab\optimfun\optimget.m

Your path is probably not complete (if I recall it correctly, we had similiar issues here). Add the \toolbox\matlab\optimfun directory to your path and see if it works then.

[Nabil]

It seems that is an error with the installation of the optimization toolbox!!, the output is:

which optimget

C:\MATLAB_R2012a\toolbox\sldo\sloptim\sloptim\@ResponseOptimizer\optimget.m  % ResponseOptimizer method

 

I cannot add this directory to the path for the @

 

Just for trying, I have added to the path the directory where the old version (R2009b) has this file I mean:

C:\MATLAB\R2009b\toolbox\matlab\funfun\optimget.m

Now the output is:

>> which linprog
linprog(1,-1,1)
O = optimset('fmincon');O.Algorithm
C:\MATLAB_R2012a\toolbox\optim\optim\linprog.m

Optimization terminated.

ans =

   -1.0000

Warning: Warning: feval on script names will not work, or may work differently,
 in a future version of MATLAB.  To make your code insensitive to any change
 and to suppress this warning message:
  - Either change the script to a function.
  - Or use eval instead of feval.
 The script file in question is fmincon.
> In optimset at 141
Error using optimset (line 143)
No default options available for the function 'fmincon'.

 

;-(

 

 

 

[Johan Löfberg]

I don't understand why you cannot add

C:\MATLAB_R2012a\toolbox\matlab\optimfun\

to your path

[Nabil]

I have done it!!!  however i got an error related with optimset!  This is the uotput:

 

which linprog
linprog(1,-1,1)
O = optimset('fmincon');O.Algorithm

C:\MATLAB_R2012a\toolbox\optim\optim\linprog.m
Optimization terminated.

ans =

   -1.0000

Warning: Warning: feval on script names will not work, or may work differently,
 in a future version of MATLAB.  To make your code insensitive to any change
 and to suppress this warning message:
  - Either change the script to a function.
  - Or use eval instead of feval.
 The script file in question is fmincon.

> In optimset at 135
Error using optimset (line 137)

[Johan Löfberg]

Looks like your fmincon does not work as expected. Result from this?

which optimset -all
which fmincon -all
fmincon('defaults')

[Nabil]

Now it seems that is working partially.  Yalmiptest command runs ok.  It was a problem in the active path that was in conflict with some directories of the previous version.  I reset the path to the default values and added the needed directories.

However, my codes do not run.  Something very odd is happennig.  I run my codes (they worked perfectly in the previous version) and I get the same error than before.  I mean the unknown problem in solver.  The odd thing is that If i run yalmiptest command, the next execution works fine.  The following one does not run.  I guess that is something that is innitialized by the yalmiptest routine that is lost when i execute my codes.  Running wiht the debug utility I get this message:

 

>>Sol=solvesdp(Const,Obj,opts)
Error using barrier (line 26)
Nonlinear constraint function is undefined at initial point. Fmincon cannot continue.

Error in fmincon (line 841)
    [X,FVAL,EXITFLAG,OUTPUT,LAMBDA,GRAD,HESSIAN] = barrier(funfcn,X,A,B,Aeq,Beq,l,u,confcn,options.HessFcn, ...

Error in callfmincon (line 66)
[xout,fmin,flag,output,lambda] =
fmincon('fmincon_fun',model.x0,model.A,model.b,model.Aeq,model.beq,model.lb,model.ub,callback_con,model.options.fmincon,model);

Error in solvesdp (line 337)

    eval(['output = ' solver.call '(interfacedata);']);

 

However, I run yalmiptest, and solving exactly the same problem returns a satisfactory answer:

>> Sol=solvesdp(Const,Obj,opts)
Optimization terminated.

Sol =

    yalmiptime: 0.0470
    solvertime: 0.0160
          info: 'Successfully solved (LINPROG)'
       problem: 0
        dimacs: [NaN NaN NaN NaN NaN NaN]

 

I really do not understand what is happening.  My codes used to work perfectly in the matlab previous version.  Do you have any idea what might be happening or something that I am missing.  Thank you so much for your help, I really appreciatte it!

[Johan Löfberg]

I don't understand what you mean. Is the the order and actions

1. Define problem P. solvesdp crashes in fmincon
2. runs yalmiptest
3. Define problem P. Now it succeeds, but calls linprog

[Johan Löfberg]

BTW, after having had a very messy setup, I hope you have restarted MATLAB.

[Nabil]

Indeed I restarted Matlab many times. And yes basically what you have commented summarizes the problem

1. define a problem - it crashes with fmincon

2. i run yalmiptest - runs ok

3. I run the command only to solve the problem i.e., Sol=solvesdp(Const,Obj,opts) and it is solved by LINPROG.

4. I run a new iteration and crashes again with fmincon.

 

It is very odd. Any idea?

 

[Johan Löfberg]

I need to see your code

[Johan Löfberg]

The Const and Obj objects are messed up entirely when you run yalmiptest (it performs yalmip('clear'))

By pure luck, it somehow survives partially and linprog is called on some weird remaining structures.

Still, to understand your underlying problem with fmincon, I need the code.

[Nabil]

The code is this:

 

N=1; Nt=5; Nr=1; std_dev_hb=1; std_dev_he=1; std_dev_nb=1; std_dev_ne=1;
rb=1; alpha=2; Pmax=20; R=1; Beta=0.75; rsmin=0.01; rsmax=1; K1=1; K3=1;
hb=(std_dev_hb/sqrt(2))*(randn(Nt,Nr,N)+1i*randn(Nt,Nr,N));
sdpvar a b rs;
c2=1;
Const=[a>=(2^R-1)*rb^alpha*std_dev_nb^2/abs(hb(:,:,c2)'*hb(:,:,c2)),
        (1-(1/(1+b/a*(a*abs(hb(:,:,c2)'*hb(:,:,c2))+rb^alpha*std_dev_nb^2*(1-2^R))/((Nt-1)*2^R*rb^alpha*std_dev_nb^2))^(Nt-1))*exp(-rs^alpha*std_dev_ne^2*(rb^alpha*std_dev_nb^2*(1-2^R)+a*abs(hb(:,:,c2)'*hb(:,:,c2)))/(2^R*a*rb^alpha*std_dev_nb^2*std_dev_he^2)))>=Beta,
        b>=0,
        a+b<=Pmax,
        rs>=rsmin,
        rs<=rsmax];
    Obj=K1*(a+b)/Pmax+K3*rs/rsmax;
    opts=sdpsettings('debug','flag');
    Sol=solvesdp(Const,Obj,opts)

 

What I do is to run this, i get the error, then I run yalmiptest, and then whitout clearing the variables or doing nothing I run  Sol=solvesdp(Const,Obj,opts).

 I reckon that is a matter of luck, but the question is why it happens and it could give some light about what is happening.  Btw this code was working perfectly before the update.

Thanks again

[Johan Löfberg]

The fact that it solves it after running yalmiptest is entirely unrelated to your issues. When yalmip('clear') is performed and new variables are defined inside yalmiptest, your nonlinear expressions dissapear and reference completely random stuff inside yalmiptest instead. That is why linprog is called instead all of a sudden.

[Nabil]

I have forced the tool to use fmincon, and also works after the yalmiptest.  So it does no matter the solver, after the yalmiptest execution, the problem can be solved even by fmincon.

Odd!!

opts=sdpsettings('debug','flag','solver','fmincon');
    Sol=solvesdp(Const,Obj,opts)

Sol =

    yalmiptime: 0.2650
    solvertime: 0.3910
          info: 'Successfully solved (FMINCON)'

[Johan Löfberg]

Well, not really to surprising, the completely random problem you happen to solve is a simple LP, so fmincon has no issues with such a trivial problem.

Anyway, your problem fails in R2012a here too. It runs in 2011a though. The difference turns out to be the choice of algorithm. The default in R2012a is 'trust-region-reflective' which has problems when the initial guess isn't feasible. You haven't supplied an initial guess, so YALMIP sends 0.

In previous version of MATLAB, I guess the default choice of algorithm was probably different, not causing this issue.

This seems to work

opts=sdpsettings('debug','flag','fmincon.Algorithm','active-set');

[Nabil]

I see your point.  However the vaiables and constraint that i defined previusly seem to remain.  The difference between before and after the yalmiptest execution is the internal structure of the Constraints:

Before

 

Consti
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|   ID|      Constraint|                                      Type|
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|   #1|   Numeric value|               Element-wise inequality 1x1|
|   #2|   Numeric value|   Element-wise inequality (sigmonial) 1x1|
|   #3|   Numeric value|               Element-wise inequality 1x1|
|   #4|   Numeric value|               Element-wise inequality 1x1|
|   #5|   Numeric value|               Element-wise inequality 1x1|
|   #6|   Numeric value|               Element-wise inequality 1x1|
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 

After

+++++++++++++++++++++++++++++++++++++++++++++++++++++++
|   ID|      Constraint|                          Type|
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
|   #1|   Numeric value|   Element-wise inequality 1x1|
|   #2|   Numeric value|   Element-wise inequality 1x1|
|   #3|   Numeric value|   Element-wise inequality 1x1|
|   #4|   Numeric value|   Element-wise inequality 1x1|
|   #5|   Numeric value|   Element-wise inequality 1x1|
|   #6|   Numeric value|   Element-wise inequality 1x1|
+++++++++++++++++++++++++++++++++++++++++++++++++++++++

 

What I don't understand is why running the yalmiptest changes the constraints type. ;-(

Thanks for your help again!!!

[Nabil]

Awesome.  It is working!!!  Thank you so much for all your support. 

[Johan Löfberg]

Because YALMIP variables are essentially pointers to things inside the yalmip core

yalmip('clear')
x = sdpvar(1)
y = sdpvar(1)
z = x*x*y

YALMIP has no notion of variable names, but only indicies. YALMIP core has three variables, 1,2,3. You can see these numbers through getvariables(x) etc

In addition to variable indicies, there is a so called monomial table inside the core which describe nonlinearirites. At this point this matrix is

[1 0 0;
 0 1 0;
 2 1 0]

meaning first variable is first variable^1, second is second^1 and the third variable is first^2*second, i.e. a monomial

Now we run yalmip('clear');q = sdpvar(3,1) in an external function. That function deletes everything in the core, and create 3 new variables, and the monomial table will be the identity.

Now we return to the original work-space. We still have three variables, pointing at 1,2,3. However, the core in YALMIP now has a monomial table which is identity, hence x,y and z are considered linear, the connection between variable 3 and the two others are lost. minimizing z will simply be minimizing a linear function.

[Nabil]

I see, that is why the internal structure changed.  Indeed the returned results made non sense.  It was confusing though.  Thank you so much for the clarification and the help.  Now my simulations are on track again and I have learned a bit about the internal processing of YALMIP.

Fantastic support!

Many thanks!