Uninformative error message when having free variables in NLP functions

[Fredrik Magnusson]

I hope that the following code and the printed error messages will sufficiently explain my issue.

x = msym("x")
y = msym("y")
f = MXFunction([x], [y])
f.init()
a = msym("a")
[f_call] = f.call([a])
nlp = MXFunction(nlpIn(x=a), nlpOut(f=f_call))
solver = IpoptSolver(nlp)
solver.init()
solver.solve() # Gives a proper error message

This piece of code gives the following error message:

This is Ipopt version 3.11.3, running with linear solver ma27.

Number of nonzeros in equality constraint Jacobian...:        0
Number of nonzeros in inequality constraint Jacobian.:        0
Number of nonzeros in Lagrangian Hessian.............:        0

eval_grad_f failed:  on line 393 of file "/work/fredrikm/NewCasADiInterface2/ThirdParty/CasADi/CasADi/symbolic/fx/mx_function_internal.cpp"
Cannot evaluate "function("unnamed_mx_function")" since variables [MX(y)] are free.

Error evaluating objective gradient at user provided starting point.
  No scaling factor for objective function computed!
Total number of variables............................:        1
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        0
                     variables with only upper bounds:        0
Total number of equality constraints.................:        0
Total number of inequality constraints...............:        0
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

eval_grad_f failed:  on line 393 of file "/work/fredrikm/NewCasADiInterface2/ThirdParty/CasADi/CasADi/symbolic/fx/mx_function_internal.cpp"
Cannot evaluate "function("unnamed_mx_function")" since variables [MX(y)] are free.


Number of Iterations....: 0

Number of objective function evaluations             = 0
Number of objective gradient evaluations             = 1
Number of equality constraint evaluations            = 0
Number of inequality constraint evaluations          = 0
Number of equality constraint Jacobian evaluations   = 0
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations             = 0
Total CPU secs in IPOPT (w/o function evaluations)   =      0.000
Total CPU secs in NLP function evaluations           =      0.000

EXIT: Invalid number in NLP function or derivative detected.
time spent in eval_f: 0 s.
time spent in eval_grad_f: 0 s.
time spent in eval_g: 0 s.
time spent in eval_jac_g: 0 s.
time spent in eval_h: 0 s.
time spent in main loop: 0 s.
time spent in callback function: 0 s.
time spent in callback preparation: 0 s.

So far, so good. However, when I change the dimension of the decision variable, it is no longer possible to initialize the IpoptSolver object, which returns a strange error message.

x = msym("x", 2)
y = msym("y")
f = MXFunction([x], [y])
f.init()
a = msym("a", 2)
[f_call] = f.call([a])
nlp = MXFunction(nlpIn(x=a), nlpOut(f=f_call))
solver = IpoptSolver(nlp)
solver.init() # Gives an uninformative error message

This second piece of code gives the following error message:

ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (28110, 0))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
    173             else:
    174                 filename = fname
--> 175             __builtin__.execfile(filename, *where)

/work/fredrikm/U013/src/free_mx.py in <module>()
     20 nlp = MXFunction(nlpIn(x=a), nlpOut(f=f_call))
     21 solver = IpoptSolver(nlp)
---> 22 solver.init() # Gives an uninformative error message

/work/fredrikm/NewCasADiInterface2-install/Python/casadi/casadi.pyc in init(self, allow_reinit)
   4700
   4701         """
-> 4702         return _casadi.SharedObject_init(self, allow_reinit)
   4703
   4704     def isInit(self):

RuntimeError: The assertion "(sparsity().scalar(false) || y.scalar() || (sparsity().size1()==y.size1() && size2()==y.size2()))" on line 472 of file "/work/fredrikm/NewCasADiInterface2/ThirdParty/CasADi/CasADi/symbolic/mx/mx_node.cpp" failed.
Dimension mismatch.lhs is 2-by-1 (0/2 nz), while rhs is 0-by-0 (dense)

I would have expected (and desired) pretty much the same behavior as in the first piece of code.

[Joel Andersson]

OK, thank you for reporting! I've created an issue for this https://github.com/casadi/casadi/issues/966. Joel

[Fredrik Magnusson]

The actual problem I am experiencing seems to be unrelated to free variables.

x = msym("x", 3)
y = msym("y", 2)
f = MXFunction([x, y], [1.])
f.init()
aa = msym("aa", 5)
a = aa[:3]
b = aa[3:]
[f_call] = f.call([a, b])
nlp = MXFunction(nlpIn(x=aa), nlpOut(f=f_call))
solver = IpoptSolver(nlp)
solver.init() # Gives the error message below

ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (28110, 0))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
    173             else:
    174                 filename = fname
--> 175             __builtin__.execfile(filename, *where)

/work/fredrikm/U013/src/free_mx.py in <module>()

     11 nlp = MXFunction(nlpIn(x=aa), nlpOut(f=f_call))
     12 solver = IpoptSolver(nlp)
---> 13 solver.init() # Gives the error message below

/work/fredrikm/jmodelica-install/Python/casadi/casadi.pyc in init(self, allow_reinit)

   4700
   4701         """
-> 4702         return _casadi.SharedObject_init(self, allow_reinit)
   4703
   4704     def isInit(self):

RuntimeError: The assertion "(sparsity().scalar(false) || y.scalar() || (sparsity().size1()==y.size1() && size2()==y.size2()))" on line 472 of file "/work/fredrikm/jmodelica/ThirdParty/CasADi/CasADi/symbolic/mx/mx_node.cpp" failed.
Dimension mismatch.lhs is 3-by-1 (0/3 nz), while rhs is 0-by-0 (dense)

I do not see why this should not work. Is it a bug? If not, can you please explain why it does not work?

[Joel Andersson]

Well, an uninformative error message is a bug by default.

I am not quite sure what is going on. One theory would be that the gradient of the objective is calculated, which is just zero since `x` doesn't enter in the objective. And that the interface does not have proper error handling for zero gradients. But this is just speculation from my part. Anyway, the gradient is zero because of the free variables in some sense, so https://github.com/casadi/casadi/issues/966 would still be relevant then.

Joel

[Joel Andersson]

I was a bit too quick to answer. As you point out, this has nothing to do with free variables, it most probably has to do with a bug in the NLP solver base class. I have create a separate issue for this: https://github.com/casadi/casadi/issues/970. I suggest continuing the discussion there.