EXIT: Restoration Failed! Ipopt 3.12.8: Restoration Phase Failed.

[Mirou]

hello

i need your help have a problem  when i use the option  ipopt_options 'hessian_approximation=limited-memory'

Objective...............:  -0.0000000000000000e+00   -0.0000000000000000e+00
Dual infeasibility......:   1.0000000000000000e+00    1.0000000000000000e+00
Constraint violation....:   4.2399999996999998e+02    5.5618866427394028e+02
Complementarity.........:   4.1573000847000003e+02    4.1573000847000003e+02
Overall NLP error.......:   4.2399999996999998e+02    5.5618866427394028e+02


Number of objective function evaluations             = 2
Number of objective gradient evaluations             = 2
Number of equality constraint evaluations            = 2
Number of inequality constraint evaluations          = 2
Number of equality constraint Jacobian evaluations   = 2
Number of inequality constraint Jacobian evaluations = 2
Number of Lagrangian Hessian evaluations             = 0
Total CPU secs in IPOPT (w/o function evaluations)   =     26.240
Total CPU secs in NLP function evaluations           =      0.276

EXIT: Restoration Failed!
 
Ipopt 3.12.8: Restoration Phase Failed.

suffix ipopt_zU_out OUT;
suffix ipopt_zL_out OUT;
dridi@calcul:~/code/Jstagefixe$

but i can't know the origin of this problem.


cordially

[ptiwari]

IPOPT throws restoration phase failed message under a number of circumstances. It could be the problem is really nasty, derivatives are incorrect, and the problem is badly scaled... You could try to use the exact second order derivative method instead of the approximate one. Moreover, there is a IPOPT mailing list https://list.coin-or.org/mailman/listinfo/ipopt and they could provide better help problems related to IPOPT solver. 

[Mirou]

thank you for your reply, but i try without this option but hi exceeds 3000 itiration without solution.

[ptiwari]

IPOPT converges faster when you use exact second derivative. It doesn't make sense that IPOPT takes more number of iterations when exact derivative was used than when hessian approximation was used. Double check IPOPT options and make sure that IPOPT is using the exact second order derivative of your problem formulation.

[Mirou]

thank for you advice but i try to chek up the perform test of first and second derivatives then i used this option 'derivative_test=second-order'  but i have any information for the derivate, also if my problem does not satisfy the constraint qualification how i cant know?

thank you
 
Le vendredi 26 janvier 2018 03:25:40 UTC+1, ptiwari a écrit

[ptiwari]

When you use AMPL, you don't need to check derivative because AMPL is calculating derivative for you. So, you can turn off the IPOPT derivative checker. I believe IPOPT  assumes  the gradients of the active constraints are linearly independent. I recommend to ask to IPOPT forum about what exactly constraint qualification mean for IPOPT.

Thanks,

Paras

[coolpart...@gmail.com]

Hi

I'm also facing this issue. I'm new to this ipopt optimization.

But my case, the problem is solved by adding ipopt_options 'hessian_approximation=limited-memory' in my local environment as well as local docker setup.

When I run the optimization in docker at server(aws instance) throws another error as,

"WARNING: Problem in step computation; switching to emergency mode.
Cholesky factorization failed for LBFGS update! Skipping update."

Question:
1. If at all "the problem is really nasty, derivatives are incorrect, and the problem is badly scaled" means why its running in local docker setup not in server?

2. How to resolve "Cholesky factorization failed" error?

Note: Ipopt version 3.12.4

Thanks,
Parthipan.E

[AMPL Google Group]

Have you run your problem using the exact second order derivative? IPOPT converges in fewer iterations and is more robust when you use the exact second order derivative rather than the approximate one.

--
Paras Tiwari
am...@googlegroups.com

{#HS:520524496-306#}

--
You received this message because you are subscribed to the Google Groups "AMPL Modeling Language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ampl+uns...@googlegroups.com.
To post to this group, send email to am...@googlegroups.com.
Visit this group at https://groups.google.com/group/ampl.
For more options, visit https://groups.google.com/d/optout.

[coolpart...@gmail.com]

Thanks Paras.

Yes I did. That is my first troubleshoot.

I don't understand why its working in local docker not in server. I have checked the memory, CPU cores and python libraries in both machine. I couldn't find the difference.

Is there any hardware specification related to this issue? if its there means, how can I find ?

Immediate help will be appreciated. I have been struggling for a week.


Please check the below log which is using 'second-order" derivative.

Forming constraint system for linear solver.
Optimization terminated successfully.
         Current function value: -1263783.783784
         Iterations: 18
Making constraint system for optimizer.
check derivatives for NaN/inf

List of user-set options:

                                    Name   Value                used
                         derivative_test = second-order          yes
                      print_user_options = yes                   yes

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit http://projects.coin-or.org/Ipopt
******************************************************************************

This is Ipopt version 3.12.4, running with linear solver mumps.
NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

Starting derivative checker for first derivatives.

Starting derivative checker for second derivatives.


No errors detected by derivative checker.

Number of nonzeros in equality constraint Jacobian...:        0
Number of nonzeros in inequality constraint Jacobian.:       66
Number of nonzeros in Lagrangian Hessian.............:       33

Total number of variables............................:       33
                     variables with only lower bounds:        0
                variables with lower and upper bounds:       33
                     variables with only upper bounds:        0
Total number of equality constraints.................:        0
Total number of inequality constraints...............:       34
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:       34
        inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0 -2.2187422e+01 0.00e+00 1.57e-06  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
   1 -4.4859492e+01 0.00e+00 5.61e+00  -1.0 2.23e+00    -  1.33e-01 1.00e+00f  1
   2 -5.7370545e+01 0.00e+00 2.74e-02  -1.0 2.57e+00    -  9.79e-01 1.00e+00f  1
   3 -1.9328700e+02 0.00e+00 6.47e-01  -1.0 1.23e+02    -  4.09e-02 1.00e+00f  1
   4 -2.5722470e+02 0.00e+00 5.06e-04  -1.0 1.28e+02    -  9.81e-01 1.00e+00f  1
   5 -1.2171548e+03 0.00e+00 1.17e-02  -1.0 6.47e+03    -  3.89e-02 1.00e+00f  1
   6 -1.7674873e+03 0.00e+00 8.84e-06  -1.0 6.54e+03    -  9.82e-01 1.00e+00f  1
   7 -1.0334649e+04 0.00e+00 1.79e-04  -1.0 2.55e+05    -  5.09e-02 1.00e+00f  1
   8 -1.3990191e+04 0.00e+00 8.31e-07  -1.0 1.77e+05    -  9.90e-01 1.00e+00f  1
   9 -2.3893696e+04 0.00e+00 6.28e-07  -1.0 7.81e+05    -  4.90e-01 1.00e+00f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  10 -2.3429114e+04 0.00e+00 4.86e-09  -1.0 3.12e+04    -  9.93e-01 1.00e+00f  1
  11 -2.3429187e+04 0.00e+00 1.53e-16  -1.7 9.88e+01    -  1.00e+00 1.00e+00h  1
  12 -2.3431034e+04 0.00e+00 1.40e-17  -3.8 2.82e+01    -  1.00e+00 1.00e+00h  1
  13 -2.3674418e+04 0.00e+00 5.74e-08  -5.7 3.88e+03    -  9.58e-01 1.00e+00h  1
  14 -2.7007527e+04 0.00e+00 1.38e-08  -8.6 5.91e+04    -  8.12e-01 9.90e-01f  1
  15 -2.9146468e+04 0.00e+00 4.80e-09  -8.6 6.12e+04    -  6.56e-01 1.00e+00f  1
  16 -2.9962451e+04 0.00e+00 1.37e-09  -8.6 2.57e+04    -  7.15e-01 1.00e+00f  1
  17 -3.0164781e+04 0.00e+00 3.30e-10  -8.6 1.32e+04    -  7.59e-01 1.00e+00h  1
  18 -3.0200785e+04 0.00e+00 8.27e-11  -8.6 6.30e+03    -  7.49e-01 1.00e+00h  1
  19 -3.0207674e+04 0.00e+00 2.47e-12  -8.6 2.53e+03    -  1.00e+00 1.00e+00h  1
In iteration 19, 1 Slack too small, adjusting variable bound
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  20 -3.0213890e+04 0.00e+00 2.70e-12 -12.9 1.58e+03    -  4.73e-01 9.94e-01f  1
  21           -nan 0.00e+00 1.07e-13 -12.9 4.59e+02    -  9.67e-01 9.95e-01h  1

WARNING: Problem in step computation; switching to emergency mode.

Restoration phase is called at point that is almost feasible,
  with constraint violation 0.000000e+00. Abort.

Number of Iterations....: 21

                                   (scaled)                 (unscaled)
Objective...............:                     -nan                      -nan
Dual infeasibility......:   1.0671229803429667e-13    1.0671229803429666e-05
Constraint violation....:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   5.0894571745858957e-10    5.0894571745858957e-02
Overall NLP error.......:   5.0894571745858957e-10    5.0894571745858957e-02


Number of objective function evaluations             = 22
Number of objective gradient evaluations             = 22
Number of equality constraint evaluations            = 0
Number of inequality constraint evaluations          = 22
Number of equality constraint Jacobian evaluations   = 0
Number of inequality constraint Jacobian evaluations = 22
Number of Lagrangian Hessian evaluations             = 22
Total CPU secs in IPOPT (w/o function evaluations)   =      0.032
Total CPU secs in NLP function evaluations           =      0.000

EXIT: Restoration Failed!
irrelevant_keys []
keys_to_ignore []
first_pass_spends {(5152,): 58497.163779327173, (5116,): 0.0, (5143,): 25864.105465434968, (5154,): 18350.519755692094, (5118,): 21782.197162133925, (5129,): 946.76139106706364, (5156,): 130067.56817085089, (5131,): 8410.0427112657162, (5158,): 35472.557118652163, (5121,): 70945.225211223107, (5148,): 10681.558729465383, (5123,): 62713.311945559057, (5150,): 112235.35462301216, (5161,): 14189.18915642108, (5125,): 2837.8376675429026, (5136,): 2364.8645180980207, (5138,): 9135.0943804430553, (5115,): 79317.656528686508, (5142,): 1965.4728434628942, (5153,): 117633.05536988769, (5117,): 6848.5409452111408, (5119,): 5314.7311784192461, (5130,): 3260.6642555865692, (5157,): 17213.447898561404, (5134,): 44723.307722005164, (5145,): 7094.5940937729611, (5147,): 118027.81749455762, (5122,): 3362.9311640433921, (5160,): 56756.75633403039, (5124,): 189189.13454155941, (5126,): 9783.9034442970078, (5137,): 14188.970578843124, (5139,): 4609.4586518808801}
initial_promote_keys [(5152,), (5143,), (5154,), (5156,), (5158,), (5121,), (5148,), (5150,), (5161,), (5136,), (5138,), (5142,), (5153,), (5157,), (5134,), (5145,), (5147,), (5160,), (5124,), (5137,), (5139,)]
knapsack_keys [(5118,), (5123,), (5130,), (5122,)]
knapsack_spend_keys [(5130,), (5123,), (5118,), (5122,)]
spend_for_knapsack 91119.1045273
included_tactics [0, 2, 3]
total_spend 69192.5346922
included_knapsack_keys [(5118,), (5130,), (5122,)]
max_spend 91119.1045273
knapsack_spend_dict {(5130,): 14988.447581081082, (5123,): 101449.98159694197, (5118,): 38616.681745171445, (5122,): 15587.405365897703}
demotion_keys [(5123,)]
final constraint_dict {(5130,): {'peg': None, 'bounds': (14988.447581081082, 59953.79032432433)}, (5123,): {'peg': None, 'bounds': (0, 0)}, (5118,): {'peg': None, 'bounds': (38616.681745171445, 154466.72698068578)}, (5122,): {'peg': None, 'bounds': (15587.405365897703, 236486.4864864865)}}
Making constraint system for optimizer.
check derivatives for NaN/inf

List of user-set options:

                                    Name   Value                used
                         derivative_test = second-order          yes
                      print_user_options = yes                   yes
This is Ipopt version 3.12.4, running with linear solver mumps.
NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

Starting derivative checker for first derivatives.

Starting derivative checker for second derivatives.


No errors detected by derivative checker.

Number of nonzeros in equality constraint Jacobian...:        1
Number of nonzeros in inequality constraint Jacobian.:       66
Number of nonzeros in Lagrangian Hessian.............:       33

Total number of variables............................:       33
                     variables with only lower bounds:        0
                variables with lower and upper bounds:       33
                     variables with only upper bounds:        0
Total number of equality constraints.................:        1
Total number of inequality constraints...............:       34
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:       34
        inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0 -2.2187422e+01 3.97e+04 1.57e-06  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
   1 -2.4449351e+01 3.97e+04 1.54e+00  -1.0 4.01e+04    -  6.30e-03 9.80e-03h  1
   2 -2.3133309e+01 3.97e+04 1.95e+00  -1.0 3.97e+04    -  2.33e-06 9.90e-05h  1
   3 -2.6047654e+01 3.97e+04 2.05e+00  -1.0 1.82e+05    -  8.64e-07 1.01e-06f  2
   4 -3.5209622e+01 3.97e+04 2.75e+00  -1.0 2.74e+05    -  9.68e-07 3.94e-06f  2
   5 -5.4709651e+01 3.97e+04 4.07e+00  -1.0 3.10e+05    -  4.84e-06 1.81e-05h  1
   6 -9.5900584e+01 3.97e+04 5.24e+00  -1.0 3.31e+05    -  2.27e-05 9.51e-05f  1
   7 -5.7803492e+02 3.94e+04 2.13e+01  -1.0 3.33e+05    -  1.18e-04 7.75e-03f  1
   8 -9.8166833e+03 3.94e+02 9.99e+00  -1.0 3.31e+05    -  7.79e-03 9.90e-01f  1
   9 -9.8851832e+03 3.83e+00 2.01e-01  -1.0 3.31e+03    -  9.90e-01 9.90e-01h  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  10 -9.9470365e+03 0.00e+00     inf  -1.0 1.55e+02    -  9.87e-01 1.00e+00f  1
  11           -nan 0.00e+00     nan  -1.0     nan    -  1.00e+00 1.00e+00   0

WARNING: Problem in step computation; switching to emergency mode.

  12r          -nan 0.00e+00     nan   nan 0.00e+00  20.0 0.00e+00 0.00e+00R  1

WARNING: Problem in step computation; switching to emergency mode.

  13r          -nan 0.00e+00     nan   nan 0.00e+00  20.0 0.00e+00 0.00e+00R  1

WARNING: Problem in step computation; switching to emergency mode.

  14r          -nan 0.00e+00     nan   nan 0.00e+00    -  0.00e+00 0.00e+00R  1

WARNING: Problem in step computation; switching to emergency mode.

  15r          -nan 0.00e+00     nan   nan 0.00e+00    -  0.00e+00 0.00e+00R  1


Regards,
Parthipan.E

[AMPL Google Group]

In your 21 iteration , I see nan in the objective function value.

21 -nan 0.00e+00 1.07e-13 -12.9 4.59e+02 - 9.67e-01 9.95e-01h 1

You need to make sure that the function can be evaluated at all points in your bounds. Sometimes, there are function like sqrt() which is defined for positive number, but not for negative number. It seems that objective function can't be evaluated at 21th iteration. But you said, IPOPT can solve the problem in your local computer. Can you check the value of the objective function in each iteration of the optimization in your local system and server. Do they match?

Thanks,
Paras

--
Paras Tiwari
am...@googlegroups.com

{#HS:520524496-306#}

[coolpart...@gmail.com]

Hi Paras,

Upto 20th iteration the value is same. After that throw the following warning and continue the iteration.

  20 -3.0213890e+04 0.00e+00 2.70e-12 -12.9 1.58e+03    -  4.73e-01 9.94e-01f  1

Warning: Cutting back alpha due to evaluation error
  21 -3.0214101e+04 0.00e+00 8.30e-13 -12.9 4.59e+02    -  9.67e-01 4.98e-01h  2
  22 -3.0214378e+04 0.00e+00 2.89e-14 -12.9 3.64e+02    -  9.94e-01 1.00e+00h  1
  23 -3.0214393e+04 0.00e+00 1.07e-15 -12.9 3.21e+01    -  1.00e+00 1.00e+00f  1
  24 -3.0214393e+04 0.00e+00 4.09e-18 -12.9 2.00e+00    -  1.00e+00 1.00e+00f  1

Number of Iterations....: 24

Forget about this local and server comparison.

I have reproduced the same issue in another server(aws instance). In that server I got the "Optimal Solution Found".

Somewhat 'hessian_approximation=limited-memory' option fixed "Restoration Failed" issue.

Now what I need to know are,

 1. Is no.of iteration of the Ipopt optimization is related to system config?

 2. How to find the difference of the server's system configuration?
     FYI: I have checked the basic server configuration of this two server.
     Size: c4.xlarge
     OS: Ubuntu 14.04 LTS
     Architecture: 64bit
     Ram: 8gb
     Volume: 40gb

PFA, I have attached the optimization problem iteration log of both server.

Regards,
Parthipan.E

[AMPL Google Group]

I don't think it's related to system configuration. You need to look into your function and bounds and make sure function can be evaluated at each point. You are getting the message about an "Error in an AMPL evaluation" .This is a warning that occurs when IPOPT asks AMPL to evaluate some objective or constraint function at a point where the function is not defined. For example, the evaluation could involve taking a square root or log of a negative number. IPOPT has ways of dealing with this situation automatically, as indicated by the messages such as "SOC step rejected" and "Cutting back alpha." Setting the AMPL option ipopt_options to contain "halt_on_ampl_error yes" will cause IPOPT to stop and presumably tell you more about why the function could not be evaluated, but normally you will want to leave this option off so that IPOPT deals with the situation automatically. IPOPT is slowed by the need to cope with such situations, so you might want to experiment with choosing a different starting point or tightening the bounds so as to confine IPOPT iterates to regions where the problem functions can be successfully evaluated.

--
Paras Tiwari
am...@googlegroups.com

{#HS:520524496-306#}

[coolpart...@gmail.com]

You are correct Paras.

There is no difference in the system configuration. I have figured it out which causes this issue. There is a very slight difference in couple of constraints which are very weird when I found. I don't think this will make huge mess, that's what I ignore the constraints difference while initial debugging.

I am getting the constraints value from the Postgres database. I will tell you how I did debug in the database. I have inserted the data  from excel file into database which is refer as version 1. Again I did the insertion of same data which is refer version 2.
While getting sum() value of one type of data from version 1 db returns "8164669.8546934", the same query of version 2 returns "8164669.85469339".

Like this, couple of constrains I'm getting rounded value. I know this makes no sense at all. Ipopt should work for rounded value(8164669.8546934), But its not.

I couldn't able to debug Ipopt solver. Can you please tell me What/where I'm missing.?

Note: I have tried "halt_on_ampl_error" option. no luck :(

Thanks,
Parthipan.E

[AMPL Google Group]

So, your function can't be evaluated when there is a small change in the bound of a constraint? You could turn off the constraint to check if the constraint is causing the problem or not. You might need to look into your function and identify the function that is sensitive to the bound. You could turn off the objectives and constraint and introduce one by one to isolate the function that is causing the problem. After that, you could study the behavior of the function to understand the problem .

Thanks,
Paras

--
Paras Tiwari
am...@googlegroups.com

{#HS:520524496-306#}

On Mon, Feb 19, 2018 at 7:13 PM UTC, <am...@googlegroups.com> wrote:

You are correct Paras.

There is no difference in the system configuration. I have figured it out which causes this issue. There is a very slight difference in couple of constraints which are very weird when I found. I don't think this will make huge mess, that's what I ignore the constraints difference while initial debugging.

I am getting the constraints value from the Postgres database. I will tell you how I did debug in the database. I have inserted the data from excel file into database which is refer as version 1. Again I did the insertion of same data which is refer version 2.
While getting sum() value of one type of data from version 1 db returns "8164669.8546934", the same query of version 2 returns "8164669.85469339".
Like this, couple of constrains I'm getting rounded value. I know this makes no sense at all. Ipopt should work for rounded value(8164669.8546934), But its not.

I couldn't able to debug Ipopt solver. Can you please tell me What/where I'm missing.?

Note: I have tried "halt_on_ampl_error" option. no luck :(

Thanks,
Parthipan.E

On Mon, Feb 12, 2018 at 7:38 PM UTC, AMPL Google Group <am...@googlegroups.com> wrote:

I don't think it's related to system configuration. You need to look into your function and bounds and make sure function can be evaluated at each point. You are getting the message about an "Error in an AMPL evaluation" .This is a warning that occurs when IPOPT asks AMPL to evaluate some objective or constraint function at a point where the function is not defined. For example, the evaluation could involve taking a square root or log of a negative number. IPOPT has ways of dealing with this situation automatically, as indicated by the messages such as "SOC step rejected" and "Cutting back alpha." Setting the AMPL option ipopt_options to contain "halt_on_ampl_error yes" will cause IPOPT to stop and presumably tell you more about why the function could not be evaluated, but normally you will want to leave this option off so that IPOPT deals with the situation automatically. IPOPT is slowed by the need to cope with such situations, so you might want to experiment with choosing a different starting point or tightening the bounds so as to confine IPOPT iterates to regions where the problem functions can be successfully evaluated.

--
Paras Tiwari
am...@googlegroups.com