Iterative Model Solving

Cecilia Guo

unread,

Jun 16, 2016, 3:00:17 PM6/16/16

to Pyomo Forum

Hello

I am using Pyomo to solve optimization problem repeatitively for each time period.

From each time period to the next one, some modification on parameters would be done.

Currently, I am using Concrete Model and am trying to create instance for each time period so they can be solved for multiple times with different parameter data.

However, while I am trying in the same .py for the second time solving the problem

Error message rises:

File "Anaconda3\lib\site-packages\pyomo\core\base\PyomoModel.py", line 224, in load_from

"with bad status: %s" % str(results.solver.status))

ValueError: Cannot load a SolverResults object with bad status: error

And also I know that doing this way is rather a waste of memory space because I cannot erase the last instance.

Or I think it should be possible I just do instance=createMyModel(DataIns) again but I need to know what is the reason for my solver cannot just solve another instance? Is it because there are still some stored value in solver/model space?

Btw, my createMyModel is to specify parameters in ConcreteModel use a self-defined data instance

Regarding my problem, does anyone have suggestions on how my iterative optimization can be done efficiently?

(P.S. I am afraid I really need to use Concrete Model because I need to receive data and automatic fill, so Abstract might be hard)

Yours

Cecilia

jose santiago rodriguez

unread,

Jun 17, 2016, 10:28:49 AM6/17/16

to Pyomo Forum

Hi Cecilia,

Can you provide the output of the solver?

instance = model

opt = SolverFactory('your_solver')

results = opt.solve(instance,tee=True)

That will help to understand why the solver is not returning a good status. Does the first time solves find (before you modify parameters)?

Santiago

Cecilia Guo

unread,

Jun 17, 2016, 11:52:54 AM6/17/16

to pyomo...@googlegroups.com

Hello Santiago

Here is all the output of the solver.

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Path 4.7.01 (Thu Aug 2 10:31:46 2007)

Written by Todd Munson, Steven Dirkse, and Michael Ferris

INITIAL POINT STATISTICS

Maximum of X. . . . . . . . . . 0.0000e+00 var: (_svar[1])

Maximum of F. . . . . . . . . . 1.0000e+12 eqn: (_scon[94])

Maximum of Grad F . . . . . . . 1.0000e+00 eqn: (_scon[120])

var: (_svar[1])

INITIAL JACOBIAN NORM STATISTICS

Maximum Row Norm. . . . . . . . 6.0000e+00 eqn: (_scon[106])

Minimum Row Norm. . . . . . . . 1.0000e+00 eqn: (_scon[1])

Maximum Column Norm . . . . . . 5.0000e+00 var: (_svar[1])

Minimum Column Norm . . . . . . 1.0000e+00 var: (_svar[31])

Crash Log

major func diff size residual step prox (label)

0 0 4.2435e+12 0.0e+00 (_scon[94])

pn_search terminated: no progress.

Major Iteration Log

major minor func grad residual step type prox inorm (label)

0 0 13 1 4.2435e+12 I 1.0e+01 1.0e+12 (_scon[94])

1 39 14 2 4.6311e+12 1.0e+00 SO 4.0e+00 1.1e+12 (_scon[96])

2 46 15 3 1.5057e+13 1.0e+00 RO 1.6e+00 3.4e+12 (_scon[24])

3 1 16 4 1.5057e+13 1.0e+00 RO 6.4e-01 3.4e+12 (_scon[24])

4 26 17 5 1.5191e+13 1.0e+00 RO 2.6e-01 3.2e+12 (_scon[117])

5 35 18 6 1.5191e+13 1.0e+00 CO 1.0e-01 3.2e+12 (_scon[117])

6 3 19 7 1.5191e+13 1.0e+00 RO 4.1e-02 3.2e+12 (_scon[117])

7 2 25 8 3.5928e+12 2.6e-01 RG 4.0e+00 8.2e+11 (_scon[104])

8 30 26 9 3.5551e+12 1.0e+00 RO 1.6e+00 8.1e+11 (_scon[104])

9 183 27 10 3.5523e+12 1.0e+00 CO 6.4e-01 8.1e+11 (_scon[104])

10 133 28 11 3.5457e+12 1.0e+00 CO 2.6e-01 8.1e+11 (_scon[104])

11 4 29 12 3.5450e+12 1.0e+00 RO 1.0e-01 8.1e+11 (_scon[104])

12 5 43 13 3.7393e+12 1.2e-06 RB 4.1e-02 8.1e+11 (_scon[104])

13 44 48 14 3.0558e+12 4.1e-01 RG 1.0e-01 6.2e+11 (_scon[29])

14 20 49 15 3.0224e+12 1.0e+00 SO 1.0e+01 5.9e+11 (_scon[29])

15 13 50 16 3.1479e+12 1.0e+00 RO 4.0e+00 6.4e+11 (_scon[102])

16 5 51 17 3.1770e+12 1.0e+00 RO 1.6e+00 6.5e+11 (_scon[96])

17 6 52 18 2.8418e+12 1.0e+00 RO 6.4e-01 6.0e+11 (_scon[102])

18 26 53 19 2.2508e+12 1.0e+00 RO 2.6e-01 4.9e+11 (_scon[102])

19 10 54 20 8.9054e+11 1.0e+00 RO 1.0e-01 2.0e+11 (_scon[101])

20 2 61 21 6.7533e+11 1.7e-01 RG 4.1e-02 1.8e+11 (_scon[123])

21 20 62 22 1.9355e+12 1.0e+00 RM 1.6e-02 5.5e+11 (_scon[80])

22 3 67 23 3.1198e+12 3.3e-01 RB 6.6e-03 6.7e+11 (_scon[95])

23 1 68 24 3.1198e+12 1.0e+00 RO 2.6e-03 6.7e+11 (_scon[95])

24 6 78 25 3.1451e+12 9.2e-03 RB 1.0e-03 6.8e+11 (_scon[95])

25 2 88 26 2.4159e+12 9.2e-03 RB 1.0e+01 6.8e+11 (_scon[95])

26 7 95 27 5.4046e+11 1.7e-01 RG 4.1e-02 1.3e+11 (_scon[101])

27 31 96 28 1.2864e+12 1.0e+00 RM 1.6e-02 6.2e+11 (_scon[79])

28 1 97 29 1.2864e+12 1.0e+00 RO 6.6e-03 6.2e+11 (_scon[79])

29 1 98 30 1.2864e+12 1.0e+00 RO 2.6e-03 6.2e+11 (_scon[79])

30 6 99 31 7.5143e+12 1.0e+00 RL 1.0e+01 3.2e+12 (_scon[15])

31 12 100 32 7.3766e+12 1.0e+00 RD 4.0e+00 3.2e+12 (_scon[15])

32 3 107 33 4.4856e+11 1.7e-01 RG 4.1e-02 1.0e+11 (_scon[123])

33 30 108 34 1.9085e+12 1.0e+00 RM 1.6e-02 6.2e+11 (_scon[80])

34 1 109 35 1.9085e+12 1.0e+00 RO 6.6e-03 6.2e+11 (_scon[80])

35 2 118 36 1.6887e+12 2.3e-02 RB 1.0e+01 6.1e+11 (_scon[80])

36 2 119 37 1.6937e+12 1.0e+00 RO 4.0e+00 6.1e+11 (_scon[80])

37 4 120 38 1.7585e+12 1.0e+00 RO 1.6e+00 5.7e+11 (_scon[79])

Restart Log

proximal_perturbation 0

crash_method none

crash_perturb yes

nms_initial_reference_factor 2

lemke_search_type slack

proximal_perturbation 1.0000e-01

Major Iteration Log

major minor func grad residual step type prox inorm (label)

37 0 127 39 4.2435e+12 R 1.0e-01 1.0e+12 (_scon[94])

Basis not invertible at best point.

38 51 128 40 4.2435e+12 1.0e+00 TO 1.0e+01 1.0e+12 (_scon[94])

39 1 129 41 4.6311e+12 1.0e+00 SM 4.0e+00 1.1e+12 (_scon[96])

40 18 130 42 4.6430e+12 1.0e+00 RM 1.6e+00 1.1e+12 (_scon[94])

41 38 131 43 4.5499e+12 1.0e+00 RM 6.4e-01 1.1e+12 (_scon[95])

42 30 132 44 4.5416e+12 1.0e+00 RM 2.6e-01 1.1e+12 (_scon[95])

43 27 133 45 4.5520e+12 1.0e+00 RM 1.0e-01 1.1e+12 (_scon[95])

44 4 134 46 4.7217e+12 1.0e+00 RM 1.0e+01 1.1e+12 (_scon[95])

45 2 140 47 3.5928e+12 2.6e-01 RG 1.0e+01 8.2e+11 (_scon[104])

46 31 141 48 3.5886e+12 1.0e+00 RM 4.0e+00 8.2e+11 (_scon[104])

47 16 142 49 4.4321e+12 1.0e+00 RM 1.6e+00 1.4e+12 (_scon[54])

48 7 143 50 4.2535e+12 1.0e+00 RM 6.4e-01 1.4e+12 (_scon[54])

49 7 144 51 4.2029e+12 1.0e+00 RM 2.6e-01 1.3e+12 (_scon[54])

50 3 158 52 4.2192e+12 1.2e-06 RB 1.0e-01 1.3e+12 (_scon[54])

51 11 163 53 3.1195e+12 4.1e-01 RG 4.0e+00 6.5e+11 (_scon[73])

52 31 164 54 4.4628e+12 1.0e+00 RM 1.6e+00 1.0e+12 (_scon[96])

53 6 165 55 4.4020e+12 1.0e+00 RM 6.4e-01 1.0e+12 (_scon[96])

54 2 166 56 4.4038e+12 1.0e+00 RM 2.6e-01 1.0e+12 (_scon[96])

55 6 179 57 4.4016e+12 4.4e-05 RB 1.0e-01 1.0e+12 (_scon[96])

56 2 180 58 4.4016e+12 1.0e+00 RO 4.1e-02 1.0e+12 (_scon[96])

57 3 187 59 2.3724e+12 1.7e-01 RG 4.0e+00 6.6e+11 (_scon[118])

58 26 188 60 4.1742e+12 1.0e+00 SM 1.6e+00 1.2e+12 (_scon[54])

59 1 189 61 4.1742e+12 1.0e+00 RO 6.4e-01 1.2e+12 (_scon[54])

60 4 190 62 2.1508e+12 1.0e+00 RM 2.6e-01 5.6e+11 (_scon[54])

61 9 191 63 2.6520e+11 1.0e+00 RM 1.0e-01 7.5e+10 (_scon[25])

62 9 192 64 2.6481e+11 1.0e+00 RM 4.1e-02 7.5e+10 (_scon[25])

63 11 193 65 1.4756e+11 1.0e+00 RM 1.6e-02 4.5e+10 (_scon[25])

64 6 199 66 1.3032e+11 2.6e-01 RG 1.0e+01 3.2e+10 (_scon[23])

65 25 200 67 1.0570e+11 1.0e+00 SM 4.0e+00 3.1e+10 (_scon[138])

66 16 201 68 3.5330e+11 1.0e+00 SM 1.6e+00 8.5e+10 (_scon[140])

67 19 202 69 3.4392e+11 1.0e+00 CM 6.4e-01 8.3e+10 (_scon[140])

68 9 203 70 3.2580e+11 1.0e+00 RM 2.6e-01 7.9e+10 (_scon[140])

69 3 204 71 3.2580e+11 1.0e+00 RO 1.0e-01 7.9e+10 (_scon[140])

70 5 205 72 3.2580e+11 1.0e+00 RO 4.1e-02 7.9e+10 (_scon[140])

71 4 206 73 3.2580e+11 1.0e+00 RO 1.6e-02 7.9e+10 (_scon[140])

72 4 213 74 6.8152e+10 1.7e-01 RG 4.0e+00 2.3e+10 (_scon[25])

73 26 214 75 1.1305e+11 1.0e+00 SM 1.6e+00 4.1e+10 (_scon[24])

74 12 215 76 1.1253e+11 1.0e+00 RM 6.4e-01 4.1e+10 (_scon[22])

75 8 216 77 1.1001e+11 1.0e+00 RM 2.6e-01 3.9e+10 (_scon[22])

76 13 217 78 1.1001e+11 1.0e+00 RO 1.0e-01 3.9e+10 (_scon[22])

77 3 218 79 1.1001e+11 1.0e+00 RO 4.1e-02 3.9e+10 (_scon[22])

78 13 219 80 1.1021e+11 1.0e+00 RM 1.6e-02 4.0e+10 (_scon[22])

Restart Log

proximal_perturbation 0

crash_method none

crash_perturb no

nms_initial_reference_factor 10

nms_memory_size 2

nms_mstep_frequency 1

lemke_search_type slack

Major Iteration Log

major minor func grad residual step type prox inorm (label)

78 0 227 81 4.2435e+12 R 0.0e+00 1.0e+12 (_scon[94])

79 58 228 82 4.2435e+12 1.0e+00 RM 0.0e+00 1.0e+12 (_scon[94])

80 8 229 83 4.2435e+12 1.0e+00 RM 0.0e+00 1.0e+12 (_scon[94])

81 67 234 84 2.5462e+12 3.3e-01 RB 0.0e+00 6.0e+11 (_scon[127])

82 18 235 85 2.6263e+12 1.0e+00 SM 1.0e+01 6.1e+11 (_scon[96])

83 8 251 86 8.9649e+10 1.0e+00 SG 0.0e+00 5.7e+10 (_scon[52])

84 50 252 87 8.9649e+10 1.0e+00 RM 0.0e+00 5.7e+10 (_scon[52])

85 47 253 88 1.2140e-03 1.0e+00 SM 0.0e+00 1.1e-03 (_scon[151])

86 1 254 89 1.1970e-14 1.0e+00 SM 0.0e+00 1.1e-14 (_scon[138])

FINAL STATISTICS

Inf-Norm of Complementarity . . 1.0658e-14 eqn: (_scon[138])

Inf-Norm of Normal Map. . . . . 1.0658e-14 eqn: (_scon[138])

Inf-Norm of Minimum Map . . . . 3.5527e-15 eqn: (_scon[24])

Inf-Norm of Fischer Function. . 1.0658e-14 eqn: (_scon[138])

Inf-Norm of Grad Fischer Fcn. . 1.1990e-14 eqn: (_scon[94])

Two-Norm of Grad Fischer Fcn. . 2.4099e-14

FINAL POINT STATISTICS

Maximum of X. . . . . . . . . . 1.0000e+12 var: (_svar[116])

Maximum of F. . . . . . . . . . 1.0000e+12 eqn: (_scon[31])

Maximum of Grad F . . . . . . . 1.0000e+00 eqn: (_scon[120])

var: (_svar[1])

** EXIT - solution found.

Major Iterations. . . . 86

Minor Iterations. . . . 2069

Restarts. . . . . . . . 2

Crash Iterations. . . . 0

Gradient Steps. . . . . 13

Function Evaluations. . 254

Gradient Evaluations. . 89

Basis Time. . . . . . . 0.031000

Total Time. . . . . . . 0.062000

Residual. . . . . . . . 1.196990e-14

Path 4.7.01: Solution found.

86 iterations (0 for crash); 2069 pivots.

254 function, 89 gradient evaluations.

Problem:

- Lower bound: -inf

Upper bound: inf

Number of objectives: 1

Number of constraints: 0

Number of variables: 171

Sense: unknown

Solver:

- Status: ok

Message: Path 4.7.01\x3a Solution found.; 86 iterations (0 for crash); 2069 pivots.; 254 function, 89 gradient evaluations.

Termination condition: optimal

Id: 0

Error rc: 0

Time: 0.12801289558410645

Solution:

- number of solutions: 0

number of solutions displayed: 0

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I tried to re-open and re-run the program for several times.

And I did find sometimes the second iteration works but with warnings as follows:

WARNING: Implicitly replacing the Component attribute bv (type=<class 'pyomo.core.base.var.SimpleVar'>) on block LRcontract[8] with a new Component (type=<class 'pyomo.core.base.var.SimpleVar'>).

This is usually indicative of a modelling error.

To avoid this warning, use block.del_component() and block.add_component().

Can you understand from these where went wrong in my programming?

Yours

Cecilia

Gabriel Hackebeil

unread,

Jun 18, 2016, 12:24:08 AM6/18/16

to pyomo...@googlegroups.com

The WARNING output at the end of your email means that you are overwriting a variable somewhere on your model (block LRcontract[8]) with a new variable. This is okay as long as the old variable is not referenced anywhere (e.g., in another constraint).

To make that warning go away, you should remove the old variable first by calling: model.LRcontract[8].del_component(<variable_name>)

If it is referenced in another constraint, then you also need to remove that constraint, or fix the code so that existing variables are not overwritten.

It’s hard to say why your model fails after the second iteration, but it might have something to do with the warning, or perhaps the modifications you are making to your model cause it to become infeasible. The solver output or the statuses on the results object will help you determine if this is the case.

Gabe

Yingjian

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Cecilia Guo

unread,

Jun 20, 2016, 9:48:02 AM6/20/16

to Pyomo Forum

Hi Gabe and Santiago

Thanks so much for answering. But I did not get why my program would overwrite the new variable in the case that I solve the problem once, and update some parameters in instance, and then solve it another time without any modification on model formulation. And also, since I am re-solving the model, shouldn't all variables be automatically updated to the new solution? Is that an overwrite process and that's why the warning is there?

In addition, I read on Pyomo iterative solving part that I need a pre-solve before solving again.

Solve instance

Change someting in instance

Call presolve

Solve instance again

However, I have no idea how to call presolve.

Moreover, since I need to solve the model repetitively with updated data using Concrete Model. I am thinking about ways to update data.

I am trying to update data automatically.

One option is to create instance with updated dataset again. But in that case, I guess the previous solution would be gone and if my problem is large, I cannot use the previous solution as a solving starting point and it might be slow to get each time solution from a fresh start.

Alternatively I am thinking to modify instance directly without erasing and re-create. But the problem I am facing is in assigning new parameter value.

I cannot use

for Para in ParaHeader:

setattr(instance,Para,data[Para])

such type of code. So I have no idea how to assign instance.Para automatically. Do you have any clue on this?

Among the two mentioned ways, which do you think can be faster? Or do they vary much on speed? Or do you have any other better suggestion on how I should manage this?

Thanks so much

Looking forward to your reply

Regards

Cecilia (Yingjian)

Gabriel Hackebeil

unread,

Jun 20, 2016, 10:03:14 AM6/20/16

to pyomo...@googlegroups.com

Here is an example of how to solve a model, update a parameter value, and solve the model again.

# create the model

model = ConcreteModel()

model.x = Var()

model.p = Param(mutable=True) # allows the parameter value to be updated

model.o = Objective(expr= model.x)

model.c = Constraint(expr= model.x >= model.p)

# create a solver

solver = SolverFactory(“cplex”)

# set the initial parameter value

model.p.value = 2.0

# solve the model (the solution is loaded automatically)

results = solver.solve(model)

# print the solver termination condition

print(results.solver.termination_condition)

# update the parameter value

model.p.value = 3.0

# solve the model again

results = solver.solve(model)

# print the solver termination condition

print(results.solver.termination_condition)

When I say overwriting a variable, I mean that wherever you are defining the block LRcontract[8], there is some section of Python code that is assigning the a variable with the same name multiple times to that block. E.g.,

model = ConcreteModel()

model.x = Var()

model.x = Var() # this produces the warning you are seeing

Gabe

jose santiago rodriguez

unread,

Jun 20, 2016, 10:15:40 AM6/20/16

to Pyomo Forum

Cecilia,

Since you are working with a concrete model i would suggest creating your model inside a function that takes as arguments the parameters of your model.

def create_model(parameter):

m = ConcreteModel()

m.x = Var(...)

m.obj = Objective(expr = m.x*parameter...)

return m

opt = solverFactory("PATH")

for i in xrange(100):

parameter = i

instance = create_model(parameter)

opt.solve(instance)

Your model only has 171 variables, no constraints and 1 objective, so i would not think creating the model many times will be very expensive.

Santiago

Cecilia Guo

unread,

Jun 21, 2016, 11:41:07 AM6/21/16

to Pyomo Forum

Hello Gabe and Santiago

Yes I have tried to re-create the instance, and the problem keeps appearing.

I have done several round of testing and here is what I got stuck.

When I try to modify some parameters and run

t=2010;

EndT=2100

while (t<=EndT):

print(t)

opt = SolverFactory('path')

results = opt.solve(instance);

print(results)

t+=5

DataIns.Para_max['A']+=0.000005;

# this added value is really very small compared to the DataIns.Para_max['A'] current value and the solution should absolutely give feasible solution for all rounds

del instance,results,opt

instance=createMyModel(DataIns)

Error still shows like following:

File "...Anaconda3\lib\site-packages\pyomo\core\base\PyomoModel.py", line 224, in load_from

"with bad status: %s" % str(results.solver.status))

ValueError: Cannot load a SolverResults object with bad status: error

I think it's showing me that the solver cannot get feasible results.

Then I tried two methods to test. Firstly I tried to reset everything and run only one time with parameter specified as the updated parameters in different rounds.

However, the results start show infeasible after 2-3 rounds of modification. (But actually the parameters should give feasible solution) Because I exit spyder editor and python and then restart the program with fresh 1st round parameter specified as the 2-3 round unsuccessful value and this time the solver gives feasible solution.

So it's like when I restart python and solve a new problem with the modified parameters, it can give feasible solution. Whereas if I solve one problem, and either [instance=createMyModel(newDataIns) and resolve] or [reset everything and try to solve with fresh instance=createMyModel(newDataIns)], it cannot give feasible solution.

I guess it might be related to how Pyomo handle solution and solver and release memory but I really have no idea how it is.

Could you please kindly let me know how I should dislink and start every round freshly as re-open the program for solver but still keep all the DataIns data?

Yours

Cecilia

Cecilia Guo

unread,

Jun 23, 2016, 10:32:16 AM6/23/16

to Pyomo Forum

Hello

I looked into pyomo\opt\base\solvers.py file and tried to modify the _presolve function to eliminate the load_result process (set load_results = False ) and thus it can be solved for each time even though some rounds may generate no results.

However, my problem of getting/not getting solution with the same optimization problem with the same parameter is not solved.

From what I have tested, if I start python and solve model1, and get optimal solution. Later when I reset everything and re-solve model1, I can always get the optimal solution. However, if I close python environment and re-start, probably python pyomo with the solver cannot reach optimal solution (with internal solver error: not enough progress). And this time even though I try re-set multiple times, re-create the model and re-run, model1 cannot be solved. So I am guessing something related to result/solution be stored somewhere and cannot be erased unless I re-open the python environment.

Hence, I took a look into the solve function etc (the other several jpg attached) but cannot figure out how solver is called in pyomo.

The comments in apply_solver says that is the solving process, but the definition of this function is rather simple without interacting with solvers. Am I missing some parts?

Also, I didn't understand in try: of solve function, when is the self.results generated, as that is passed to results in post_solve function.

Thanks so much if anyone can help me understand all of this. I am thinking to de-link the solver, re-open it each time when doing the iterative solving but don't know how to do it.

Best,

Cecilia

Pre_solve_modify.JPG

apply_solver.JPG

def_solve.JPG

Opt_solver.JPG

solve_try.JPG

Gabriel Hackebeil

unread,

Jun 23, 2016, 7:40:21 PM6/23/16

to pyomo...@googlegroups.com

Cecilla,

I would recommend not looking in that file for help. It is very difficult to follow how the solver plugins work (even for me, and I’ve been developing in Pyomo for a few years). The workflow is basically

1) translate the Pyomo model to something that the solver can read (e.g., an LP file, or an Cplex optimization model using the Cplex Python interface).

2) tell the solver to optimize the input

3) translate the solver results into a format Pyomo can load into the model (e.g., parse the solver solution file)

There are many different source files where these three steps are implemented, and which file to look in depends on what solver you use and what interface you use for that solver (and each step can occur different files).

In cases like yours, where you think that you are solving the same model but with different outcomes depending on whether the model is solved first or after previous rounds of modifications that lead to that model, it is best to inspect the solver input file to verify that the models are actually the same. For instance, if you are using the default interface for Cplex (LP files), you can tell Pyomo to keep the temporary LP file after the solve is finished, which you can then inspect for issues. E.g.,

model = …

# solve the first time

with SolverFactory(“cplex”) as opt:

# keepfiles: print the solver input files (and don’t delete them afterwards)

# symbolic_solver_labels: use human readable names in the solver input files

results = opt.solve(model, keepfiles=True, symbolic_solver_labels=True)

… make modifications to the model …

# solve the second time

with SolverFactory(“cplex”) as opt:

# keepfiles: print the solver input files (and don’t delete them afterwards)

# symbolic_solver_labels: use human readable names in the solver input files

results = opt.solve(model, keepfiles=True, symbolic_solver_labels=True)

Now you can compare the two LP files whose name will be printed, to see exactly what the differences are (as seen by the solver). When you are debugging, it is also best to make your model as simple as possible such at the issue still occurs, so that the LP files are easy to read and compare.

If you want to send a minimal working example that reproduces the issue, we can help you figure out what the problem is as well.

Gabe

<Pre_solve_modify.JPG><apply_solver.JPG><def_solve.JPG><Opt_solver.JPG><solve_try.JPG>

Cecilia Guo

unread,

Jun 24, 2016, 10:21:04 AM6/24/16

to Pyomo Forum

Hello Gabriel

Thanks so much for your reply.

Yes I did what you said and tried to compare the two input file for solvers.

The solver I am using is PATH (cos I am formulating MCP/MPEC) and it is the AMPL interface version so the file is in .nl.

I tried to compare and here I attached some parts I found different. However cos I do not understand how to interpret .nl file, so I am still quite confused.

First is in the RHS range part whereas some other differences exist in the intermediate Jacobian column length part. (P.S. screen capture LHS as the solved file and RHS as unsolved one)

For the files generated, there are also tmpwiwk85dd.pyomo.col and tmpwiwk85dd.pyomo.row, shall I use them to interpret the .nl file together?

Moreover, I also took a more detailed look into the solving log file and observed something in the "residual" value.

Basically, the solved version in some iteration get a sudden change in "residual" and then decrease the value to a very small number whereas for the unsolved one, the residual value keeps being very large without sudden decreasing. (Screen cap also LHS solved and RHS unsolved) What do you think may be the problem? Is it because of my scaling or it is because of the solving algorithm?

I attached the corresponding pictures and I am currently reduce the problem size and re-formulate my problem and do the test. (Well this current is actually a simplified version)

As soon as I get a minimal working example with the same issue, I would like to send it to you for some check.

And please accept my sincere thanks. So grateful for your help

Cheers,

Cecilia

Jacobian_diff.JPG

RHS_diff_2.JPG

Jacobian_diff_1.JPG

Jacobian_diff_2.JPG

Jacobian_diff_3.JPG

Jacobian_diff_4.JPG

Log_diff.JPG

Log_Header.JPG

RHS_diff.JPG

RHS_diff_1.JPG

jose santiago rodriguez

unread,

Jun 24, 2016, 11:05:49 AM6/24/16

to Pyomo Forum

You probably dont want to get into the details of the NL file and you dont need to. However if you are curios about it you can take a look at

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.9659&rep=rep1&type=pdf

To answer your question, the .row and .col files have information about your constraints and variables. When they go to the solver they no longer have names but indices. With the .col file you can get information about the mapping of variable names to indices and with the .row file you get map of constraint names to indices. What you are interested in is the parameters that you changed. Since you are using the same model and just changing the values of some parameters i would expect the .row and .col files to be the same and the nl files to change only in those lines where the parameter value is specified.

On windows or linux i would recommend you to use meld

http://meldmerge.org/

This tool will quickly let you diff the two nl files. Moreover, you can pass a whole directory and will show you difference between them. It will highlight the lines that changed and the lines that were aded. You should only see the former.

On Mac you can get something similar if you use opendiff

Santiago

On Thursday, June 16, 2016 at 3:00:17 PM UTC-4, Cecilia Guo wrote:

Cecilia Guo

unread,

Jun 27, 2016, 1:19:53 PM6/27/16

to Pyomo Forum

Hi Santiago and Gabriel

Here I looked into the .nl file and compare them between each other for solved/unsolved conditions in different rounds.

So here, before each test, I re-open the python environment. And in each test round, I modify an already over-defined maximum capacity parameter Lmax with an additional value of 5 and continuously delete instance and results, and re-create the instance with new DataIns and resolve the problem. I kept files for the test.

Theoretically, since the basic version result shows the constraint of Lmax is not activated (corresponding dual =0), adding value to this constraint should not affect the results. (In addition, the max added value is around 0.00002% of the original value and hence should not have any influence at all)

The observation is that

1. Still in the same test in those rounds, some can be solved while the others cannot. The .nl file only vary from each other in the "r #171 ranges (rhs's)" part with corresponding different body value as the changed parameter. (Fig 1,2,3 attached) But even though the others are exactly the same and theoretically all should give feasible and the same solution, the result is still some with feasible same solution while the others give solver internal error not enough progress.

2. For different tests but same parameter, the sequence of body in r ranges (rhs's) are different but the contents are the same. So I don't think that's what causes the problem. The other difference exists in the Intermediate Jacobian column lengths (Fig 4,5). I think J0-J170 represent for calculations related to 1-171 variables (k section). But for some of those Js, not only the index (1st column) but also the values (2nd column) are different (Highlighted in Fig 5). This doesn't make a difference even if in different tests with the same parameter both are solvable or one is solvable one is not-solved or neither is solved.

Hence my question is how is the Jacobian used and computed in .nl and how the values would affect the solution?

In addition, I tried initialize in each instance exactly the same as the solution, but still not making any improvement and cannot ensure solution.

I am still trying to simplify the model and try to study how I should do scaling for the problem.

Yours,

Cecilia

1.JPG

2.JPG

3.JPG

4.JPG

5.JPG

Gabe Hackebeil

unread,

Jun 27, 2016, 3:33:15 PM6/27/16

to pyomo...@googlegroups.com

Cecilia,

Understanding the NL format is admittedly difficult. If you can drop the nonlinear part of your model temporarily, it might be easier to debug using an LP file. You can write an LP file using human readable names in the file from a Pyomo model by doing:

model.write("junk.lp", symbolic_solver_labels=True)

Gabe

--

You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<1.JPG>

<2.JPG>

<3.JPG>

<4.JPG>

<5.JPG>

Cecilia Guo

unread,

Jul 19, 2016, 10:18:44 AM7/19/16

to pyomo...@googlegroups.com

Hello Gaberial

Sorry for not replying asap.

I tried

model.write("junk.lp", symbolic_solver_labels=True)
but got error as

TypeError: write() got an unexpected keyword argument 'symbolic_solver_labels'

I am indeed using the model as a concrete model, but cannot get the argument.

Is that because of update version? I already use the latest version.

May I know where I got wrong?

Also, I still cannot figure out the reason and hence am thinking to increase solver restart times as I found from comparison of solved/unsolved log files, sometimes the first round both cannot get residual significantly smaller whereas the solved one get a sharp decrease in residual in the 2nd round. So I was thinking whether I can increase the restart time to 10-20 (now is 3) so that there are higher chances the algorithm can reach to optimal?

Yours

Cecilia

solved_log.txt

unsolved_log.txt

Gabriel Hackebeil

unread,

Jul 19, 2016, 1:08:43 PM7/19/16

to pyomo...@googlegroups.com

Sorry, I got the syntax wrong. It should be

model.write(“junk.lp”, io_options={“symbolic_solver_labels”: True})

Also, I still cannot figure out the reason and hence am thinking to increase solver restart times as I found from comparison of solved/unsolved log files, sometimes the first round both cannot get residual significantly smaller whereas the solved one get a sharp decrease in residual in the 2nd round. So I was thinking whether I can increase the restart time to 10-20 (now is 3) so that there are higher chances the algorithm can reach to optimal?

I can’t really offer any advice here. You should check the documentation for your solver and play around with different option settings until you find something that works.

Gabe

Venkatachalam Lakshmanan

unread,

Apr 6, 2018, 9:21:55 AM4/6/18

to Pyomo Forum

Hi Cecilia,

I am getting the similar error. I use Gurobi and the error message is "ValueError: Cannot load a SolverResults object with bad status: aborted"

Did you find any solution? If so, can you please share with me.

Thanks and best regards,

Venkat.

jose santiago rodriguez

unread,

Apr 6, 2018, 10:19:22 AM4/6/18

to pyomo...@googlegroups.com

Hi,

That means gurobi is not finding an optimal solution to your optimization problem. Check the the log from gurobi to try to find out what is going wrong.

at the solve call make sure to pass tee=True

opt = pe.SolverFactory('gurobi')

results = opt.solve(model, tee=True)

--

You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum+unsubscribe@googlegroups.com.

Venkatachalam Lakshmanan

unread,

Apr 9, 2018, 6:48:31 AM4/9/18

to pyomo...@googlegroups.com

Hi Jose,

Thanks for the swift response.

The warning says the model contains a solution WARNING - Loading a SolverResults object with an 'aborted' status, but containing a solution

Here is the instance creation with options

instance = model.create_instance("Example/data_Integer.dat")

opt = SolverFactory('gurobi')

opt.options["TimeLimit"] = 300

results = opt.solve(instance, tee=True)

instance.solutions.load_from(results)

Here is the whole listing.

(C:\Programs\Anaconda3) n:\Desktop\test>python test.py

Changed value of parameter TimeLimit to 300.0

Prev: 1e+100 Min: 0.0 Max: 1e+100 Default: 1e+100

Optimize a model with 192738 rows, 157699 columns and 551969 nonzeros

Variable types: 140178 continuous, 17521 integer (0 binary)

Coefficient statistics:

Matrix range [2e-08, 3e+04]

Objective range [5e-01, 5e-01]

Bounds range [0e+00, 0e+00]

RHS range [1e+00, 3e+04]

Warning: Model contains large matrix coefficient range

Consider reformulating model or setting NumericFocus parameter

to avoid numerical issues.

Presolve removed 35060 rows and 70098 columns

Presolve time: 0.35s

Presolved: 157678 rows, 87601 columns, 376726 nonzeros

Variable types: 70080 continuous, 17521 integer (0 binary)

Root relaxation: objective 7.260309e+04, 55664 iterations, 2.08 seconds

Nodes | Current Node | Objective Bounds | Work

Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time

0 0 72603.0929 0 15371 - 72603.0929 - - 5s

H 0 0 2346760.3192 72603.0929 96.9% - 15s

0 0 96534.6947 0 15093 2346760.32 96534.6947 95.9% - 21s

0 0 96534.6947 0 15093 2346760.32 96534.6947 95.9% - 29s

0 0 96577.7722 0 14074 2346760.32 96577.7722 95.9% - 34s

0 0 96577.7722 0 7686 2346760.32 96577.7722 95.9% - 37s

H 0 0 151646.33060 96577.7722 36.3% - 51s

0 0 96582.3363 0 7516 151646.331 96582.3363 36.3% - 54s

0 0 96614.6450 0 7409 151646.331 96614.6450 36.3% - 55s

0 0 96636.4399 0 7429 151646.331 96636.4399 36.3% - 56s

0 0 96636.6068 0 7430 151646.331 96636.6068 36.3% - 56s

0 0 96639.6246 0 7422 151646.331 96639.6246 36.3% - 56s

0 0 96639.9005 0 7421 151646.331 96639.9005 36.3% - 57s

0 0 96642.8075 0 7422 151646.331 96642.8075 36.3% - 57s

0 0 96642.9984 0 7422 151646.331 96642.9984 36.3% - 57s

0 0 96644.7453 0 7034 151646.331 96644.7453 36.3% - 59s

0 0 96645.0149 0 7047 151646.331 96645.0149 36.3% - 59s

0 0 96647.5460 0 7045 151646.331 96647.5460 36.3% - 60s

0 0 96647.8813 0 7054 151646.331 96647.8813 36.3% - 60s

0 0 96652.7980 0 7039 151646.331 96652.7980 36.3% - 61s

0 0 96653.1453 0 7044 151646.331 96653.1453 36.3% - 62s

0 0 96656.8657 0 7036 151646.331 96656.8657 36.3% - 63s

0 0 96656.8657 0 7035 151646.331 96656.8657 36.3% - 64s

0 2 96656.8657 0 7035 151646.331 96656.8657 36.3% - 67s

18 17 97477.1836 10 7505 151646.331 97432.2517 35.8% 622 70s

33 31 100042.717 14 6931 151646.331 97432.2517 35.8% 894 75s

55 52 97499.8270 20 7498 151646.331 97432.2517 35.8% 938 80s

74 73 97506.1064 26 7496 151646.331 97432.2517 35.8% 966 86s

99 97 100669.223 39 6817 151646.331 97432.2517 35.8% 1071 91s

149 149 97545.6738 57 7471 151646.331 97432.2517 35.8% 793 95s

208 207 145981.827 88 4880 151646.331 97432.2517 35.8% 696 102s

233 234 97574.7286 92 7404 151646.331 97432.2517 35.8% 685 108s

253 253 97575.2019 94 7402 151646.331 97432.2517 35.8% 691 112s

257 254 97575.4459 95 7401 151646.331 97432.2517 35.8% 720 116s

272 273 97576.9326 101 7395 151646.331 97432.2517 35.8% 786 121s

282 280 97577.4362 103 7393 151646.331 97432.2517 35.8% 844 128s

307 307 97578.4447 107 7389 151646.331 97432.2517 35.8% 821 130s

328 326 97579.7426 112 7384 151646.331 97432.2517 35.8% 839 136s

368 368 97581.0405 117 7379 151646.331 97432.2517 35.8% 843 143s

374 374 97581.8196 120 7376 151646.331 97432.2517 35.8% 871 146s

379 376 97582.0799 121 7375 151646.331 97432.2517 35.8% 904 151s

387 387 97582.8636 124 7372 151646.331 97432.2517 35.8% 932 156s

413 415 97585.2639 133 7363 151646.331 97432.2517 35.8% 967 163s

420 420 97585.7987 135 7361 151646.331 97432.2517 35.8% 999 166s

428 422 97586.6008 138 7358 151646.331 97432.2517 35.8% 1029 170s

458 459 97589.5420 149 7347 151646.331 97432.2517 35.8% 1009 175s

467 468 97591.1610 155 7341 151646.331 97432.2517 35.8% 1031 180s

506 504 97595.2854 170 7326 151646.331 97432.2517 35.8% 1058 188s

527 524 97596.1109 173 7323 151646.331 97432.2517 35.8% 1061 192s

552 553 97599.9635 187 7309 151646.331 97432.2517 35.8% 1058 197s

565 565 97601.9015 194 7302 151646.331 97432.2517 35.8% 1077 203s

587 586 97602.7434 197 7299 151646.331 97432.2517 35.8% 1088 208s

612 610 97606.9881 212 7284 151646.331 97432.2517 35.8% 1089 213s

644 647 131295.242 217 5189 151646.331 97432.2517 35.8% 1089 218s

658 659 97609.2519 220 7276 151646.331 97432.2517 35.8% 1103 223s

667 663 97609.5349 221 7275 151646.331 97432.2517 35.8% 1130 229s

698 698 97616.8955 247 7249 151646.331 97432.2517 35.8% 1122 234s

706 705 97617.4669 249 7247 151646.331 97432.2517 35.8% 1153 238s

714 713 97618.3268 252 7244 151646.331 97432.2517 35.8% 1173 245s

751 749 97618.9046 254 7242 151646.331 97432.2517 35.8% 1165 251s

771 771 97621.2286 262 7234 151646.331 97432.2517 35.8% 1181 258s

791 789 97622.9731 268 7228 151646.331 97432.2517 35.8% 1208 265s

905 907 97642.1645 334 7162 151646.331 97432.2517 35.8% 1100 271s

920 914 97643.3276 338 7158 151646.331 97432.2517 35.8% 1114 279s

946 947 97645.0750 344 7152 151646.331 97432.2517 35.8% 1124 288s

966 967 97648.5846 356 7141 151646.331 97432.2517 35.8% 1142 294s

978 973 111372.922 362 5605 151646.331 97432.2517 35.8% 1168 300s

Cutting planes:

Gomory: 109

MIR: 863

Explored 1012 nodes (1323665 simplex iterations) in 300.06 seconds

Thread count was 4 (of 4 available processors)

Solution count 3: 151646 2.34676e+06 2.34676e+06

Pool objective bound 97432.3

Time limit reached

Best objective 1.516463306007e+05, best bound 9.743225165270e+04, gap 35.7503%

WARNING - Loading a SolverResults object with an 'aborted' status, but containing a solution

Traceback (most recent call last):

File "Test.py", line 31, in <module>

instance = Model_Resolution_Integer(model)

File "n:\Desktop\test\test.py", line 190, in Model_Resolution_Integer

instance.solutions.load_from(results) # Loading solution into instance

File "C:\Programs\Anaconda3\lib\site-packages\pyomo\core\base\PyomoModel.py", line 241, in load_from

"with bad status: %s" % str(results.solver.status))

ValueError: Cannot load a SolverResults object with bad status: aborted

Best regards,

Venkat.

--

Venkatachalam Lakshmanan

Postdoctoral Researcher,

Faculty of Information Technology and Electrical Engineering,

Norwegian University of Science and Technology - NTNU

Trondheim, Norway

Venkatachalam Lakshmanan

unread,

Apr 11, 2018, 7:52:24 AM4/11/18

to pyomo...@googlegroups.com

Hi,

I am able to stop the GLPK solver through pyomo for a given time limit and get the available integer solution. How can I continue the the solver from that point? The warm start option produces " Invalid option '--warmstart'" error.

TIME LIMIT EXCEEDED; SEARCH TERMINATED

Time used: 401.4 secs

Memory used: 26.4 Mb (27714865 bytes)

Writing MIP solution to 'C:\Users\venkat\AppData\Local\Temp\tmpnxp26pjh.glpk.raw'...

28846 lines were written

current time limit =400

Solving the model - Warm start

Invalid option '--warmstart'; try C:\Programs\Anaconda3\Library\bin\glpsol.exe --help

ERROR: "[base]\site-packages\pyomo\opt\base\solvers.py", 600, solve

Solver (glpk) returned non-zero return code (1)

ERROR: "[base]\site-packages\pyomo\opt\base\solvers.py", 603, solve

See the solver log above for diagnostic information.

Traceback (most recent call last):

File "Micro-Grids.py", line 31, in <module>

instance = Model_Resolution_Integer(model)

File "t:\Desktop\test\Model_Resolution.py", line 194, in Model_Resolution_Integer

results = opt.solve(instance, tee=True)

File "C:\Programs\Anaconda3\lib\site-packages\pyomo\opt\base\solvers.py", line 607, in solve

"Solver (%s) did not exit normally" % self.name)

pyutilib.common._exceptions.ApplicationError: Solver (glpk) did not exit normally