Re: [AMPL 4901] Re: Slow execution

[maziar kermani]

Hello,

I am trying to use "include" command in order to add more items to my sets and as a result also define the parameters that are indexed over these sets. For doing this I am using the "include" and in the file I have some nested for loops that go through several sets and updates the sets and params using "let" command. my ".run" file is like:

model model.mod

data data.in

include handler.in # the for loops

and then it continues with setting options and solving the model.

The idea of the loops is that for each units and their associated temperatures I am adding a unit that is "unit_name" &"temperature". For 10 units and 5 temperature it takes less than a minute. However for 50 units and 25 temperature it takes around 12 to 15 minutes, here are some information on the model:

_nvars = 280152

_ncons = 26878

_nobjs = 1

_snvars = 263777

_sncons = 11559

_snbvars = 2032

_snivars = 0

_solve_time = 999.998

solve_exitcode = 0

solve_message =

'CPLEX 12.6.3.0: optimal integer solution within mipgap or absmipgap; objective 373659.0751\

34747 MIP simplex iterations\

226 branch-and-bound nodes\

absmipgap = 29.1708, relmipgap = 7.80679e-05\

No basis.'

solve_result_num = 3

solve_result = solved

for 26 units and 14 temperatures the solution time is:

_nvars = 42182

_ncons = 9979

_nobjs = 1

_snvars = 36003

_sncons = 4188

_snbvars = 734

_snivars = 0

_solve_time = 16.3333

solve_exitcode = 0

solve_message =

'CPLEX 12.6.3.0: optimal integer solution within mipgap or absmipgap; objective 348762.1024\

6302 MIP simplex iterations\

0 branch-and-bound nodes\

absmipgap = 31.5372, relmipgap = 9.04261e-05\

No basis.'

solve_result_num = 3

solve_result = solved

The issue is that for a big case study normally my model's size should be at the other of the first example which is my goal. However if I want to do some kind of optimization on level of temperatures, I need to run this model more than 1000 time using genetic algorithm which in this case can take up to months to run! 

I was wondering if there is anything I should do to speed up the data handling of AMPL.

Thanks in advance,

Maziar

[Robert Fourer]

The time that you show in your listings is _solve_time which is the time in the most recent run of the solver, CPLEX. It may be not a very accurate time, however, because it is the sum of the times in all of the parallel threads that CPLEX was running. Instead you should look at _solve_elapsed_time, which is the number of seconds you waited between when CPLEX began working and when it finished. Or, better yet, set

option cplex_options 'mipdisplay 2 mipinterval 10';

(or if you are already specifying a cplex_options string, add 'mipdisplay 2 mipinterval 10' to it), and then you will see a more detailed CPLEX timing report as well as much other detailed information about the CPLEX run.

You can also look at _ampl_time which is the total CPU time so far spent in the AMPL program (but not the solver). If you think AMPL is running too slowly, then you should try to avoid nested loops as explained in a different post.

Bob Fourer
am...@googlegroups.com

=======

[maziar kermani]

Thanks for your reply. I have added these options to my code. However for the first model AMPL time is around 600 seconds and solver time around 200 seconds.

In addition I was trying to avoid using "let" within a for loop and instead use an indexed let command. (https://groups.google.com/forum/#!topic/ampl/KrdZSBaQkPM) However the problem is using let to update sets. apparently AMPL waits to evaluate all the sets once a for loop ends one cycle, however if I use the indexed let command it will evaluate all the sets after the let. In this case since some of my sets are based on the other sets, after the indexed set I get the error that for one set the index is not valid. Is there a way to avoid this checking by AMPL? I am doing all these updates in a file and then "include" it after my model and data.

Thanks in advance,

Maziar

[Robert Fourer]

Can you give a simple example in AMPL code? I have read your general description but I am still not sure as to the cause of the error.

Bob Fourer
4...@ampl.com

[maziar kermani]

Dear Bob,

Attached are 4 files that I have prepared for my case. The idea is that sets of "Units", "Streams" and "StreamsOfUnit" should be updated using handler.in.

I guess the error was coming from the fact that in file "eiampl.mod" I did not initialize 

     set StreamsOfUnit{Units} within Streams default {}; 

instead it was

     set StreamsOfUnit{Units} within Streams;

in the latter case no matter what approach I was using (in handler.in either lines 1-3 or lines 5-11) I was getting an error of not a valid index. However adding the default for StreamsOfUnit solved the issue. The thing is that at each iteration of let or for loop AMPL was checking all the sets to see if they have a valid index which I would like to avoid before solve to reduce some time in my data preparation. I would like to ask your opinion on this. 

In addition which approach would you recommend line 1-3, or line 5-11 in file "handler.in". I guess this is exactly the case you mentioned in https://groups.google.com/d/msg/ampl/KrdZSBaQkPM/NMTguQ1pBQAJ. So I would say the former approach is better.

thanks in advance,

Maziar

[Robert Fourer]

With the "let" assignment

StreamsOfUnit[(u & '_' & t)] := StreamsOfUnit[(u & '_' & t)] union {(u & '_' & t & '_' & 'q')};

you have a reference to StreamsOfUnit[(u & '_' & t)] on the right side of the := operator. Hence this assignment cannot be carried out until StreamsOfUnit[(u & '_' & t)] has a value; by adding "default {}" to your definition of StreamsOfUnit you are guaranteeing that it will have the value {} when the assignment is executed, even if no value has been explicitly given to it.

For the assignments in handler.in I would normally favor

for {u in UnitsOfType["SOWE"], t in temp}
{ let Units := Units union {(u & '_' & t)};
let Streams := Streams union {(u & '_' & t & '_' & 'q')};
let StreamsOfUnit[(u & '_' & t)] := StreamsOfUnit[(u & '_' & t)] union {(u & '_' & t & '_' & 'q')};
}

because this is the clearest description. However, if {u in UnitsOfType["SOWE"], t in temp} is a very large set and you are running into trouble with the for loop being slow to execute, then it makes sense to use three indexed "let" statements instead. The indexed "let" statements will perform the assignments in a different order, but I don't see any problem with that.

Bob Fourer
am...@googlegroups.com

[maziar kermani]

Thank you so much for your explanation. I really appreciate it and it is more clear for me now how it works.
My case itself has more indexes in the for loop and I have 50 "u"s and 25 "t"s and I am also having more things to add and more parameters to change in that which makes the include about 10 minutes. 

I will modify the file in the coming days to check the performance.

[maziar kermani]

Hello,

I have finally changed the whole structure of my include file and instead of doing many "let" commands in 4 nested for loop. I do all the for loops with "let {}" or "let x := union setof{} y" commands. The timings are greatly reduced.

Form:

_nvars = 224176

_ncons = 24113

_nobjs = 1

_snvars = 209396

_sncons = 10280

_snbvars = 1822

_snivars = 0

_solve_time = 163.785

_ampl_time = 533.555

_solve_elapsed_time = 86.362

_ampl_elapsed_time = 538.905

solve_exitcode = 0

solve_message =

'CPLEX 12.6.3.0: optimal integer solution within mipgap or absmipgap; objective 10391520.34\

13758 MIP simplex iterations\

0 branch-and-bound nodes\

absmipgap = 294.799, relmipgap = 2.83692e-05\

No basis.'

solve_result_num = 3

solve_result = solved

To:

_solve_time = 179.791

_ampl_time = 7.45685

_solve_elapsed_time = 86.846

_ampl_elapsed_time = 7.691

The ampl elapsed time due to include file has reduced significantly from 538.905 s to 7.691 s. Of course the cplex time is high but I can reduce that by mainly increase the mipgap to a more reasonable value.

Thanks a lot for your help Bob.

For anyone else who wants to do many nested loops I would recommend to combine all of them in the let commands.