CPLEX pool option without generating solfile

[Alba Victoria]

Hi everyone!

I am interested on using the pool solution of CPLEX to generate multiple n-dimensional optimals solutions x* of an integer LP. Furthermore, I want to store these solutions into a parameter X_sol{1..n,1..#optimal solutions}. I have been so far using the pool option with poolstub="filename" and retrieving the solutions as

let s=0;

for {i in 1 .. Current.npool} { 

  solution ("filename" & i & ".sol"); 

let{k in 1..n} X_sol[k,s]:=x[i]; 

let s:=s+1;}

It turns out that the solving times are pretty small, but AMPL times are really large (I assume that this arises from creating the files, reading the solution from them, and storing the values in X_sol).  For example, for a particular instance I manage to generate 105724 solutions with solving times of 3105 but AMPL times of 239164.

This makes the whole algorithm I am implementing completely intractable; is there any option to store the solutions CPLEX+pool opton is delivering into X_sol without creating the files and reading from them?

Thanks for your help!

Alba

[AMPL Google Group]

CPLEX can compute multiple solutions in one solve run, but AMPL needs to read the solutions one at a time from files. There is not an option to get the solutions any other way.

Are you writing 105724 files and then reading them in the AMPL for-loop? Then AMPL is taking about 2.25 seconds per file, but still that adds up to a lot. There are some things you could do to test what is taking all the time and to speed up the loop.

First, AMPL normally displays a solver message every time that a "solution" command is executed, and that is inefficient. You can turn off those solver messages by adding this command (before the loop):

option solver_msg 0;

Second, it seems to me that the variables should be x[k] (running over k in 1..n}, and also that you can make the loop more efficient by not using s:

for {i in 1 .. Current.npool} {
   solution ("filename" & i & ".sol");
   let {k in 1..n} X_sol[k,i] := x[k];
};

Third, you should check how many numbers are being stored in X_sol, by giving this command (after solving):

print n * Current.npool;

If this number is more than 100 million, you may be running out of physical memory to store X_sol, which will slow down the "let" statement considerably. If this is the situation, there are some more tests you can do to measure the slowdown.

Finally, it can make a difference how you measure time. Are you using _ampl_time, or _ampl_elapsed_time, or some other measure?

--
Robert Fourer
am...@googlegroups.com

{#HS:1890120811-110112#}

[Alba Victoria]

Hi Robert,

Thank you for your quick reply. I'll implement your suggestions and check if they improve the running times enough for me to run my algorithm. In the meantime, let me reply to your questions one by one to see if you could give me any more tips to make this whole procedure faster:

1. Yes, at the end of the day I am creating 105724 files and then reading them in the AMPL for-loop. I do generate those files in batches though: they come from solving 1000 problems. I solve one problem, generate around 0-50 files, read the solutions from those files into X_sol, and go into solving the next problem. Hence I never have 105725 files; I use the same poolstub so the next problem I solve rewrites the previous files and I do not end having a huge amount of .sol files in my folder.
2. n varies between 2 and 20, so the largest instance I have solved would likely have an X_sol storing more than 2 millions numbers. At the moment I do have a 2Gb memory per core, but I could use more.
3.  I am using _ampl_time and _total_solve_time to measure the times.

Thanks for your help!

Alba

[AMPL Google Group]

That information is helpful, and it suggests two more questions:

What is your AMPL definition of X_sol? It seems you start writing to X_sol before you know the total number of solutions that will be found, so it may be important consider how X_sol is handled.

Also, what are the timings if you solve just the first problem, or the first 10 problems? The answer may give an idea of whether the difficulty is only with file read times, or whether it involves something else that becomes expensive when you solve larger numbers of problems.

--
Robert Fourer
am...@googlegroups.com

{#HS:1890120811-110112#}

On Mon, May 23, 2022 at 10:33 AM UTC, AMPL Modeling Language <am...@googlegroups.com> wrote:

Hi Robert,

Thank you for your quick reply. I'll implement your suggestions and check if they improve the running times enough for me to run my algorithm. In the meantime, let me reply to your questions one by one to see if you could give me any more tips to make this whole procedure faster:

1. Yes, at the end of the day I am creating 105724 files and then reading them in the AMPL for-loop. I do generate those files in batches though: they come from solving 1000 problems. I solve one problem, generate around 0-50 files, read the solutions from those files into X_sol, and go into solving the next problem. Hence I never have 105725 files; I use the same poolstub so the next problem I solve rewrites the previous files and I do not end having a huge amount of .sol files in my folder.
2. n varies between 2 and 20, so the largest instance I have solved would likely have an X_sol storing more than 2 millions numbers. At the moment I do have a 2Gb memory per core, but I could use more.
3. I am using _ampl_time and _total_solve_time to measure the times.

Thanks for your help!
Alba

On Fri, May 20, 2022 at 4:50 PM UTC, AMPL Google Group <am...@googlegroups.com> wrote:

CPLEX can compute multiple solutions in one solve run, but AMPL needs to read the solutions one at a time from files. There is not an option to get the solutions any other way.

Are you writing 105724 files and then reading them in the AMPL for-loop? Then AMPL is taking about 2.25 seconds per file, but still that adds up to a lot. There are some things you could do to test what is taking all the time and to speed up the loop.

First, AMPL normally displays a solver message every time that a "solution" command is executed, and that is inefficient. You can turn off those solver messages by adding this command (before the loop):

option solver_msg 0;

Second, it seems to me that the variables should be x[k] (running over k in 1..n}, and also that you can make the loop more efficient by not using s:

for {i in 1 .. Current.npool} {
   solution ("filename" & i & ".sol");
   let {k in 1..n} X_sol[k,i] := x[k];
};

Third, you should check how many numbers are being stored in X_sol, by giving this command (after solving):

print n * Current.npool;

If this number is more than 100 million, you may be running out of physical memory to store X_sol, which will slow down the "let" statement considerably. If this is the situation, there are some more tests you can do to measure the slowdown.

Finally, it can make a difference how you measure time. Are you using _ampl_time, or _ampl_elapsed_time, or some other measure?

--
Robert Fourer
am...@googlegroups.com