how to set maxTime without getting an error

[Luciana Salete Buriol]

Dear all,

I have the following situation:

1) An instance-list.txt file with a single instance and target-runner runs
for about 120 s per run.

2) Six parameters with the ranges below
c (TRUE, FALSE)
c (TRUE, FALSE)
i (1, 6)
c (1.005, 1.007, 1.0001)
i (0, 10)
c (0.05, 0.5)

3) The scenario.txt and target-runner are in attachment

When I run irace, I get the error that appears in the last line of the
message below (in the message I removed just the first prints of irace). In
summary, the error says that I need a maxTine of at lest 3666 sec. However, I
set maxTime to 4000. Moreover, I was expecting irace would run without error
independently of the maxTime set, even if a short time would be used. How can
I know in advance a maxTime to set without getting an error?

If you can help me on that, I would be very thankful.
Best regards,
Luciana



--- below the message of error I got after executing

# 2020-01-21 00:12:23 UTC: Estimating execution time using 2% of 4000 = 80
# 2020-01-21 00:12:23 UTC: /home/buriol/irace/target-runner 1 1 144706868
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 3 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.007 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 5 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:12:23 UTC: /home/buriol/irace/target-runner 2 1 144706868
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 2 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.005 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 0 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:14:26 UTC: DONE (1) Elapsed wall-clock seconds: 122.69
# 2020-01-21 00:14:26 UTC: DONE (2) Elapsed wall-clock seconds: 122.69
0 123
0 123
# 2020-01-21 00:14:26 UTC: /home/buriol/irace/target-runner 1 1 277353133
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 3 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.007 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 5 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:14:26 UTC: /home/buriol/irace/target-runner 2 1 277353133
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 2 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.005 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 0 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:16:29 UTC: DONE (1) Elapsed wall-clock seconds: 122.70
0 123
# 2020-01-21 00:16:29 UTC: DONE (2) Elapsed wall-clock seconds: 122.78
0 123
# 2020-01-21 00:16:29 UTC: /home/buriol/irace/target-runner 1 1 844214326
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 3 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.007 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 5 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:16:29 UTC: /home/buriol/irace/target-runner 2 1 844214326
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 2 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.005 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 0 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:18:32 UTC: DONE (1) Elapsed wall-clock seconds: 122.64
0 122
# 2020-01-21 00:18:32 UTC: DONE (2) Elapsed wall-clock seconds: 122.69
0 122
# 2020-01-21 00:18:32 UTC: /home/buriol/irace/target-runner 1 1 5546303
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 3 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.007 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 5 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:18:32 UTC: /home/buriol/irace/target-runner 2 1 5546303
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 2 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.005 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 0 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:20:34 UTC: DONE (2) Elapsed wall-clock seconds: 122.68
0 122
# 2020-01-21 00:20:34 UTC: DONE (1) Elapsed wall-clock seconds: 122.69
0 122
# 2020-01-21 00:20:34 UTC: /home/buriol/irace/target-runner 1 1 1331882145
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 3 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.007 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 5 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:20:34 UTC: /home/buriol/irace/target-runner 2 1 1331882145
/home/buriol/irace/WK30_UK -D ALGO USE_NA_BRANCH FALSE -D ALGO
ENABLE_MWIS FALSE -D ALGO LS_RESET_NOTIMPROVING_ITERS_MIN 2 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWPHI_LE_FRAC 1.005 -D ALGO
LS_RESET_NOTIMPROVING_IF_NEWROUTES_LE 0 -D ALGO iLS_RESET_DESTROY_FRAC 0.5
# 2020-01-21 00:22:37 UTC: DONE (2) Elapsed wall-clock seconds: 122.69
0 123
# 2020-01-21 00:22:37 UTC: DONE (1) Elapsed wall-clock seconds: 122.70
0 123
# 2020-01-21 00:22:37 UTC: Estimated execution time is 122.6 based on 2
configurations and 5 instances. Used time: 1226
# 2020-01-21 00:22:37 UTC: Warning: with the current settings and estimated
time per run (122.6) irace will not have enough budget to execute the minimum
number of iterations (4). Execution will continue by assuming that the
estimated time is too high and reducing the minimum number of iterations,
however, if the estimation was correct or too low, results might not be
better than random sampling.
# 2020-01-21 00:22:37 UTC: Warning: with the current settings and estimated
time per run (122.6) irace will not have enough budget to execute the minimum
number of iterations (3). Execution will continue by assuming that the
estimated time is too high and reducing the minimum number of iterations,
however, if the estimation was correct or too low, results might not be
better than random sampling.
# 2020-01-21 00:22:37 UTC: Warning: with the current settings and estimated
time per run (122.6) irace will not have enough budget to execute the minimum
number of iterations (2). Execution will continue by assuming that the
estimated time is too high and reducing the minimum number of iterations,
however, if the estimation was correct or too low, results might not be
better than random sampling.
Error: == irace == Insufficient budget: With the current settings and
estimated time per run (122.6) irace will require a value of 'maxTime' of at
least '3678'.
[buriol@mira-dev-benchmark-2a-83e2f470]~/irace%

[Manuel López-Ibáñez]

Hi Luciana,

Thanks for reporting this. There is a mistake in the number reported: irace will require a value of 'maxTime' of at least '3678' +  the used time for estimating the average runtime: 1226  = 4904

This mistake will be fixed in the next version of irace.

irace tries very hard to execute at least a minimum number of iterations, otherwise you are simply doing random search. If there is no enough time to execute that minimum, then irace gives the warning and reduces the number of iterations. However, the remaining time may be so small that not even a single iteration can be executed. That is when irace gives an error.

irace also tries very hard to not go over time: irace tries to stop before reaching the time limit, not after reaching it.

In your case, the average execution time is 122.6. Roughly speaking (see computeMinimumBudget function for details, just one iteration of irace requires (minSurvival + 1) * (firstTest+eachTest), thus 30 evaluations with default values, 122.6*30 =  3678 seconds in addition to the time consumed during the initial estimation. Of course, this is an upper bound. In fact, we re-use the results calculated during estimation if we can, so that time will be saved later. The function computeMinimumBudget could return more precise upper bounds. If you have a suggestion about it, please let us know.

Also, it seems you are using wall-clock time to measure the time require by your algorithm (as returned by date), but WC time will be affected by running configurations in parallel (and anything else running in your computer). You should use CPU time.

Cheers,

Manuel.