No run in all instances

[D P]

I need to tune the Differential Evolution parameters to benchmark with 12 problems. I declare 12 instances in the instances document but IRACE only tests in 9 instances and sometimes in 8. My question is how I can make it run in the 12 instances. I already increased the max experiments from 250 to 500 and it is still working only on 8 or 9 instances

[Manuel López-Ibáñez]

If you want irace to see all 12 instances before removing any configuration, you can use "firstTest=12" in the scenario file. If you want that irace sees the 12 instances as a single block before any elimination test, then you should add also "eachTest=12". Of course, that would increase the computational effort since usually irace tries to remove configurations with less information.

However, the statement "I need to tune Algorithm X to benchmark Y" sounds like a conceptual error. Tuning for a specific benchmark does not show that an algorithm is good for anything but the specific problems in that benchmark. In fact, we know that there is an inflection point at which the more you tune for a benchmark, the worse your algorithm becomes for problems outside the benchmark. Reaching that point is easier with smaller benchmarks.

Best wishes,

Manuel.