Final best solution is not the best found

[Camilla Melo]

I'm using irace to optimize hiperparameters of a ConvNet. I don't have instances, so I guess each instance on irace is just a different seed, which is passed to the target-runner.

I notice that in the end of each iteration and sometimes even in the end of the irace, the configuration that appears on the best-so-far ignores new configurations when they didn't run for all instances. As an example:

Even though configuration 108 was the best for each instance on the iteration, the best-so-far configuration was 54.

In the case bellow, it was clear that despite configuration IDs 223 and 267 are way better, they are not considered because they didn't run for 1 instance.

Another example is bellow, where it finds the best solution in the end, but it takes about 2 iteration to consider it as the best so far.

So my question is if this is the expected behavior or am I missing some configuration here? Is that a way to ensure that on each itaration the new best configurations run to all the instances existent before? I'm asking this because as I don't have a "finite" number of instance, it creates a new one on each iteration.

Thanks in advance.

[Manuel López-Ibáñez]

Hi Camilla,

It would be useful to get the irace.Rdata file that you used to generate those plots and also all the textual output printed by irace . See also my answers inline below.

On Monday 1 July 2024 at 07:38:32 UTC+1 camill...@gmail.com wrote:

I'm using irace to optimize hiperparameters of a ConvNet. I don't have instances, so I guess each instance on irace is just a different seed, which is passed to the target-runner.

I notice that in the end of each iteration and sometimes even in the end of the irace, the configuration that appears on the best-so-far ignores new configurations when they didn't run for all instances. As an example:

Even though configuration 108 was the best for each instance on the iteration, the best-so-far configuration was 54.

Yes, the best-so-far configuration may not be the best of the iteration if the iteration did not see all available instances, like you show in the example above. There are multiple reasons why an iteration may be cut short, for example, because there are two few alive configurations. In that case, it would not be a good idea to choose 108 because it is the best for instances 8 and 9 when the best for all instances (1 to 9) is actually 54. But there is something strange in your example above. irace should not discard configurations after seeing only 2 instances unless you change the parameter firstTest.

In the case bellow, it was clear that despite configuration IDs 223 and 267 are way better, they are not considered because they didn't run for 1 instance.

I would like to see the irace.Rdata file for the above example. This is indeed surprising but it may make sense once one looks at what happened during the whole run of irace. It may also be a bug. Unfortunately, those happen.

 

Another example is bellow, where it finds the best solution in the end, but it takes about 2 iteration to consider it as the best so far.

I would need to see the irace.Rdata file. One reason why a configuration that looks good on average at a certain iteration is not chosen as the best is the difference in the number of instances. As a extreme case, imagine if the best-so-far has been evaluated in 100 instances and the new contender has been evaluated in just 5. Should we just discard the best-so-far and replace it with the new one?

 

So my question is if this is the expected behavior or am I missing some configuration here? Is that a way to ensure that on each itaration the new best configurations run to all the instances existent before? I'm asking this because as I don't have a "finite" number of instance, it creates a new one on each iteration.

The best-so-far configuration is always evaluated in all instances seen before. The "creation" of a new configuration at each iteration can be controlled by parameter --elitist-new-instances. Nevertheless, it would be interesting to look at the irace.Rdata file that generated the above plots to understand better what happened here. If there is a bug, I would definitely like to fix it.

Best wishes,

Manuel.

[Camilla Melo]

Hi, Manuel. Thank you for the quick response.

I'm attaching the outputs in txt formats and the .Rdata for 4 tests I've done. test 1 refers to the first screenshot, 3 and 4 are the ones that generated both boxplotssent before, in the same order.

The differences between them are: 1 and 2 are using maxTime as limit, while 3 and 4 use maxExperiments. 1,2 and 3 use parameter "firstTest = 2", while 4 use the standard value.

Just reinforcing, I don't think it should discard the best-so-far whenever a new better configuration appears without testing on all instances, but for example in test 3, I would expect that it would do the tests on remaining instance on configurations 223 and 267, as it finished with 23 on remainingBudget and there was only 7 on elite, so I think it would have enough budget to have one more test on last iteration. However, I'm looking forward to hearing your feedback.

Thanks and Regards.

--
You received this message because you are subscribed to a topic in the Google Groups "The irace package: Iterated Racing for Automatic Configuration" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irace-package/uAK4sRjx5xM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to irace-packag...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/irace-package/7574cb23-43a7-454c-80f2-1f7fac08aff8n%40googlegroups.com.

--

Att,

Camilla R. M. Fernandes

[Manuel López-Ibáñez]

Hi Camilla,

Thanks for sending this. This has helped me to improve iraceplot: https://github.com/auto-optimization/iraceplot

It will now print the mean rank as well.

Let me explain what I see in each run:

Run 1. In iteration 7, the race stops with configuration 108 being the best of the race. The race stops because only 5 candidates remain alive (<= minNbSurvival). However, we have only seen 2 instances in this race. When looking at all the results, configuration 54 is still the best one. Note that 108 is not discarded. It still appears in iteration 8. In fact, it appears at the end because irace cannot statistically discard it (but its average rank is worse than 54).

In Run 2, you are using irace version: 3.3.2238:2239. There have been many bugs fixed from version 3.3 to 3.5: https://mlopez-ibanez.github.io/irace/news/index.html#irace-35  Nevertheless, one could argue that configuration 51 looks better than the one recommended by irace (25). However, 25 is evaluated on 5 instances and 51 only in 4 and irace doesn't know what will happen when it evaluates 51 on the 5th instance. It may perform much worse than anything else. Notice that irace does return 51 among the final "Best configurations", thus you could look at the results recorded by irace for each of them (using iraceplot or directly reading the values from the logfile as explained in the user-guide) and decide to take the risk of choosing 51 based on only 4 runs.

Run 3 is also using an outdated version of irace. In this case, it is clearer that something strange is going on because iteration 20 has not consumed all its budget and it is not doing elimination tests (all alive configurations are elite), so it should continue evaluating instances. Could you re-run this experiment with --debug-level 3? Use "--seed 123456" so that irace will repeat all steps.

Thanks!

Manuel

[Camilla Melo]

Hi Manuel. Thanks for checking.

I installed irace through Anaconda, only now I noticed that it only has version 3.3.

Should I update my version of irace before running again experiment 3 with debug level 3 or do you prefer that I run it on the same version?

Thanks,

Camilla

To view this discussion on the web, visit https://groups.google.com/d/msgid/irace-package/5770db49-54ca-443f-9fb9-ad805e337c36n%40googlegroups.com.

[Manuel López-Ibáñez]

Could you run it before updating to make sure we get this result? If you update
first, it may not happen for the same seed value and we will have to wait until
you notice this case again.

Thanks!

Manuel.

[Camilla Melo]

Sure. I'm running it again.

Thanks,

Camilla

[Manuel López-Ibáñez]

Could you open an issue here https://github.com/MLopez-Ibanez/irace/issues ? It is a better place to track this topic  than the mailing list.

Best wishes,

Manuel.

[Camilla Melo]

I opened https://github.com/MLopez-Ibanez/irace/issues/69.

Thanks.