Questions about configurations and seed

[Sarah Lee]

Hello irace team,

I am a researcher working on a project that needs irace to tune the parameters of my algorithm. It ran all well, and it produced several best configurations. However, when I tested them on my algorithm, I found these best configurations led to different results (or cost values). I thought they should yield the same cost values as the inputs (instances) were the same. It seems some online examples, such as the example in the irace package file, also share the same phenomenon. How should it be interpreted? 

Another question is that if I want the results (best configuration) to be reproducible, do I just need to put a seed in the scenario list? Can I use the seed in the experiment instead? What is the difference between the seed in the experiment and the seed in the scenario list (except the seed number is different)? 

Thanks in advance.

Best,

Sarah Lee

[Manuel López-Ibáñez]

Hi Sarah,

If the results are different is either because the algorithm has some stochasticity (which you may be able to control specifying a random seed) or because there are some external factors (like CPU-load) that affect the execution.

If the algorithm is stochastic, it will hopefully allow you to control its random seed. If so, you should use the random seed provided by irace to the target runner to set the random seed of the algorithm. irace will generate a different random seed for each instance and for each replication (repetition of the same configuration on the same instance).

The random seed in the scenario controls the stochastic behavior of irace. If you want the behavior of irace to be repeatable then you need two things:

1) Set irace's random seed in the scenario to some fixed value. This will make irace generate the same sequence of instances, configurations and random seeds for the target-runner.

2) The target algorithm has to obey the random seed passed by irace to the target-runner.

Note that if the target-algorithm is not repeatable (that is, setting its random seed is not enough to make it deterministic because its result depends on CPU load or some library or package whose random seed cannot controlled), then irace's behavior is also not repeatable, because irace will make different decisions depending on how the target algorithm behaves.

I use the term 'repeatable'  to mean `Exactly repeat the original experiment, generating precisely the same results` as defined in (Reproducibility in Evolutionary Computation, 2021 https://dl.acm.org/doi/10.1145/3466624) because `reproducible` may have different meanings.

If you can think where in the documentation this could be better explained, please feel free to suggest what to change.

I hope the above helps!

Cheers,

Manuel.

[Sarah Lee]

Hi Manuel,

Thanks for the reply. It is very helpful!

Just a few things to clarify, is the random seed provided by irace and passed to the target algorithm the same as the random seed in the scenario list? If so, my understanding is that I should pass the scenario$seed to the target algorithm in the target runner function in order to get the reproducible best configurations. Is that correct? 

Moreover, if the final best configurations are several and I want to test them on the target algorithm for cost values, I don't quite understand why the cost values are not the same, assuming that everything else is the same, except the parameter values to be used in the target algorithm. I tried the example from the package file:

scenario <- list(targetRunner = target_runner,
                 instances = weights[1:10],
                 maxExperiments = 1000,

                 seed=12345,
                 # Do not create a logFile
                 logFile = "")

target_runner <- function(experiment, scenario) {
  instance <- experiment$instance
  configuration <- experiment$configuration
  D <- 3
  par <- runif(D, min = -1, max = 1)
  fn <- function(x) {    # function to be optimized, x can be in any dimension
    weight <- instance
    return(weight * f_rastrigin(x) + (1 - weight) * f_rosenbrock(x))
  }
  res <- stats::optim(par, fn, method="SANN",
                      control=list(maxit=1000
                                   , tmax = as.numeric(configuration[["tmax"]])
                                   , temp = as.numeric(configuration[["temp"]])))
  return(list(cost = res$value))
}

tuned_confs <- irace(scenario = scenario, parameters = parameters)

test <- function(configuration) {
  res <- lapply(weights[1:10],
                function(x) target_runner(   # run target_runner using default values
                  experiment = list(instance = x,
                                    configuration = configuration),
                  scenario = scenario))
  return (sapply(res, getElement, name = "cost"))
}

tuned <- test(removeConfigurationsMetaData(tuned_confs[1,]))

In this example, I used the same instances (weights[1:10]) in the test function as the ones (weights[1:10]) used for irace tuning. But the tuned_confs produces 3 best configurations, and the tuned values of tuned_confs[1,] are different from the tuned values of tuned_confs[2,] for instance. Is there some reason why this happens? 

Thanks a lot!

Sarah

[Manuel López-Ibáñez]

On Friday, 3 March 2023 at 17:56:55 UTC sarahk...@gmail.com wrote:

Just a few things to clarify, is the random seed provided by irace and passed to the target algorithm the same as the random seed in the scenario list? If so, my understanding is that I should pass the scenario$seed to the target algorithm in the target runner function in order to get the reproducible best configurations. Is that correct?

It is not the same. irace generates a unique random seed for each call to target runner with the same instance and the same configuration (so that they are different in some way). The global random seed in the scenario affects the sequence of target runner random seeds generated.

 

irace already generates and passes  'experiment$seed'. You do not need to pass anything to target-runner. Your algorithm has to use it.

You can find the list of random seeds generated in irace.Rdata (you can also use https://auto-optimization.github.io/iraceplot/articles/example/report_example.html to visualize it). Or you can save them to some log file within target.runner.

Moreover, if the final best configurations are several and I want to test them on the target algorithm for cost values, I don't quite understand why the cost values are not the same, assuming that everything else is the same, except the parameter values to be used in the target algorithm. I tried the example from the package file:

Yes, this example is not as good as it should be. It should set/restore the random seed before calling optim(). Something like:

res <-withr::with_seed(instance$seed,

           stats::optim(par, fn, method="SANN",
                      control=list(maxit=1000
                                   , tmax = as.numeric(configuration[["tmax"]])

                                   , temp = as.numeric(configuration[["temp"]]))))

 

Unfortunately, that would require adding a new dependency on the package 'withr'. Perhaps we should do it. This package is probably useful for other things.

Best,

Manuel.

[Sarah Lee]

Hi Manuel,

Thank you for your clarifications. 

On Saturday, March 4, 2023 at 2:47:41 AM UTC-7 Manuel López-Ibáñez wrote:

It is not the same. irace generates a unique random seed for each call to target runner with the same instance and the same configuration (so that they are different in some way). The global random seed in the scenario affects the sequence of target runner random seeds generated.

 

irace already generates and passes  'experiment$seed'. You do not need to pass anything to target-runner. Your algorithm has to use it.

Okay, so I understand that 'scenario$seed' controls over the target runner random seeds passed to 'experiment$seed'. Then, in the case where my algorithm needs a seed to control the stochasticity, I should use and pass the 'experiment$seed' in my algorithm. 

 

Yes, this example is not as good as it should be. It should set/restore the random seed before calling optim(). Something like:

res <-withr::with_seed(instance$seed,

           stats::optim(par, fn, method="SANN",
                      control=list(maxit=1000
                                   , tmax = as.numeric(configuration[["tmax"]])

                                   , temp = as.numeric(configuration[["temp"]]))))

 

 I tried your code with 'withr' package. Did you mean to use 'experiment$seed inside the with_seed() function in this example? The instance$seed incurred errors. 

Best regards,

Sarah Lee

[Manuel López-Ibáñez]

Okay, so I understand that 'scenario$seed' controls over the target runner random seeds passed to 'experiment$seed'. Then, in the case where my algorithm needs a seed to control the stochasticity, I should use and pass the 'experiment$seed' in my algorithm. 

Correct.

 

Yes, this example is not as good as it should be. It should set/restore the random seed before calling optim(). Something like:

res <-withr::with_seed(instance$seed,

           stats::optim(par, fn, method="SANN",
                      control=list(maxit=1000
                                   , tmax = as.numeric(configuration[["tmax"]])

                                   , temp = as.numeric(configuration[["temp"]]))))

 

 I tried your code with 'withr' package. Did you mean to use 'experiment$seed inside the with_seed() function in this example? The instance$seed incurred errors. 

Yes, you are correct. Apologies for the mistake .

Fixed now here: https://mlopez-ibanez.github.io/irace/reference/irace.html#ref-examples

[Sarah Lee]

Okay, that is no problem. Thanks very much for the clarifications! :-)

Sarah