Inquiry about implementation of irace

[Dan Liu]

Hi there,

I am currently trying to use irace to analyze our results and find the best parameters that lead to the best result. But I really have difficulty understanding the concepts and applying irace in our scenario. 

The data that will be used in irace are like (100 rows for example)

par1  par 2  par 3  outcome

1.5     0.1     a        8.2

0.9     0.2     a        11.3

1.8     0.2     b        10.4

2.3     0.3     a        9.2

4.9     0.3     b        13.1

......

I am hoping to find the best parameters (par1, par2, par3) that minimize the outcome. I am able to write out the parameters (par1, 2 are real values, and par3 is categorical). But I don't know how I should write the targer_runner function. It seems that the outcome is already known, rather than a function related to par1, par2, and par3 in my case. 

I checked a few examples online, such as f_rastrigin and rastrigin_run, they all need to write an objective function that includes the parameters, while in my case, the outcome is already there, and this is the data I need to use to find the best parameters (par1, par2, par3) that minimize the outcome. So I wonder about how to implement irace and write a correct targer_runner function in this case. Also, the instances that will be implemented should be this full dataset, if I want to use all the rows, correct? 

Thank you very much for your help in advance! If there are similar examples, they are also appreciated. 

Best,

Dan

[Manuel López-Ibáñez]

Could you explain what you are trying to do?

irace is typically used to tune the parameters of an algorithm over a number of training instances (datasets, problem instances, scenarios, etc) that are representative of a larger instance space and the goal is to obtain a parameter configuration that performs well over the larger instance space.

Best wishes,

Manuel.

[Dan Liu]

Hi Manuel,

Sorry for the confusion. I am trying to apply my method to find the best parameters using irace. So, the table above is the mid-result where the best parameter is found, corresponding to the best result value. 

Now, I am trying to implement the irace with my target_runner function. But it always reports an error when I check the irace scenario checkIraceScenario(..): 

_______________________________________________________________________________________________

# 2023-02-XXXXXX: Checking target runner.
# Executing targetRunner ( 2 times)...

# Error occurred while executing targetRunner:
incorrect number of dimensions
# targetRunner returned:

NULL
Error: == irace == Check unsuccessful

_______________________________________________________________________________________________

The code is:

# target runner function
target.runner <- function(experiment, scenario) {
 
  instance <- experiment$instance
  configuration <- experiment$configuration

  res <- myfunction(instance, configuration, method=1)
 
  return(list(cost = res$value))
   
}

# scenario
scenario <- list(targetRunner = target.runner,
                 instances = data[1:500,],  
                 maxExperiments = 200,  
                 logFile = "")  

# check that the scenario is valid. 
checkIraceScenario(scenario, parameters = parameters)
_______________________________________________________________________________________________

I have a dataset called data. Only the first 500 rows are used as the instance (all columns will be included). If I assign the data to be the instance and configuration values are set for each parameter. There is a cost value for res. However, if I run this chunk of code, there is an error for the incorrect number of dimensions. I wonder what the cause of an incorrect number of dimensions can be in this case. 

Thank you again!

Best,

Dan

[Manuel López-Ibáñez]

Hi Dan,

Is `myfunction(instance, configuration, method=1)` returning a list that contains "value" ? The problem seems to be in `myfunction` and the `incorrect number of dimensions` seems to happen inside your `myfunction`.

This code is also wrong:

 instances = data[1:500,], 

instances must be either a one dimensional vector or a list. If you scenario only has one instance and this instance is represented by `data` then, you should do:

instances = list(data[1:500,])

Complete example:

library(irace)
data <- matrix(runif(1000), nrow=500)

# target runner function
target.runner <- function(experiment, scenario) {
  instance <- experiment$instance
  configuration <- experiment$configuration

 # I'm not saying that doing this makes any sense, just illustrating how a matrix could be used as a single problem instance.

  res <- list(value = instance[sample(nrow(x), 1), as.numeric(configuration[["mode"]])])
  return(list(cost = res$value))
}

parameters <- readParameters(text='
mode "" c (1,2)
')

scenario <- list(targetRunner = target.runner,

                 instances = list(data[1:500,]),  

                 maxExperiments = 200,  
                 logFile = "")  

checkIraceScenario(scenario, parameters = parameters)