Ablation Number of instances

[Thomas]

Hello, I have a problem.
I'm using the function ablation() to analyze parameter importance.
The number of train instances is about 3000.
 iraceResults$scenario$firstTest - > 5

Here is the R code I execute :
ablog <- ablation(iraceResults = iraceResults,
         ablationLogFile = "ablationLogFile.rda",
         type="full",
         seed= 1,
         n_instances=1)

The problem is that program run during hours on the initial configuration of iraceResults on all instances from the set of 3000.
I thought that it will each configuration on only n_instances* iraceResults$scenario$firstTest , so in this case: 5 instances.
I wonder how to force the algorithm to only test configurations on only 5 instances.
Maybe code test the initial configuration on all but it is too long for me.
Thank you if you have answers! 

[Manuel López-Ibáñez]

Hi Thomas,

It is great to know that more people are using the ablation code!

Are you be able to install irace from github (https://mlopez-ibanez.github.io/irace/#github-development-version) ?

If so, the interface for ablation is being revamped for irace 3.6: See https://mlopez-ibanez.github.io/irace/reference/ablation.html

ablog <- ablation( iraceResults,

                                    ablationLogFile = "ablationLogFile.rda",
                                    type="full",
                                    seed= 1,

                                    isntancesFile="instances-for-ablation.txt")

You can then create a file "instances-for-ablation.txt" listing the 5 instances that you wish to use for ablation.

Note also that there have been improvements in the plotting: https://mlopez-ibanez.github.io/irace/reference/plotAblation.html

and in the command-line interface: https://mlopez-ibanez.github.io/irace/reference/ablation_cmdline.html

Also, you can plot ablation results using ggplot2 with: https://auto-optimization.github.io/iraceplot/reference/ablation_plot.html

I hope the above is useful!

Cheers,

Manuel.

[Thomas]

Thank you very much for your quick reply!

I will do that. But I would like to know what is the meaning of parameters n_instances ?  I thought that it allowed to choose the number of instances but here, it does not work ( for me).
Thank you in advance for your help

[Manuel López-Ibáñez]

That parameter is gone in the version in github. It was too difficult to use and not what people actually want. There is a new parameter "nrep".

[Thomas]

Sorry for the multiple messages. But I just tried to use instancesFile="instances-for-ablation.txt" but it's not an ablation parameter : Unknown scenario settings given .
I also tried trainInstancesFile, this time no error but the file is not used. 

The program takes all the instances made during irace not the new set specific to ablation.

[Manuel López-Ibáñez]

Are you sure you are using the version installed from github? What does 'irace::irace.version' returns?

(by the way, if you subscribe to the group, your messages would not have to wait for moderation)

[Manuel López-Ibáñez]

But I noticed there was a typo in my example, it should be "instancesFile" not "isntancesFile":

ablog <- ablation( iraceResults,

                                    ablationLogFile = "ablationLogFile.rda",
                                    type="full",
                                    seed= 1,

                                    instancesFile="instances-for-ablation.txt")

[Thomas Feutrier]

Hello,

I am using " "3.5.6863679" version , when i'm using instancesFile, I got an error: 

"Error: == irace == Unknown scenario settings given: instancesFile"

[Manuel López-Ibáñez]

The version in GitHub is 3.5.1.22532d

https://github.com/MLopez-Ibanez/irace/blob/master/R/version.R

You need to update using the R command I mentioned earlier.

[Thomas Feutrier]

Indeed, with this version of irace: instancesFile works perfectly !

 
One last question before "closing" this conversation:
The new "nrep" corresponds replications per
instance, but what do the replications per instance correspond to? The maximum number of tested configurations per instance?

Thank you ! 

[Manuel López-Ibáñez]

I changed the docs to this:

Number of replications per instance used in \code{"full"} ablation. If \code{nrep > 1}, the each configuration will be executed \code{nrep} times on each instance with different random seeds.

Is it clearer now?

Cheers,

Manuel

[Thomas Feutrier]

Yes that is perfect like this, thank you for all your work!