Debugging the target runner

[Amirhossein Moosavi]

Hello Manuel,

I hope you are well. I have two questions regarding the target function of the iRace algorithm, which I will explain below.

I have coded the target-runner in Python in order to link iRace (in RStudio) with my algorithm (coded in Matlab). I have included four text files in order to define:

(1) parameters: there is only one integer parameter (--nPop)

nPop     "--nPop "   i      (100, 4000)

(2) configurations (I have introduced an initial value for --nPop)

nPop
800

(3) instances (I have defined the title of instances that must be passed to my Matlab codes)

# data1
# data2
# data3
1
2
3

(4) scenario.

The target runner must call the Matlab function with two input arguments: (1) --nPop and (2) the title of instances. Both of these input arguments must be double values.

For target runner, I have used the template target-runner.py file that is provided by iRace. As I understood, the iRace passes five input arguments to the target runner as follows:

configuration_id = sys.argv[1]
instance_id = float(sys.argv[2])
seed = sys.argv[3]
instance = sys.argv[4]
cand_params = sys.argv[5:]

In order to determine --nPop, I am using the below code:

while cand_params:
    # Get and remove first and second elements.
        param = cand_params.pop(0)
        value = cand_params.pop(0)
        if param == "--nPop":
                nPop = float(value)
        else:
                target_runner_error("unknown parameter %s" % (param))

And for the instance number, I am passing float(instance_id) to the Matlab function. Would you please confirm that I am passing the correct information to my Matlab function?

Unfortunately, I do not know how to debug and check the input and output arguments of iRace and the target runner. Do you have any suggestions for me on how I can debug these scripts and see check whether they are working properly?

Many Many thanks, Amirhossein.

[Manuel López-Ibáñez]

Dear Amirhossein,

Without having access to the complete scripts, it is difficult to tell if everything is correct. But so far, your explanation sounds correct. I'd suggest to also check the Appendix B in the user guide in case of problems: https://mlopez-ibanez.github.io/irace/irace-package.pdf

Also, you can use "irace --check --debug-level 3" for a quick check that everything is working as it should.

Normally, the hardest part is not to communicate with irace, but to make your algorithm (specially MATLAB) do what irace is expecting. Therefore, I'd recommend that you run target-runner.py outside irace to make sure it works before using it within irace. How do you call a MATLAB function from Python?

Cheers,

Manuel.

[Amirhossein Moosavi]

Hi Manuel,

Sorry for my late reply. I was finally able to run iRace! I used RStudo to run iRace, Python for the target-runner, and MATLAB for my algorithm. In order to call MATLAB from Python, I used "matlab.engine." I am willing to write a step-by-step guide for the users that want to use iRace under this setting. Please let me know if that could be helpful.

Now, I have two questions. I have a computationally expensive algorithm. Therefore, I should be so careful in determining the termination criterion and should accelerate iRace as much as possible. I have ten test instances and 32 sets of configurations in total. I know that there is no general rule, and it might depend on the variability of the performance of the algorithm, but in your opinion, how large maxExperiment should be set? I have set it equal to 200 for now.

In addition, I want to use parallelization to accelerate iRace or use the created log file (recoveryFile) to be able to start running iRace multiple times. For example, for parallelization, I add "parallel = 1" to the scenario file, and run iRace in RStudio by "irace.cmdline("--parallel 3")". Is that correct? I am not sure whether the algorithm is using the parallelization option. If my approach is correct, would it be possible to help how I can activate both recoveryFile option and parallelization options at the same time in the RStudio console?

Best, Amirhossein.

[Amirhossein Moosavi]

Another question! In parallelization, how much should I set the parallelization value for a normal laptop with a Core i7 CPU and 16 GB of RAM? In the above example, I have set it equal to 3.

Many thanks, Amirhossein.

[Manuel López-Ibáñez]

Dear Amirhossein,

If you can write some step by step guide, that would be great. I know many users struggle to call MATLAB from irace. If you don't want to share your algorithm, you could replace your target-algorithm by some "dummy" algorithm.

About the maxExperiments setting, it depends on so many things:...How many parameters you have? How large is your configuration space? How many instances do you estimate should a good configuration should see to trust its performance? How long can you run? 200 is quite low but if irace runs and you cannot do more, then it is what it is. irace may refuse to run if you have too many parameters and not enough experiments.

If each run of your algorithm may require different runtimes, it may be a good idea to specify a maximum time instead of a maximum number of experiments (but then the target-runner has to tell irace how long the run took).

If your algorithm is anytime (http://lopez-ibanez.eu/publications#LopStu2013ejor), you may wish to use the capping methods discussed here:  https://github.com/souzamarcelo/capopt

In a nutshell, the capopt "plugin" to irace allows irace to stop (cap) long running executions if they do not show progress so that time can be spent on exploring other configurations. If you decide to use this (it may require some changes in your target-algorithm and target-runner), please let me know and I can recommend you good settings.

Yes, if you use irace.cmdline("--parallel 3") that should execute 3 runs of the target-runner in parallel. If you open the Task Manager in Windows (there must be something equivalent in MacOS) you should see three MATLAB processes running.

The above will run irace one time (one copy). If what you want is multiple copies of irace running in parallel (instead of one copy of irace running multiple copies of target-runner in parallel), then you need to open multiple R consoles in Rstudio and create different execution directories and tell irace to use them. For example, in one R console you will call:

dir.create("execdir1")

irace.cmdline("--exec-dir execdir1")

then in a different R console you will call:

dir.create("execdir2")

irace.cmdline("--exec-dir execdir2")

and so on.

The recoveryFile option is only useful to continue a run of irace that was previously stopped. It has nothing to do with running irace in parallel.

I hope the above helps!

Manuel.

[Manuel López-Ibáñez]

On Tuesday, 2 March 2021 at 18:44:08 UTC+1 amirhossei...@gmail.com wrote:

Another question! In parallelization, how much should I set the parallelization value for a normal laptop with a Core i7 CPU and 16 GB of RAM? In the above example, I have set it equal to 3.

I don't know how many cores your laptop has. That information is usually in the Task Manager or in the control panel in Windows, no? It does not make sense to set --parallel to a number larger than the number of cores. If the performance of your algorithm is time sensitive, running in parallel may affect runtime measures (due to competition of resources since there may be other things running in your laptop that may steal the CPU from MATLAB), so you have to take that into account.

It also depends on how much RAM  each copy of your MATLAB algorithm requires. Irace itself should not consume much memory, but MATLAB might. Do some preliminary experiments and look at the performance indicators (memory consumed, cpu usage, etc) in the Task Manager.

Cheers,

Manuel.

[Amirhossein Moosavi]

Your explanations are really helpful. Thank you so much! Please let me clarify that I am using Linux Ubuntu as my OS.

I will try to prepare a step-by-step guide and try to include a sample algorithm (a basic genetic algorithm).

I have the CPU time of the algorithm as one of the performance metrics (I do not want to have a setting that takes very long to run but provides solutions that are not much better as compared to other settings). Thus, according to your suggestion, I cannot use parallelization. Is that right? However, I still think parallelization should not make a problem for my case. If iRace is running multiple settings at the same time, so, the CPU time of all runs are deteriorated at the same time. It means that the CPU time of all settings might increase, and the final comparison might be fair. Am I correct?

Regarding running one or multiple copies of iRace, I have only an algorithm, and I want to tune its parameters. I do not think that running multiple copies of iRace could make any difference for me. If I am wrong, please let me know.

With respect to the recoverFile option, I want to use this file to be able to continue a run of iRace that was previously stopped (as mentioned my algorithm is computationally expensive, and it might not be possible to run it at once.) With that being said, would it be possible to let me how I can continue a run with a currently available log file in the working directory (in other words, how should I call iRace with the recoverFile option from RStudio's console)? What if I want to use both recoverFile and parallelization options at the same time?

Many thanks for your consideration and time.

Best, Amirhossein.

[Manuel López-Ibáñez]

In Linux you can find how many cores you have by opening a terminal and running "cat /proc/cpuinfo". You can find the same information from Rstudio using parallel::detectCores().

Yes, you can still use parallelization, but if you overload the computer, the time measurements will be affected slightly. If you are measuring differences in the order of minutes, it should not matter. If you are measuring differences in seconds, then results may be affected. It is not possible to predict which runs may be affected since irace cannot keep all CPUs equally busy all the time (and your computer may run other things in parallel that only affect some CPUs). If you are concerned about this, it may be a good idea to leave one CPU empty.

You are right that in your setting, it may not make sense to run multiple copies of irace in parallel.

To use the recovery, you just need to do something like:

irace.cmdline("--recovery-file recovery.Rdata")

or

irace.cmdline("--recovery-file recovery.Rdata --parallel 2")

where recovery.Rdata is the irace.Rdata file that irace generates. It is recommended to rename it (or change its location) before recovery because irace will generate another irace.Rdata that will overwrite the existing one.

I hope the above helps!

Manuel.

[Amirhossein Moosavi]

Now, everything makes much more sense. Thanks a lot! I will get back to you once I have prepared the step-by-step guide.

Have a great day, Amirhossein.

[Elcio Silveira]

Hello Amirhossein. 

I need to use IRACE to help define some parameters of my algorithm that was encoded in MatLab.

I would like to know if you could provide me the adaptations that you have made in the iRace to work with MatLab?

If possible, please contact me by e-mail elcio.s...@gmail.com

Thank you.