Problems executing irace tutorial (ACOTSP)

[bern...@gmail.com]

Hello.

I'm trying to execute the ACOTSP example (acotsp-example.tar.gz) in Irace, to get some knowledge about this interesting package.
But I'm having some problems to test this example. Please find the following questions. Maybe some of these questions are trivial. I tried to solve it myself, but I could not:

1) After run the check command irace --scenario scenario.txt --check, it is shown the following messages in the console:
    The output of targetRunner should not be more than two numbers!
    .........
    .........
    The output was:
    ECHO est desactivado (comment: desactivado means Deactivated)
    This is not a bug in irace, but means that something failed when running the commands...
    ........
    Execution halted
   
   If I execute irace --scenario scenario.txt, it is shown some similar messages to above.

   In these cases, the target-runner.bat file (I'm running in Windows), includes the 'exe' variable which is pointing
    to the acotsp file (without sufix: neither .c, neither .o) included in the project.
    Should I compile the entire project to get an executable file, or that file is Ok?

2) I suppose  it is only neccesary to set the executable program path (the 'exe' variable) in the target-runner.bat file.
    And other parameters like candidate, instance_id, seed or instance is set by irace by means of parameters-acotsp.txt and scenario.txt files.
    Is it true?
   

3) I have some doubts related to the acotsp program parameters:
    I found the main method in acotsp.c. And in this method it is shown: "input none" related to argc and argv variables.
    But I suppose that this method must input some flag parameters defined in parameters-acotsp.txt file like "--alpha ", "--beta ", 
    the instance file, etc. I'm sure I'm in a mistake and the acotsp program gets this information in some way, but I don't know how.

Sorry for the inconvenience, and thank you in advance.

Best regards
Bernardo.

[Leslie Perez Caceres]

On Wednesday, 29 March 2017 00:39:40 UTC+2, bern...@gmail.com wrote:

Hello.

Hello Bernardo, 

 

I'm trying to execute the ACOTSP example (acotsp-example.tar.gz) in Irace, to get some knowledge about this interesting package.
But I'm having some problems to test this example. Please find the following questions. Maybe some of these questions are trivial. I tried to solve it myself, but I could not:

1) After run the check command irace --scenario scenario.txt --check, it is shown the following messages in the console:
    The output of targetRunner should not be more than two numbers!
    .........
    .........
    The output was:
    ECHO est desactivado (comment: desactivado means Deactivated)
    This is not a bug in irace, but means that something failed when running the commands...
    ........
    Execution halted
   
   If I execute irace --scenario scenario.txt, it is shown some similar messages to above.

This  message is telling you there is a problem with the target runner, it is expecting to obtain from it the evaluation of a configuration and instead is obtaining the line "ECHO est desactivado ". Could you please post the code you have in your target runner so we can help you a bit more?

   In these cases, the target-runner.bat file (I'm running in Windows), includes the 'exe' variable which is pointing
    to the acotsp file (without sufix: neither .c, neither .o) included in the project.
    Should I compile the entire project to get an executable file, or that file is Ok?

Yes, you must compile ACOTSP in order to obtain the acotsp executable and write the correct path in the exe variable. 

2) I suppose  it is only neccesary to set the executable program path (the 'exe' variable) in the target-runner.bat file.
    And other parameters like candidate, instance_id, seed or instance is set by irace by means of parameters-acotsp.txt and scenario.txt files.
    Is it true?

Yes, in the target runner (if you are using the template example) you must only set the exe variable. The other variables are passed by irace when executing target runner. You need only to change the scenario.txt file to set the correct paths to the instances and configuration files  (parameters-acotsp.txt, etc...)

   

3) I have some doubts related to the acotsp program parameters:
    I found the main method in acotsp.c. And in this method it is shown: "input none" related to argc and argv variables.
    But I suppose that this method must input some flag parameters defined in parameters-acotsp.txt file like "--alpha ", "--beta ", 
    the instance file, etc. I'm sure I'm in a mistake and the acotsp program gets this information in some way, but I don't know how.

In order to execute acotsp you must compile it to obtain the executable. Then you can manually execute it with the flags of its parameters, for example:

acotsp -i instance.tsp --alpha 0.5 --beta 0.6 --tours 2000

This is what the target runner does for irace, irace gives him as input the instance and parameters and the target runner must execute it and return the value of the evaluation.

Sorry for the inconvenience, and thank you in advance.

I hope this can help you, 

Bests,

Leslie.

Best regards
Bernardo.

[bern...@gmail.com]

Hello again,

I can't solve the error running the acotsp program with irace.

I followed theses steps (in Windows):

1) I got the ACOTSP (ACOTSP 1.04-tuning) project files and included them in a Eclipse C project.
2) I built an exe file by means of the MinGW compiler (I obtained the AcoTsp.exe file)
3) I executed this exe file:
    AcoTsp.exe -i d198.tsp --alpha 0.5 --beta 0.6 --tours 2000
4) It is shown on the console iteration results. I.e.:
    ..........
    try 9, Best 15790, found at iteration 5, found at time 0.1350000
        t_avgbest = 4.357100
        t_avgtotal = 10.010100

    It seems there is not any problem in the run (you could find a more detailed result in the first image, at the top of this message).



5) I set the target-runner.bat file modifiyng (ONLY) the exec line:
    SET "exe=C:\Users\berna\Documents\doct\doct\irace\acotsp\2\acotsp-example\acotsp-example\AcoTsp.exe"
   
    But you could see the entire file in the following lines:

::##############################################################################
:: This script is the command that is executed every run.
:: Check the examples in examples/
::
:: This script is run in the execution directory (execDir, --exec-dir).
::
:: PARAMETERS:
:: %1 is the candidate configuration number
:: %2 is the instance ID
:: %3 is the seed
:: %4 is the instance name
:: The rest (%* after `shift 4') are parameters to the run
::
:: RETURN VALUE:
:: This script should print one numerical value: the cost that must be minimized.
:: Exit with 0 if no error, with 1 in case of error
::##############################################################################

@echo off

:: Please change the path to the correct one
:: SET "exe=D:\bin\acotsp"
SET "exe=C:\Users\berna\Documents\doct\doct\irace\acotsp\2\acotsp-example\acotsp-example\AcoTsp.exe"
SET "fixed_params=--time 20 --tries 1 --quiet"

SET candidate=%1
SHIFT
SET instance_id=%1
SHIFT
SET seed=%1
SHIFT
SET instance=%1
SHIFT

SET candidate_parameters=%*

SET "stdout=c%candidate%-%instance_id%.stdout"
SET "stderr=c%candidate%-%instance_id%.stderr"

::if not exist %exe% error "%exe%: not found or not executable (pwd: %(pwd))"

:: If the program just prints a number, we can use 'exec' to avoid
:: creating another process, but there can be no other commands after exec.
::exec %exe %FIXED_PARAMS% -i %instance %candidate_parameters%
:: exit 1
::
:: Otherwise, save the output to a file, and parse the result from it.
:: (If you wish to ignore segmentation faults you can use '{}' around
:: the command.)
%exe% %FIXED_PARAMS% -i %instance% --seed %seed% %candidate_parameters% 1>%stdout% 2>%stderr%

::if %errorlevel% neq 0 exit /b %errorlevel%

:: # This may be used to introduce a delay if there are filesystem
:: # issues.
:: SLEEPTIME=1
:: while [ ! -s "%stdout%" ]; do
::     sleep %SLEEPTIME
::     let "SLEEPTIME += 1"
:: done
::
:: This is an example of reading a number from the output.
:: It assumes that the objective value is the first number in
:: the first column of the last line of the output.

::if exist %stdout% (

    setlocal EnableDelayedExpansion
    set "cmd=findstr /R /N "^^" %stdout% | find /C ":""

    for /f %%a in ('!cmd!') do set numberlines=%%a

    set /a lastline=%numberlines%-1

    for /f "tokens=3" %%F in ('more +%lastline% %stdout%') do set COST=%%F
   
    echo %COST%

    ::del %stdout% %stderr%
   
    exit 0
   
::) ELSE (

    ::exit /b
::)



6) I run the check command and the irace command, but there are some problems: In the second image at the top of this message, you can find a more detailed
information about this problem

irace --scenario scenario.txt

I run this sentence in the irace configuration files path

And finally, I have a doubt:
I read in The irace Package: User Guide document that "the program targetRunner must print a real number,
which corresponds to the cost measure of the candidate configuration for the given instance
and optionally its execution time (mandatory when maxTime is used)". Then I suppose that the executable program (AcoTsp) must output ONLY a real number. But I noted ACOTSP program output more than a real number. Perhaps this error is happened because last sentences of target-runner.bat cannot get properly the output of acotsp executable file.

I sorry for the inconvenience.

Thank you!
Bernardo.

[Manuel López-Ibáñez]

Leslie, you added the file. Could you comment on what the lines below are doing?

On Thursday, 30 March 2017 00:08:36 UTC+1, berni.ep4 wrote:

    echo %COST%

I think the output that you see is caused because %COST% doesn't have a value so Windows calls "echo" which returns "echo está desactivado"

Why it doesn't have a value? My BAT knowledge is limited and I'm not sure what the following is doing exactly:

    set "cmd=findstr /R /N "^^" %stdout% | find /C ":""

    for /f %%a in ('!cmd!') do set numberlines=%%a

    set /a lastline=%numberlines%-1

    for /f "tokens=3" %%F in ('more +%lastline% %stdout%') do set COST=%%F

This is all much easier in Linux or OSX. If you understand BAT programming or know someone who does, perhaps they can suggest a more robust way to parse the output of ACOTSP. The Linux version does:

COST=$(cat ${STDOUT} | grep -o -E 'Best [-+0-9.e]+' | cut -d ' ' -f2)
echo "$COST"

which means: Pipe the stdout file to a command that looks for a line that contains "Best" followed by a floating-point number, then keep just the floating-point number.
If you know how to do the above in Windows, please let us know.

Another alternative is to write target-runner in a language that you feel comfortable with and that can be executed in Windows from the command-line. Again, this is trivial in Linux but I'm not sure how to do it in Windows.

Yet another alternative is to edit ACOTSP code to change the output so that it only prints one number. Then, in the BAT file above, you can just print the contents of stdout as the COST.

I read in The irace Package: User Guide document that "the program targetRunner must print a real number,
which corresponds to the cost measure of the candidate configuration for the given instance
and optionally its execution time (mandatory when maxTime is used)". Then I suppose that the executable program (AcoTsp) must output ONLY a real number. But I noted ACOTSP program output more than a real number. Perhaps this error is happened because last sentences of target-runner.bat cannot get properly the output of acotsp executable file.

The reasons for having a target-runner are:

1) To translate from the command-line parameters that irace uses to evaluate a configuration to the command-line parameters used by your target-algorithm (ACOTSP).
2) To translate from the output of your target-algorithm to the output expected by irace.

Hence, it is OK for ACOTSP to print whatever it wants, but the target-runner must parse this output, extract the actual solution cost and print it. If your target-algorithm is written in a way that you do not need those translation steps, you case set targetRunner as your target-algorithm, but usually this is not the case and modifying the target-algorithm is too cumbersome.

But you are right, the error happens because target-runner.bat cannot properly parse the output of acotsp. You need to investigate why. You don't need to run irace to investigate this, you can directly invoke target-runner using the command printed by irace when showing the error.

Or you can use Linux where all this is so much easier.

Cheers,

Manuel.

[Leslie Perez Caceres]

As Manuel explains we do not normally execute irace with bat files, so we can't help you a lot with BATCH programming. 

The parsing should be working, because it was tested for ACOTSP, I think the problem is that in the execution line the FIXED_PARAMS variable should not be in caps (because is not defined in caps). It seems (from the output provided) that ACOTSP it is executing several trials and they should be set as 1. If thats the case then the parsing will not work. 

Before trying something else, please change the execution line in your bat file to:

%exe% %fixed_params% -i %instance% --seed %seed% %candidate_parameters% 1>%stdout% 2>%stderr%

The variable fixed_params should not be in caps, this is making ACOTSP to execute several repetitions and the parsing might not be working because the output is not as expected. This would make the execution fail.

Cheers, 

Leslie.

[souza.a...@gmail.com]

Hi Bernardo,

This happens ("ECHO est desactivado (comment: desactivado means Deactivated)... This is not a bug in irace, but means that something failed when running the commands...") because the target-runner.bat isn't fine at all (version inside of acotsp-example.tar.gz file, downloaded from http://iridia.ulb.ac.be/~lperez/matheuristics2016-irace/).

I did some modifications in target-runner.bat and all the things worked very well, i.e., the execution of instructions in R "> checkIraceScenario(scenario=scenario)" and "> irace(scenario = scenario, parameters = parameters)".

Basically, two parts of the script were modified:

1. I replaced this part:

SET candidate=%1

SHIFT

SET instance_id=%1

SHIFT

SET seed=%1

SHIFT

SET instance=%1

SHIFT

SET candidate_parameters=%*

by this:

FOR /f "tokens=1-4*" %%a IN ("%*") DO (

SET candidate=%%a

SET instance_id=%%b

SET seed=%%c

SET instance=%%d

SET candidate_parameters=%%e

)

2. I also replaced this part:

%exe% %FIXED_PARAMS% -i %instance% --seed %seed% %candidate_parameters% 1>%stdout% 2>%stderr%

By this:

%exe% %fixed_params% -i %instance% --seed %seed% %candidate_parameters% 1>%stdout% 2>%stderr%

In 1., the reason is that "%*" will always expand to all original parameters, sadly.

In 2., although variable names are not case-sensitive in Windows batch script, I decided, for the reason of coherence, keep the same spelling for the same variable ("fixed_params" appears in lower case at the beginning of the script).

I can send you the target-runner file if you want (I think I cannot attach it here in this group).

Best wishes,

Andre

[Marwa Daagi]

Hi Irace Team,

I am trying to execute a runnable jar file in Irace to tune NSGA-II parameters. I am running in Windows and I am using the new version of the targetRunner.bat.

After running  the check command checkIraceScenario(scenario=scenario), the following messages appeared in the console:

# 2022-05-19 15:22:45 CEST: Checking target execution.
# Executing targetRunner ( 2 times)...
# 2022-05-19 15:22:45 CEST: C:/Users/Prince/Documents/R/win-library/4.0/irace/tuning/target-runner.bat testConfig1 7 1234567 C:/Users/Prince/Documents/R/win-library/4.0/irace/tuning/Instances/f4_test.csv   100 100 0.3058 0.8407
# 2022-05-19 15:22:46 CEST: DONE (testConfig1) Elapsed wall-clock seconds: 0.81
Commande ECHO d‚sactiv‚e.

# Error occurred while executing targetRunner:
== irace == The output of targetRunner should not be more than two numbers!
== irace == The call to targetRunner was:
C:/Users/Prince/Documents/R/win-library/4.0/irace/tuning/target-runner.bat testConfig1 7 1234567 C:/Users/Prince/Documents/R/win-library/4.0/irace/tuning/Instances/f4_test.csv   100 100 0.3058 0.8407
== irace == The output was:
Commande ECHO d‚sactiv‚e.
== irace == This is not a bug in irace, but means that something failed when running the command(s) above or they were terminated before completion. Try to run the command(s) above from the execution directory 'C:/Users/Prince/Documents/R/win-library/4.0/irace/tuning/exec-dir' to investigate the issue. See also Appendix B (targetRunner troubleshooting checklist) of the User Guide (https://cran.r-project.org/package=irace/vignettes/irace-package.pdf).
# targetRunner returned:
NULL
Error: == irace == Check unsuccessful.

I have a doubt about the Instances defined in the scenario. In fact, my jar requires a dataset as  shown in the figure (10-fold-cross validation).

 So,  I put the .csv files in a folder called "Instances/". Then, I create a file "train-instances.txt" with the following contents:

##########

f1_test.csv

f1_train.csv

f2_test.csv

f2_train.csv

...

##########

This is the targetRunner.bat:

Perhaps this error is happened because target-runner.bat cannot get properly the output of my jar file which is a csv file that contains values of precision, recall, F1... computed from the solutions.

Thank you for taking the time to read my message.

Best,

Marwa

[Manuel López-Ibáñez]

Hi Marwa,

According to this (https://stackoverflow.com/questions/14334850/why-this-code-says-echo-is-off), it gives that error because %COST% is empty. Could you try with?

echo  "%COST% " "

Also, what are the contents of the XXX.stdout and XXX.stderr files generated?

It is difficult for me to help because I don't use Windows.

Since you seem to be proficient in Java, why not try to write your target-runner directly as a .jar file?

Create a target-runner.jar file and in scenario.txt set:

targetRunner = "./target-runner.jar"

Your 'target-runner.jar' file will need to take the command-line arguments in a similar way as your MOEA_v7_outcome but in the format provided by irace, that is,

target-runner.jar CANDIDATE_ID INSTANCE_ID SEED INSTANCE_NAME --param1 param1_value ...

and print ONLY one number, the cost that irace should minimize.

Then, you don't need target-runner.bat and you can directly call from target-runner.jar the functions in Java that you need to run your MOEA.

I hope the above helps!

Manuel.

[Manuel López-Ibáñez]

Hi,

I forgot to answer this question:

On Thursday, 19 May 2022 at 16:07:19 UTC+2 marwa...@gmail.com wrote:

I have a doubt about the Instances defined in the scenario. In fact, my jar requires a dataset as  shown in the figure (10-fold-cross validation).

 So,  I put the .csv files in a folder called "Instances/". Then, I create a file "train-instances.txt" with the following contents:

##########

f1_test.csv

f1_train.csv

f2_test.csv

f2_train.csv

...

##########

I don't think the above is correct.

Does one run of MOEA_v7_outcome.jar use all .CSV files? Or just a single pair fN_train.csv and fN_test.csv?

Imagine you were NOT using irace, how would you decide using a statistical test that one parameter setting is better than a different parameter setting? Will you run on different datasets?

Best wishes,

Manuel.

[Marwa Daagi]

Hi Manuel,

Thank you for your swift reply.

In fact, I think I will move to run with Ubuntu. I started the installation since running on linux is more stable than Windows according to the irace group.

For the dataset, yes my jar iterated over the 10 subsamples and then for each pair fx-train.csv fx-test.csv, the program computed précision, recall,... of the pareto front solutions. Finally, we take the median values of all the results.

However, I can use only one pair for exemple F1_train.csv, F1_test.csv if  irace requires this. If it is the case, could you please explain where should I put the pair (together or seperatly).

One more question about the output of my jar which is a csv file has impact on the output of the target-Runner?

Thank you very much!

Best,

Marwa

--
You received this message because you are subscribed to the Google Groups "The irace package: Iterated Racing for Automatic Configuration" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irace-packag...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/irace-package/90962c04-ac96-4526-bbae-fa61f6b49f47n%40googlegroups.com.

[Manuel López-Ibáñez]

Dear Marwa,

Even in Ubuntu, if you are more familiar with Java, you may find easier to write your target-runner using Java than using BASH shell language.

For deciding how to setup irace, it is good to start first answering the question without considering irace at all. If you compare just two parameter configurations of your algorithm, how would you perform a statistical test to decide which one is better? What experiments would you run, what data will you collect and how you will perform the test? One you know the answers to those questions (which are independent of irace), then you can setup irace accordingly to that answer so it will compare many configurations automatically in the way that you wish them to be compared. The answer depends a lot on your own scenario and goals that is why irace offers some much flexibility.

The target-runner should only print 1 value per run. You would need to get one value from your algorithm. If your algorithm returns multiple metrics, you need to aggregate them somehow, perhaps with a weighted sum. If your algorithm returns a Pareto front, you may need to compute a single value such as the hypervolume of the Pareto front or some other unary quality metric.

I hope the above helps!

Manuel.