Problem in t-test

Tarek Gamal

unread,

May 25, 2020, 12:38:56 AM5/25/20

to The irace package: Iterated Racing for Automatic Configuration

when i change F-test to t-test there is an error is happend but then continue running. did i need to back statistical test to F-test?? (i used t-test since i think it suitable for heterogeneous scenario)

Manuel López-Ibáñez

unread,

May 25, 2020, 4:13:05 AM5/25/20

to The irace package: Iterated Racing for Automatic Configuration

Hi Tarek,

The error means that the values compared are almost constant so the t-test in R complains. However, irace is able to handle that case and the error is ignored:

    # t.test may fail if the data in each group is almost constant. Hence, we
    # surround the call in a try() and we initialize p with 1 if the means are
    # equal or zero if they are different
    if (min(var(results_best), var(results_j)) < 10 * .Machine$double.eps) next

But I did a few experiments and I cannot even get the error to show up. Would it be possible to send us the output of irace when you run with --debug-level 3? This will show us what numbers your program is returning to irace.

The choice between t-test and F-test is not about homogeneous or heterogeneous scenario, it is about how you will compare two different configurations: are you comparing the mean performance or are you comparing based on "wins" or "ranks"? Or in other words, do you care about outliers or not?

Also, what version of irace and what version of R is this?

Cheers,

Manuel.

Tarek Gamal

unread,

May 25, 2020, 1:10:37 PM5/25/20

to The irace package: Iterated Racing for Automatic Configuration

https://drive.google.com/drive/folders/10g66tv4sqN4tdI-VEv01iNdJuBzDeOcA

the output saved in "error.txt"

irace version: 3.4.1

R version: 4.0.0

Reply all

Reply to author

Forward