Is it possible to have t statistics = 0 and p-value = 1 in 5*2 cv paired t-test?

[Jongyun Jung]

Hi,

First of all, thank you for sharing the module in Python. 

I tried to implement the comparison of two machine learning algorithm (Linear regression vs. Random forest) by using 5*2 cv paired t-test. However, I got the result of t statistics  = 0 and p-value = 1 in my code. 

When I implement these two algorithm, the value for MSE is different with 100 epochs. I'm not sure why I have the result of t statistics  = 0 and p-value = 1. 

Here is my code. 

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LinearRegression

from sklearn.ensemble import RandomForestRegressor

from sklearn.ensemble import GradientBoostingRegressor

import pickle

from mlxtend.evaluate import paired_ttest_5x2cv

with open('datamrosbmd1103_B1FND', 'rb') as file_handler:

    data = pickle.load(file_handler)

    X, Y = data.get('X', []).values, data.get('Y', []).values

linear = LinearRegression()

rf = RandomForestRegressor()

gb = GradientBoostingRegressor()

t, p = paired_ttest_5x2cv(estimator1=linear,

                          estimator2=rf,

                          X=X, y=Y, 

                          random_seed=25)

print("t statistic: %.5f" % t)

print("p avlue: %.5f" % p)

After I ran these code, I got t statistics  = 0 and p-value = 1. Could you help on this why I have such p-value = 0 ?

Thank you.

[Sebastian Raschka]

Hi there,

In general, it is certainly not impossible to get such values. It can be interpreted that there is not much evidence against the null hypothesis that the models perform equally well. Getting such values with linear regression vs random forest (a linear vs non-linear model) is certainly not unusual though. Do you get high R^2 values when you fit a linear regression and a random forest model on this dataset?

Best,

Sebastian

--
You received this message because you are subscribed to the Google Groups "mlxtend" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlxtend+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mlxtend/069b1873-fe74-4548-b0d3-9ff10033ae7c%40googlegroups.com.