Comparing regression models with same DV and different IVs
JohnDC
difference in R-squareds obtained from 2 regression equations, in the
same sample, and with the same dependent variable, but with different
sets of predictor variables. I would also like to know the answer to
the similar question with logistic regression. That is, how to obtain
the statistical significance of the differences in predictive ability
for two models, with the same binary dependend variables, but with a
different set of predictors. Help much appreciated.
Ray Koopman
Kevin A. Clarke. A Simple Distribution-Free Test for Nonnested
Model Selection. Political Analysis 15:3 (Summer 2007), 347-363.
http://www.rochester.edu/college/psc/clarke/SDFT.pdf
http://www.rochester.edu/college/psc/clarke/SDFTsupplement.pdf
Bruce Weaver
The same question has been posted to the SPSSX-L mailing list. For
the benefit of those who don't subscribe, you can see the thread here:
--
Bruce Weaver
bwe...@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/Home
"When all else fails, RTFM."
Rich Ulrich
wrote:
Ray,
What ever happened to AIC and BIC (in brief)?
--
Rich Ulrich
Ray Koopman
AIC is a sample statistic, that estimates a population parameter.
I know of no estimate of its standard error, let alone the s.e.
of the difference between two correlated AIC's.
For additional discussion see
D.R. Anderson & K.P. Burnham
AIC Myths and Misunderstandings,
http://warnercnr.colostate.edu/~anderson/PDF_files/AIC%20Myths%20and%20Misunderstandings.pdf
Burnham, K.P., & Anderson, D.R. (2004). Multimodel inference:
understanding AIC and BIC in model selection.
Sociological Methods & Research, 33, 261-304.
Anderson's home page http://warnercnr.colostate.edu/~anderson/
has links to other papers on comparing models.
Ray Koopman
> AIC is a sample statistic, that estimates a population parameter.
Make that "AIC/n is a sample statistic ...".
AIC grows with n and does not estimate any fixed quantity.
Ray Koopman
It depends on exactly what you mean by "model". Do you mean the
particular regression equation that you got from your current data,
or do you mean only the general form of the equation and the
particular set of predictor variables you're using? (If it's the
latter then you're talking about the regression equation that you
would get if you had population data, instead of just sample data.)
And, whichever of those it may be, you need to be specific about how
you want to measure "predictive ability". For ordinary least-squares
regression the two most obvious measures are the rms error of
prediction and the correlation between the actual and predicted
values of the d.v. For logistic regression the question is similar,
but you're comparing true and predicted probabilities, and the most
obvious measure is Kullback-Leibler divergence.
The complicating factor in all this is that, unless you're willing
to assume that the joint distribution of the d.v. and all the i.v.s
is multivariate normal, you can not assume that either model is
using the right set of variables or the correct form of the
regression function. (The multivariate normal case is simpler. If
"predictive ability" means "correlation" then Steiger's WBCORR may
be able to handle it, but I don't see how to tell the program what
to do.)