Model Selection and AIC, or BIC
danbomin...@sina.com
Dear All,
Any help will be appreciated greatly. Everybody knows that we can use AIC or BIC to do model selection.
The smaller AIC or BIC the better, by SAS. Can we use AIC or BIC to compare two models, before and after log transformation? If yes, then how? If no, then what else? Barry
Martin
If you look ay your posting below, it's come out a bit of a mess. I
don't know why, but I am viewing it from the website. Could you have
another go ? Please start with a new thread, and then I'll delete tis
one.
bw,
Martin
On May 30, 2:49 am, danbomingzhi1...@sina.com wrote:
> Dear All,
>
> Any help will be appreciated greatly. Everybody knows that we can use AIC or BIC to do model selection.
> The smaller AIC or BIC the better, by SAS. Can we use AIC or BIC to compare two models, before and after log transformation? If yes, then how? If no, then what else? Barry
Peter Flom
Barry asked
Dear All,
Any help will be appreciated greatly. Everybody knows that we can use AIC or BIC to do model selection.
The smaller AIC or BIC the better, by SAS. Can we use AIC or BIC to compare two models, before and after log transformation? If yes, then how? If no, then what else? Barry
I don’t think this is an appropriate use of AIC or BIC, I believe they are intended for nested models, and models with transformed variables are not nested. But, if the models before and after transformation have the same number of terms, then the penalty part of AIC or BIC is irrelevant. If you’ve transformed only the IVs, then you could look at R-squared (if it’s an OLS type model) or one of the pseudo R measures for logistic.
HTH
Peter
SR Millis
| Peter, Actually, you can use BIC to compare non-nested model as well as nested. See Generalized Linear Models, 2nd ed by Hardin and Hilbe (2007). Scott ~~~~~~~~~~~ Scott R Millis, PhD, ABPP, CStat, CSci Professor & Director of Research Dept of Physical Medicine & Rehabilitation Dept of Emergency Medicine Wayne State University School of Medicine 261 Mack Blvd Detroit, MI 48201 Email: aa3...@wayne.edu Email: srmi...@yahoo.com Tel: 313-993-8085 Fax: 313-966-7682 --- On Sun, 5/30/10, Peter Flom <peterflom...@mindspring.com> wrote: |
|
Frank Harrell
to compare models from different families or with different Y-
transformations. Hence you can't compare a model predicting Y with a
model predicting log(Y). Two other points:
- For most purposes BIC is terribly conservative. BIC assumes that
there exists a "real" model and that it is of finite dimension. AIC
is more natural in not assuming that "the" model exists, knowing that
we can fit more complex models reliably as N increases.
- AIC and BIC (especially AIC) were developed to compare no more than
two pre-specified models.
Frank
On May 30, 7:55 am, SR Millis <srmil...@yahoo.com> wrote:
> Peter,
>
> Actually, you can use BIC to compare non-nested model as well as nested. See Generalized Linear Models, 2nd ed by Hardin and Hilbe (2007).
>
> Scott
>
> ~~~~~~~~~~~
>
> Scott R Millis, PhD, ABPP, CStat, CSci
>
> Professor & Director of Research
>
> Dept of Physical Medicine & Rehabilitation
>
> Dept of Emergency Medicine
>
> Wayne State University School of Medicine
>
> 261 Mack Blvd
>
> Detroit, MI 48201
>
> Email: aa3...@wayne.edu
>
>
> Tel: 313-993-8085
>
> Fax: 313-966-7682
>
> --- On Sun, 5/30/10, Peter Flom <peterflomconsult...@mindspring.com> wrote:
SR Millis
What do you think of Raftery's guidelines for interpreting the BIC difference between 2 models? I'm not aware of any similar guidelines for interpreting AIC differences. Are you?
Thanks,
Scott Millis
--- On Sun, 5/30/10, Frank Harrell <f.ha...@vanderbilt.edu> wrote:
Frank Harrell
I haven't looked at his guidelines. In one paper of Raftery, he
selected a single model from a huge number of competing models on the
basis of BIC. I think that results in overfitting.
Frank
On May 31, 11:22 am, SR Millis <srmil...@yahoo.com> wrote:
> Thanks, Frank, for the clarification.
>
> What do you think of Raftery's guidelines for interpreting the BIC difference between 2 models? I'm not aware of any similar guidelines for interpreting AIC differences. Are you?
>
> Thanks,
> Scott Millis
>
SR Millis
Yes, there is no statistical free lunch when it comes to model selection and overfitting.
Raftery (1995)has proposed that differences in BIC of 2 or more provide evidence favoring one model over another; 6 or more provide strong evidence; and 10 is taken to be very strong evidence for model improvement.
Scott
--- On Mon, 5/31/10, Frank Harrell <f.ha...@vanderbilt.edu> wrote:
John Sorkin
arbitrary, or do they have some basis? Given the difficulty to defining
a p value associated with a difference between two model's AIC, BIC, or
AICC in the general case, I would not be surprised if the values were
not arbitrarily selected. We need to be careful about accepting critical
values without a critical analysis of the basis for the values.
John
John David Sorkin M.D., Ph.D.
Chief, Biostatistics and Informatics
University of Maryland School of Medicine Division of Gerontology
Baltimore VA Medical Center
10 North Greene Street
GRECC (BT/18/GR)
Baltimore, MD 21201-1524
(Phone) 410-605-7119
(Fax) 410-605-7913 (Please call phone number above prior to faxing)>>>
SR Millis <srmi...@yahoo.com> 5/31/2010 1:52 PM >>>
Hi, Frank,
Scott
Confidentiality Statement:
This email message, including any attachments, is for the sole use of
the intended recipient(s) and may contain confidential and privileged
information. Any unauthorized use, disclosure or distribution is
prohibited. If you are not the intended recipient, please contact the
sender by reply email and destroy all copies of the original message.
SR Millis
http://www.stat.washington.edu/raftery/Research/PDF/socmeth1995.pdf
Scott Millis
--- On Mon, 5/31/10, John Sorkin <jso...@grecc.umaryland.edu> wrote:
danbomin...@sina.com
Dear Scott,
I got the paper. Thank you very much! Great paper! Barry
SR Millis <srmi...@yahoo.com>
meds...@googlegroups.com
Re: {MEDSTATS} Model Selection and AIC, or BIC
2010-6-1 08:51:04
Juliet Hannah
I have found this topic confusing because of the diversity of opinion
among thoughtful
researchers. Has everyone seen this commentary?
Google: "AIC MYTHS AND MISUNDERSTANDINGS"
It is the first hit, and it is written by David Anderson and Kenneth
Burnham. I'm curious to hear everyone's thoughts.
Regards,
Juliet
Frank Harrell
compare only two models. To be honest I have not studied Akaike's
original papers as extensively as Anderson and Burnham have. But I
have made that statement publicly for years and this is the first time
someone has taken issue with it. My statement was in response to the
cavalier use of AIC to select a "best" model from among dozens of
competing models. This is just a form of stepwise variable selection
and results in overfitting unless some structure is placed on the
analysis (e.g. a pre-specified order for the variables to be added to
the model). I'd be interested in other comments on these points.
Frank
On May 31, 10:05 pm, Juliet Hannah <juliet.han...@gmail.com> wrote:
> Hi Group,
>
> I have found this topic confusing because of the diversity of opinion
> among thoughtful
> researchers. Has everyone seen this commentary?
>
> Google: "AIC MYTHS AND MISUNDERSTANDINGS"
>
> It is the first hit, and it is written by David Anderson and Kenneth
> Burnham. I'm curious to hear everyone's thoughts.
>
> Regards,
>
> Juliet
>
>
>
> On Mon, May 31, 2010 at 10:07 PM, <danbomingzhi1...@sina.com> wrote:
> > Dear Scott,
>
> > I got the paper. Thank you very much! Great paper! Barry
>
> > Re: {MEDSTATS} Model Selection and AIC, or BIC
> > 2010-6-1 08:51:04
>
> > Here's a copy of Raftery's paper---see if you agree with his BIC guidelines:
>
> >http://www.stat.washington.edu/raftery/Research/PDF/socmeth1995.pdf
>
> > Scott Millis
>
> >> Hi, Frank,
>
> >> Yes, there is no statistical free lunch when it comes to
> >> model
> >> selection and overfitting.
>
> >> Raftery (1995)has proposed that differences in BIC of 2 or
> >> more provide
> >> evidence favoring one model over another; 6 or more provide
> >> strong
> >> evidence; and 10 is taken to be very strong evidence for
> >> model
> >> improvement.
>
> >> Scott
>
> >> This email message, including any attachments, is for the
> >> sole use of
> >> the intended recipient(s) and may contain confidential and
> >> privileged
> >> information. Any unauthorized use, disclosure or
> >> distribution is
> >> prohibited. If you are not the intended recipient,
> >> please contact the
> >> sender by reply email and destroy all copies of the
> >> original message.
>
> >> --
> >> To post a new thread to MedStats, send email to MedS...@googlegroups.com
> >> .
Peter Flom
<<<<
I noticed that they go after my statement that AIC was designed to
compare only two models. To be honest I have not studied Akaike's
original papers as extensively as Anderson and Burnham have. But I
have made that statement publicly for years and this is the first time
someone has taken issue with it. My statement was in response to the
cavalier use of AIC to select a "best" model from among dozens of
competing models. This is just a form of stepwise variable selection
and results in overfitting unless some structure is placed on the
analysis (e.g. a pre-specified order for the variables to be added to
the model). I'd be interested in other comments on these points.
>>>
While I object to the cavalier use of any approach, I don't think AIC to
select among a group of models is a bad method, provided those methods are
chosen with some sort of sense. Frank uses the phrase "some structure" and
perhaps I am adopting a much broader use of this phrase than he is. To me
"some structure" implies that some thought was put into the choice, not just
of available independent variables, but to how they might combine sensibly.
In addition, we should bear in mind that (in most cases, and certainly in
the vast majority where techniques like AIC are needed) there is no one
"best" model - one model may be best for a particular sample and another for
another sample (even if the samples are truly randomly selected from a
population). Yet, often, we need to go forward with one model.
These days, I usually use LASSO or LAR, but there are options in those
methods, and I am not aware that one choice of options is best.
Regards
Peter
Peter Flom PhD.
Peter Flom Consulting LLC
5 Penn Plaza, Ste 2342
NY NY 10001
www.statisticalanalysisconsulting.com
www.IAmLearningDisabled.com
Frank Harrell
A good test would be to use a moderately small sample size and
signal:noise ratio (true R^2) and to determine the number k such that
using AIC to select from among k unstructured tests has some good
probability P of selecting the model that validates best in a new
sample. My guess is that in many cases the number k will not exceed 3
but I haven't done the simulation.
Frank
On Jun 1, 6:34 am, "Peter Flom" <peterflomconsult...@mindspring.com>
wrote:
Peter Flom
<<<
Thanks for your note Peter.
A good test would be to use a moderately small sample size and
signal:noise ratio (true R^2) and to determine the number k such that
using AIC to select from among k unstructured tests has some good
probability P of selecting the model that validates best in a new
sample. My guess is that in many cases the number k will not exceed 3
but I haven't done the simulation.
>>>>
Hi Frank
I am not sure I agree completely. Suppose your initial pool of IVs has 50
variables (some, of course, have many more). Through various means, you
narrow it to 10 models. Let's say all contain various combinations of 15
variables. Let's also say that N is large enough that a 15 variable model
isn't silly, and that checks for collinearity are OK.
Then, rather than do the test you propose, I'd suggest ranking those 10
using AIC on one sample, and testing those rankings on another sample. If
the rankings are similar and the models chosen in first place are similar as
well, I'd say AIC is doing its job.
The data may simply not be good enough to yield a single model that works
best. Failure to repeatedly find it is, then, not a fault of the method,
but of the data.
Of course, this varies from field to field. But I work a lot in fields
where things are not measured that precisely.
I also do not value simplicity of model as highly as some people do; nor do
I value statistical sig. of each IV as highly as some do. Indeed, sometimes
finding a SMALL and nonsignificant effect is more interesting that finding a
large effect. This would be so, for instance, if previous studies had found
a large effect.
On my business card I have my motto:
Questions answered, answers questioned.
I also like the remark apparently said by David Cox
"There are no routine statistical questions, only questionable statistical
routines"
Interesting discussion!
Peter
Frank Harrell
worth doing. However many people try to pick a "winner" so that's
worth studying too. But the bigger point is that the "winner"
selected by AIC or BIC or whatever will have inflated coefficients.
Frank
On Jun 1, 2:28 pm, "Peter Flom" <peterflomconsult...@mindspring.com>
wrote:
Andrew L. Vitiello
I am hoping some of my more learned colleagues can help me with this problem.
I am looking at a data set collected over the past 2 years from local amateur football players from a number of different teams. The research question we asked was whether or not pre-season injury status (ie answering either yes or no) was influenced/or could be predicted by the following independent variables;
- Age
- Height
- Weight
- Team played for
- Position played
- Experience playing football (number of years)
- Pre-season expectation of performance
- Amount of pre-season training
- Type of pre-season training
- Whether or not the assigned player position (by coach) matched the player's favoured position
Should I add all these IV's in an exploratory model and discard those that do not affect the final DV probability result? Or should I run separate BLR's for each IV against the DV and based on that include only those that affect the DV result?
I apologise in advance if this may seem like a 'pre-schooler's' question.
Thank you for your help.
Andrew Vitiello PhD
--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .
SR Millis
Andrew,
Univariate screening of variables is to be avoided. It's simply a variant of stepwise variable selection---will all of its associated problems.
Theory and your hypotheses should guide selection of your variables. Then, consider discarding variables with narrow distributions or a lot of missing data. I'd also perform collinearity diagnostics on the covariates/variables using the condition indexes and their associated variance decomposition proportions to identity any redundancy among your covariates.
You''ll also need to taken sample size into consideration: in the logistic regression context, the smaller of the 2 groups determines the upper limit as to the number of variables you can include in group model. As a rough guideline, you'll need about 10 subjects per variables. Support group 1 has 100 subejcts and group 2 has 120. You probably don't want to have more than 10 variables in your model.
These issues are covered quite well in:
1. Harrell FE, Jr. Regression modeling strategies: With applications to linear models, logistic regression, and survival analysis. New York: Springer-Verlag; 2001.
2. Steyerberg E. Clinical prediction models: A practical approach to development, validation, and updating. New York: Springer; 2009.
Scott Millis |
Andrew L. Vitiello
Thank you so much for your speedy reply. I really appreciate the effort
Andrew Vitiello PhD