Stepwise selection criteria

[Richard Schuster]

Hi all,

A little while ago I wrote a stepwise selection procedure using the
occu function, were I am using only AIC as a selection criteria. With
this I am running into trouble sometimes as models with low AIC but
huge (>500) beta's and SE's for some variables are top ranking ones.
As I am pretty certain that this is wrong I would like to incorporate
other/additional selection criteria into the selection procedure, but
am not certain which ones to use.

What I have seen used is doing an anova test. As the anova function
does not work with the "unmarkedFitOccu" class I wanted to ask if you
think an anova would be useful in this case? And worthwhile to look
into creating an appropriate function to be used with the unmarked
class?

The second method I have seen is checking in the selection procedure
if the term that is added to the model is significant on the 0.05 or
0.1 level.

Now I was wondering if any of you have recommendations as to which of
these methods to use and also if you would suggest other methods to
use in the a stepwise selection procedure, so I don't end up with
nonsense models in my model list?

Thank you very much for your help,
Richard

[Jeffrey Royle]

hi Richard -- I'm curious as to why you have these huge betas and SEs.
This suggests that the optimizer is not finding a stable optimum, its
diverging, which could be a result of poor starting values, poor
scaling of the covariates, or a basic non-identifiability. I think you
need to understand whats going on there. That said, I'm curious as to
why those situations would appear to be better models. My intuition is
that if the optimizer does something stupid then it should get stuck
in a bad part of the parameter space and return a worse likelihood
(and hence AIC). It might be useful if you could send me your data
file/R script which shows an example of this problem.

Now for your question: I think AIC should be a fine model-selection
metric, in general. I forget which criteria are used in the existing
selection procedures in R, Mallows Cp is one of them, but there are
many more. I suggest looking into that (the MASS library might have
some general functions for doing model selection which you could work
off of).

I think if you add factors based on significant coefficients at the
0.05 level then such factors will also show at least 2 units
improvement in AIC so there is a general consistency in how those two
methods should work, at least for the marginal "step" in the model
selection procedure. However , if you are using small p-values to
select , simultaneously, a bunch of covariates then you could well
wind up with different models at the end.

No model selection method is generally better or worse than any other
(which is why there are so many).

regards
andy

[Richard Chandler]

My guess is that your large SEs result from being close to a parameter
boundary when using the logit link. If so, the model could still be
good, and hence the lower AIC. Try using predict to see what your
back-transformed estimates of psi and p are. You might also want to
standardize your covariates.

In the new version of unmarked, I added a crude function for
likelihood-ratio tests. It has not been documented yet, and it is up
to the user to ensure that the models are nested. Try this:

example(modSel)
LRT(fm1, fm2)

You should probably think about adjusting your p-values if you are
going to use this approach for model selection.

Richard

[Richard Schuster]

Thank you Andy and Richard for your replies.

I have sent Andy my data and R scripts to illustrate the problem with
an example.

Richard:All my covariates are standardized already. In some cases the
huge SE's seem to result from two strongly opposing or correlated
covariates (e.g. Northing and Northing^2).
Do you anticipate that you will change/update the LRT soon, so I might
want to wait for further changes?

Right now I am thinking of including a test of significance for any
added term and this way getting rid of the huge estimates and betas.

Thanks,
Richard


On Sep 27, 11:14 am, Richard Chandler <richard.chandl...@gmail.com>
wrote:

> My guess is that your large SEs result from being close to a parameter
> boundary when using the logit link. If so, the model could still be
> good, and hence the lower AIC. Try using predict to see what your
> back-transformed estimates of psi and p are. You might also want to
> standardize your covariates.
>
> In the new version of unmarked, I added a crude function for
> likelihood-ratio tests. It has not been documented yet, and it is up
> to the user to ensure that the models are nested. Try this:
>
> example(modSel)
> LRT(fm1, fm2)
>
> You should probably think about adjusting your p-values if you are
> going to use this approach for model selection.
>
> Richard
>

[Richard Chandler]

I doubt that I will change LRT too much, unless people want to see it
implemented differently.

Richard

[Richard Schuster]

I have another quick question regarding this issue: How are the
significance values for the estimates calculated? I looked through the
unmarked code but was not able to locate this so far. As I would like
to use the significance criteria in the model selection I need to
locate this though.

Thanks again,
Richard

On Sep 28, 5:06 am, Richard Chandler <richard.chandl...@gmail.com>
wrote:

> I doubt that I will change LRT too much, unless people want to see it
> implemented differently.
>
> Richard
>
> On Mon, Sep 27, 2010 at 5:38 PM, Richard Schuster
>

> <ric.schus...@gmail.com> wrote:
> > Thank you Andy and Richard for your replies.
>
> > I have sent Andy my data and R scripts to illustrate the problem with
> > an example.
>
> > Richard:All my covariates are standardized already. In some cases the
> > huge SE's seem to result from two strongly opposing or correlated
> > covariates (e.g. Northing and Northing^2).
> > Do you anticipate that you will change/update the LRT soon, so I might
> > want to wait for further changes?
>

> > Right now I am thinking of including a test ofsignificancefor any

[Richard Schuster]

I figured it out with calculating the z-value from the estimate and SE
(or covMat).

Richard