Best local dependence measure

[Jeanette Müller]

Hi Phil,

I recently read "The Performance of Local Dependence Measures With Psychological Data" from Carrie R. Houts and Michael C. Edwards. http://apm.sagepub.com/content/37/7/541.short
They show that the G2 from Chen and Thissen is one of the best performing indexes.

I,m now asking me the following questions:
1) Is this Monte Carlo study really that convincing?
2) Is ist possible to conclude that their simulation for graded response models is also correct for Rasch Partial Credit Models?
3) If I calculate the model in - let´s say eRm and calculate the LD values in mirt is this correct?
4) I tried to cross check the values from an IRTPRO calculation and  mirt calculation, but they didn´t match - but I´m not sure wether I could "convince" IRTPRO to really do a Rasch PCM - maybe someone knows how to...

Hoping that this is not far beyond the scope of this group I would be very thankful for anybody "who could bring some light into my dark thoughts"...

Best

Jeanette

[Phil Chalmers]

Hi Jeanette,

Thanks for the reference, I'll give it a look sometime and let you
know what I think. Cheers.

Phil

> --
> You received this message because you are subscribed to the Google Groups
> "mirt-package" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mirt-package...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Jeanette Müller]

Hi Phil,

this would be very helpful!
Have you maybe an opinion about my third question "If I calculate the model in - let´s say eRm and calculate the LD values in mirt is this correct?"
I think this would have nothing to do with the paper?

Once again thank your for your time and patience!

Best

Jeanette

[Phil Chalmers]

mirt has the χ2 based residuals described by Chen and Thissen (1997),
but not the G2 stat, and are computed when calling the residuals()
function on a converged model. It's generally accepted that the χ2
statistics works as well or better than the G2, which is why I opted
to use that one (it's very simple to add an option for the G2 stat
though, if you are really *that* interested, but the χ2 should be just
as good).

Phil

[Phil Chalmers]

Hi Jeanette,

I quickly perused the paper you mentioned and here is my initial impression.

On Thu, Jan 9, 2014 at 1:19 PM, Jeanette Müller <petepa...@gmail.com> wrote:

> Hi Phil,
>
> I recently read "The Performance of Local Dependence Measures With
> Psychological Data" from Carrie R. Houts and Michael C. Edwards.
> http://apm.sagepub.com/content/37/7/541.short
> They show that the G2 from Chen and Thissen is one of the best performing
> indexes.
>
> I,m now asking me the following questions:
> 1) Is this Monte Carlo study really that convincing?

If you are interested in LD, then I suppose it's interesting. I'm more
of a nested model comparison sort of person, but these stats are
useful if you are from the 'inspect the residuals from a factor
analysis' camp of thought, and from a model diagnostic perpective.
Though I'm unsure why the authors didn't also investigate the X2
analoge statistic from Chen & Thissen

> 2) Is ist possible to conclude that their simulation for graded response
> models is also correct for Rasch Partial Credit Models?

Doubtful, the partial credit model is a highly constrained model
(slopes all equal to 1 and free intercepts + latent variance) compared
to the graded model (slopes and intercepts all freely estimated,
variance constrained to 1). The graded model has a much closer
relationship to the generalized partial credit model than the pcm.

> 3) If I calculate the model in - let´s say eRm and calculate the LD values
> in mirt is this correct?

Not sure if eRm calculates these, but residuals() does in mirt (and on
the dev, now there is an option for computing the G2 or X2 stats).

> 4) I tried to cross check the values from an IRTPRO calculation and mirt
> calculation, but they didn´t match - but I´m not sure wether I could
> "convince" IRTPRO to really do a Rasch PCM - maybe someone knows how to...

See my response above to set it up, but IRTPRO may output different
values as well. As far as I know I standardize the residuals in a
different way compared to IRTPRO, where it transforms the X2/G2 values
to a z-like statistic where mirt transforms to a correlation-like stat
(for further factor analysis inspecting).

>
> Hoping that this is not far beyond the scope of this group I would be very
> thankful for anybody "who could bring some light into my dark thoughts"...

Join the club, I'm sure we all have some pretty dark thoughts.....Cheers!

Phil

[Jeanette Müller]

Wow! This was fast and very helpful! So hopefully the last questions for today:
A value higher 3.84 below the diagonal or smaller 0.05 above the diagonal would be flagging that the itempair is LD?
And therefore the differences between the values from eRm and mirt are only cause of CML vs. MML it should be fine to calculate the thresholds in eRm and the χ2 in mirt, right?

Once again,
thank you very much!

Best Jeanette

[Phil Chalmers]

On Thu, Jan 9, 2014 at 6:53 PM, Jeanette Müller <petepa...@gmail.com> wrote:
> Okay, that is good to know. And a value higher 3.84 below the diagonal or

> smaller 0.05 above the diagonal would be flagging that the itempair is LD?

Only if the items you are comparing are both dichotomous since df =
min(ncols - 1, nrows - 1), otherwise the value can be much larger when
comparing polytomous items and still be non-significant.

> And therefore the differences between the values from eRm and mirt are only
> cause of CML vs. MML it should be fine to calculate the thresholds in eRm
> and the χ2 in mirt, right?

I'm not sure what you mean here. Both packages could theoretically
calculate these stats since you really only need the probability trace
line functions to be jointly integrated and compared to the observed
table of response combinations; I'm just not sure if eRm currently
supports these statistics natively. Hope that helps.

Phil

>
> Am Freitag, 10. Januar 2014 00:13:12 UTC+1 schrieb Phil Chalmers:
>>

[Jeanette Müller]

Thank you for your answers!

Okay, now I´m lost a little. Is there a value for χ2 or G2 in mirt where we would assume local dependence?

> And therefore the differences between the values from eRm and mirt are only
> cause of CML vs. MML it should be fine to calculate the thresholds in eRm
> and the χ2 in mirt, right?
I'm not sure what you mean here. Both packages could theoretically
calculate these stats since you really only need the probability trace
line functions to be jointly integrated and compared to the observed
table of response combinations; I'm just not sure if eRm currently
supports these statistics natively. Hope that helps.

Sorry for my "not that good english" ;-)
What I meant was the following. As far as I know eRm doesn´t currently
supports these statistics... but has some other abilities I want to use. But to check the local dependence I could use mirt.
My question was wether it would be okay to calculate all values I need in eRm, but to use mirt to "prove" that with local dependence everything is fine...
I hope now I got it right.

[Phil Chalmers]

On Fri, Jan 10, 2014 at 9:19 AM, Jeanette Müller <petepa...@gmail.com> wrote:
> Thank you for your answers!
>
> Okay, now I´m lost a little. Is there a value for χ2 or G2 in mirt where we
> would assume local dependence?
>

It depends on what the degrees of freedom are, which is why mirt and
other software like IRTPRO offer standardized values to help with
interpretation. I was just noting that the 3.84 value is only
applicable when df = 1 (the 2 item dichotomous pairing), and can be
different if there are more categories. In the mirt standardization, I
would start to become worried if several values were higher than r=.1
in the upper triangle portion of the matrix.

>
>> And therefore the differences between the values from eRm and mirt are
>> only
>> cause of CML vs. MML it should be fine to calculate the thresholds in eRm
>> and the χ2 in mirt, right?
> I'm not sure what you mean here. Both packages could theoretically
> calculate these stats since you really only need the probability trace
> line functions to be jointly integrated and compared to the observed
> table of response combinations; I'm just not sure if eRm currently
> supports these statistics natively. Hope that helps.
>
> Sorry for my "not that good english" ;-)
> What I meant was the following. As far as I know eRm doesn´t currently
> supports these statistics... but has some other abilities I want to use. But
> to check the local dependence I could use mirt.
> My question was wether it would be okay to calculate all values I need in
> eRm, but to use mirt to "prove" that with local dependence everything is
> fine...
> I hope now I got it right.

I see what you are saying now, thank you for clarifying. Yes this is
possible, but you'll need to find a good way to transform how eRm
parametrizes the models to something mirt can understand, and pass
these values as fixed parameters so that they don't change when the
model converges. This post should show you the general idea for
setting this up:
http://stats.stackexchange.com/questions/34119/estimating-ability-using-irt-when-the-model-parameters-are-known/48105#48105.

Alternately, I would just estimate the same models in mirt using MML
instead of CML estimation from eRm, and do the tests that way since
it's a bit easier. The two algorithms will give very similar results
(since Rasch models are extremely simple to estimate models, there
isn't much room for deviation), so the LD stats will invariably be
close as well.

[Jeanette Müller]

Hi Phil,

I really appreciate your help!
So the standardized values (best below 0.1) are - regardless χ2 or G2 - the important values? Do you have an article that I could cite for this?

Regarding the mirt and eRm combination: I once again wrote it in a misunderstandable way...
What I meant was the second solution you propose:
Estimate the model in eRm and get all the parameters I need and to prove that everything is fine regarding local dependence estimate the same model in mirt and use the standardized values...

Thank you really a lot!

Best Jeanette

[Phil Chalmers]

On Jan 10, 2014 11:08 AM, "Jeanette Müller" <petepa...@gmail.com> wrote:
>
> Hi Phil,
>
> I really appreciate your help!
> So the standardized values (best below 0.1) are - regardless χ2 or G2 - the important values? Do you have an article that I could cite for this?

I do not, but that advice does come from my experiences, and similar cutoffs have been proposed in the linear factor analysis literature when inspecting standardized residuals. I'm not a significance test sort of person, but if you are then you could take route as well (see the mirt-residuals documentation for more details).

Best of luck with your analysis! Cheers.

Phil