empirical reliability equation

[Steven Kalinowski]

Hello Phil,

I have been trying to understand how empirical reliability,rxx, is calculated in mirt. From what I can tell, mirt calculates rxx as

Eq. 1          rxx = s / (s+e)

using the notation in the blog: http://stats.stackexchange.com/questions/132738/reliability-in-irt-style

s = observed variance in abilities

e = average sampling variance of individuals 

I can't find a reference for Equation 1, and I do not understand its logic. Lord (1983, Eq. 51) seems to calculate rxx as (s-e) / s... which seems to follow the approach used in classical test theory. 

Samajima (1994) seems to use the same equation as Lord. So does Raju (2006).

I am not an expert on IRT and had to struggle to read Lord's paper's, so please forgive me if I am missing something. I am looking for a single statistic to summarize the reliability of instruments with.

Thank you for your help... and for the mirt package. I have found it very useful in my research.

Sincerely,

Steven Kalinowski

Montana State University

[Phil Chalmers]

On Tue, Dec 20, 2016 at 5:38 PM, Steven Kalinowski <steventk...@gmail.com> wrote:

Hello Phil,

I have been trying to understand how empirical reliability,rxx, is calculated in mirt. From what I can tell, mirt calculates rxx as

Eq. 1          rxx = s / (s+e)

using the notation in the blog: http://stats.stackexchange.com/questions/132738/reliability-in-irt-style

s = observed variance in abilities

e = average sampling variance of individuals 

I can't find a reference for Equation 1, and I do not understand its logic. Lord (1983, Eq. 51) seems to calculate rxx as (s-e) / s... which seems to follow the approach used in classical test theory. 

I'm not sure I've ever seen that ratio as an expression of reliability. Typically it is expressed either as 

rxx = 1 - e/(s + e) = 1 - e / o 

or 

rxx = s / (s + e) = s / o

where o = the total variation of true scores + error. Rxx is a marginal conceptualisation of a signal to noise ratio, and therefore some kind of pooled SE estimate has to be used for e (in CTT, e is assumed to be constant for all individuals; in IRT, an average or 'marginal' value is used instead). The name marginal reliability has also appeared in the literature for these types of measures, though I'm not a fan as 'marginal' can be taken to be many things (e.g., integrating over a theoretical density instead of an empirical approximation). 

 

Samajima (1994) seems to use the same equation as Lord. So does Raju (2006).

I am not an expert on IRT and had to struggle to read Lord's paper's, so please forgive me if I am missing something. I am looking for a single statistic to summarize the reliability of instruments with.

I'm not sure what the definitive reference is on these measures, but you can find them within older IRT software such as TESTFACT and MULTILOG, and so locating the information in the SSI manuals may be helpful. Cheers.

Phil

 

Thank you for your help... and for the mirt package. I have found it very useful in my research.

Sincerely,

Steven Kalinowski

Montana State University

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.