sample independence of IRT

[Balazs Klein]

I keep reading that IRT (at least the one parameter model) is sample independent.

I thought it means that even if I had a non-representative, biased sample the model would reflect the properties of the whole population.

So I will not need to worry about representative samples to the same extent as I had to wit CTT.

My experience however is that whenever I create a mirt model from a sample (no matter how skewed it is) the sample ability mean will always be very close to 0.

This suggests to me that if I oversampled clever people when creating the model I will still get a higher estimate for theta=0,

and if I used that model I would still underestimate the ability of the population just like it used to be with CTT.

I guess my bottomline question is: do I still need to have a representative sample if I used a simple Rasch or a 2PL model, or is an IRT model really sample independent?

If it is sample independent, what does that mean and how could I show that?

[Phil Chalmers]

Straight out of wikipedia: 

"The one-parameter model (1PL) assumes that guessing is a part of the ability and that all items that fit the model have equivalent discriminations, so that items are only described by a single parameter ( {\displaystyle b_{i}} b_{i}). This results in one-parameter models having the property of specific objectivity, meaning that the rank of the item difficulty is the same for all respondents independent of ability, and that the rank of the person ability is the same for items independently of difficulty. "

Of course, this is only true asymptotically. In practice you don't have the true intercepts parameters, so the rank-ordering of the individuals is only guaranteed to be true in larger samples (particularly when items with similar difficulty values exist, because they are likely to be exchanged due to sampling error). 

On Tue, Oct 18, 2016 at 9:40 AM, Balazs Klein <balazs...@gmail.com> wrote:

I keep reading that IRT (at least the one parameter model) is sample independent.

I thought it means that even if I had a non-representative, biased sample the model would reflect the properties of the whole population.

So I will not need to worry about representative samples to the same extent as I had to wit CTT.

Oh my no, that's never true. You ALWAYS need a representative sample for the population with which you are trying to make inferences to. Rasch/1PL are not magical, they are just models that need to be calibrated.

 

My experience however is that whenever I create a mirt model from a sample (no matter how skewed it is) the sample ability mean will always be very close to 0.

This suggests to me that if I oversampled clever people when creating the model I will still get a higher estimate for theta=0,

and if I used that model I would still underestimate the ability of the population just like it used to be with CTT.

I guess my bottomline question is: do I still need to have a representative sample if I used a simple Rasch or a 2PL model, or is an IRT model really sample independent?

If it is sample independent, what does that mean and how could I show that?

You could only show it as a didactic example, because sample data will never show this property (how could it? You only ever have estimates of the parameters, not the parameters themselves).

 

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Seongho Bae]

IRT isn't a magical theory. But Most powerful measurement theory ever. IRT can be manage "sample independent" after the first representative sampling using item linking if the item has not differential item functioning and item parameter drift. That means if you know true item parameter coefficient where gets representative sample, the ability posterior distribution may not follow N(0,1). Because you already know item parameter coefficients. Fixing mu=0 and std=1 is the useful strategy to solve parameter indeterminacy problem under the Marginal Maximum Likelihood estimate when you don't know true person ability and item parameters.

2016년 10월 18일 화요일 오후 10시 40분 17초 UTC+9, Balazs Klein 님의 말:

[Keith Lau]

Phil Chalmers於 2016年10月19日星期三 UTC+8上午11時08分28秒寫道：

Straight out of wikipedia: 

"The one-parameter model (1PL) assumes that guessing is a part of the ability and that all items that fit the model have equivalent discriminations, so that items are only described by a single parameter ( {\displaystyle b_{i}} b_{i}). This results in one-parameter models having the property of specific objectivity, meaning that the rank of the item difficulty is the same for all respondents independent of ability, and that the rank of the person ability is the same for items independently of difficulty. "

Of course, this is only true asymptotically. In practice you don't have the true intercepts parameters, so the rank-ordering of the individuals is only guaranteed to be true in larger samples (particularly when items with similar difficulty values exist, because they are likely to be exchanged due to sampling error). 

On Tue, Oct 18, 2016 at 9:40 AM, Balazs Klein <balazs...@gmail.com> wrote:

I keep reading that IRT (at least the one parameter model) is sample independent.

I thought it means that even if I had a non-representative, biased sample the model would reflect the properties of the whole population.

So I will not need to worry about representative samples to the same extent as I had to wit CTT.

Oh my no, that's never true. You ALWAYS need a representative sample for the population with which you are trying to make inferences to. Rasch/1PL are not magical, they are just models that need to be calibrated.

Using MML requires the assumption of person distribution, but not the case for conditional maximum likelihood method specifically for the Rasch model. See eRm package.

 

My experience however is that whenever I create a mirt model from a sample (no matter how skewed it is) the sample ability mean will always be very close to 0.

This suggests to me that if I oversampled clever people when creating the model I will still get a higher estimate for theta=0,

and if I used that model I would still underestimate the ability of the population just like it used to be with CTT.

I guess my bottomline question is: do I still need to have a representative sample if I used a simple Rasch or a 2PL model, or is an IRT model really sample independent?

If it is sample independent, what does that mean and how could I show that?

You could only show it as a didactic example, because sample data will never show this property (how could it? You only ever have estimates of the parameters, not the parameters themselves).

I would say "no model" can exactly be confirmed by limited sample data. You can, however, find out the "better" one among model candidates. Using item fit to check whether the Rasch model could approximate the data (generated by unknown model) good enough is a practical way. But criteria of "good enough" is somewhat subjective.
 

 

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package...@googlegroups.com.

[Seongho Bae]

Yes, CMLE using the Total raw score from delivered CTT assumption as person ability distribution. But that may draw inappropriate item parameter values in some situation (Neyman & Scott, 1948). So, the 'Divided and Conquer' approach use in current IRT models even the Rasch model.

Neyman, J. and E. L. Scott (1948), “Consistent estimates based on partially consistent observations,” Econometrica, 16:, 1-32.

Seongho

2016년 10월 19일 수요일 오후 7시 34분 49초 UTC+9, Keith Lau 님의 말:

[Balazs Klein]

Ok. So I way over-interpreted what sample independence means.

How can I adjust my model if I knew the true thetas of a subsample of my sample than?

(I can have cc.200 of my cc.10k test takers to also fill out a test that is sufficiently similar to mine - r>0.85 - and has a representative norm.)

If there is a mean difference of Xsd between the two results, can I just add that to the difficulty parameters?

[Seongho Bae]

See this: http://www.springer.com/us/book/9781493903160

Seongho

2016년 10월 19일 수요일 오후 7시 54분 51초 UTC+9, Balazs Klein 님의 말:

[Phil Chalmers]

On Wed, Oct 19, 2016 at 6:34 AM, Keith Lau <genw...@gmail.com> wrote:

Phil Chalmers於 2016年10月19日星期三 UTC+8上午11時08分28秒寫道：

Straight out of wikipedia: 

"The one-parameter model (1PL) assumes that guessing is a part of the ability and that all items that fit the model have equivalent discriminations, so that items are only described by a single parameter ( {\displaystyle b_{i}} b_{i}). This results in one-parameter models having the property of specific objectivity, meaning that the rank of the item difficulty is the same for all respondents independent of ability, and that the rank of the person ability is the same for items independently of difficulty. "

Of course, this is only true asymptotically. In practice you don't have the true intercepts parameters, so the rank-ordering of the individuals is only guaranteed to be true in larger samples (particularly when items with similar difficulty values exist, because they are likely to be exchanged due to sampling error). 

On Tue, Oct 18, 2016 at 9:40 AM, Balazs Klein <balazs...@gmail.com> wrote:

I keep reading that IRT (at least the one parameter model) is sample independent.

I thought it means that even if I had a non-representative, biased sample the model would reflect the properties of the whole population.

So I will not need to worry about representative samples to the same extent as I had to wit CTT.

Oh my no, that's never true. You ALWAYS need a representative sample for the population with which you are trying to make inferences to. Rasch/1PL are not magical, they are just models that need to be calibrated.

Using MML requires the assumption of person distribution, but not the case for conditional maximum likelihood method specifically for the Rasch model. See eRm package.

This is actually a common misinterpretation of CML, so I think I'll help clear it up here. CML is just a method to obtain ML estimates for item parameters through conditioning on a sufficient statistic supplied by the data. MML achieves the same goal, but specifies a latent distribution (i.e., a range of uncertainty) over which each individual could be located. However, when choosing the Gaussian distribution (and the model is just identified), maximal uncertainty about the location of \theta is defined, and MML parameter estimates become identical to CML. 

To convince yourself of this, consider the following simulation which compares the estimators with different test lengths, sample sizes, and latent variable distributions. 

#-------------------------------------------------------------------

library(SimDesign)

Design <- expand.grid(nitems = c(10, 20, 30),

 N = c(100, 250, 500),

 type = c('norm', 'chisq', 'bimodal'))

#-------------------------------------------------------------------

Generate <- function(condition, fixed_objects = NULL) {

Attach(condition)

while(TRUE){

a <- matrix(1, nitems)

d <- matrix(rnorm(nitems))

d <- d - mean(d)

if(type == 'norm'){

Theta <- matrix(scale(rnorm(N)))

} else if(type == 'chisq'){

Theta <- scale(rchisq(N, 3))

} else if(type == 'bimodal'){

Theta <- matrix(scale(c(rnorm(N/2, -1.5), rnorm(N/2, 1.5))))

}

dat <- simdata(a, d, N, itemtype = 'dich', Theta=Theta)

if(all(apply(dat, 2, function(x) length(unique(x))) == 2L)) break

}

dat

}

Analyse <- function(condition, dat, fixed_objects = NULL) {

emod <- eRm::RM(dat)

mmod <- mirt(dat, 1, 'Rasch', verbose=FALSE,

TOL = 1e-7, optimizer = 'nlminb') #subtract mean(d) to equate

ecfs <- coef(emod)

mcfs <- coef(mmod, simplify=TRUE, digits=Inf)$items[,'d']

mcfs <- mcfs - mean(mcfs)

data.frame(CML=ecfs, MML=mcfs)

}

Summarise <- function(condition, results, fixed_objects = NULL) {

    ret <- c(cor=mean(sapply(results, function(x) round(cor(x[,1], x[,2]), 5))),

             bias=mean(sapply(results, function(x) round(bias(x[,1], x[,2]), 5))),

             RMSE=mean(sapply(results, function(x) round(RMSE(x[,1], x[,2]), 5))))

    ret

}

#-------------------------------------------------------------------

results <- runSimulation(design=Design, replications=8, parallel = TRUE,

                         save_results=FALSE, packages=c('mirt', 'eRm'),

generate=Generate, analyse=Analyse, summarise=Summarise)

results

#-------------------------------------------------------------------

The results show that the estimates correlate .9999 and higher, have 0 bias between them, and are basically identical (RMSE typically less than .0001). If you pass save_results = TRUE then the estimates will be saved to your hard drive, so you can take a more specific look at a later time by reading in the files with readRDS().

Therefore, the estimators are given identical parameter estimates regardless of the underlying distribution of \theta. So really there is nothing special about CML, it's just ML estimation of item parameters (in this case, intercepts) using a simple trick. HTH.

Phil

 

 

My experience however is that whenever I create a mirt model from a sample (no matter how skewed it is) the sample ability mean will always be very close to 0.

This suggests to me that if I oversampled clever people when creating the model I will still get a higher estimate for theta=0,

and if I used that model I would still underestimate the ability of the population just like it used to be with CTT.

I guess my bottomline question is: do I still need to have a representative sample if I used a simple Rasch or a 2PL model, or is an IRT model really sample independent?

If it is sample independent, what does that mean and how could I show that?

You could only show it as a didactic example, because sample data will never show this property (how could it? You only ever have estimates of the parameters, not the parameters themselves).

I would say "no model" can exactly be confirmed by limited sample data. You can, however, find out the "better" one among model candidates. Using item fit to check whether the Rasch model could approximate the data (generated by unknown model) good enough is a practical way. But criteria of "good enough" is somewhat subjective. 

 

 

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package+unsubscribe@googlegroups.com.

[Keith Lau]

Phil Chalmers於 2016年10月19日星期三 UTC+8下午10時58分40秒寫道：

On Wed, Oct 19, 2016 at 6:34 AM, Keith Lau <genw...@gmail.com> wrote:

Phil Chalmers於 2016年10月19日星期三 UTC+8上午11時08分28秒寫道：

Straight out of wikipedia: 

"The one-parameter model (1PL) assumes that guessing is a part of the ability and that all items that fit the model have equivalent discriminations, so that items are only described by a single parameter ( {\displaystyle b_{i}} b_{i}). This results in one-parameter models having the property of specific objectivity, meaning that the rank of the item difficulty is the same for all respondents independent of ability, and that the rank of the person ability is the same for items independently of difficulty. "

Of course, this is only true asymptotically. In practice you don't have the true intercepts parameters, so the rank-ordering of the individuals is only guaranteed to be true in larger samples (particularly when items with similar difficulty values exist, because they are likely to be exchanged due to sampling error). 

On Tue, Oct 18, 2016 at 9:40 AM, Balazs Klein <balazs...@gmail.com> wrote:

I keep reading that IRT (at least the one parameter model) is sample independent.

I thought it means that even if I had a non-representative, biased sample the model would reflect the properties of the whole population.

So I will not need to worry about representative samples to the same extent as I had to wit CTT.

Oh my no, that's never true. You ALWAYS need a representative sample for the population with which you are trying to make inferences to. Rasch/1PL are not magical, they are just models that need to be calibrated.

Using MML requires the assumption of person distribution, but not the case for conditional maximum likelihood method specifically for the Rasch model. See eRm package.

This is actually a common misinterpretation of CML, so I think I'll help clear it up here. CML is just a method to obtain ML estimates for item parameters through conditioning on a sufficient statistic supplied by the data. MML achieves the same goal, but specifies a latent distribution (i.e., a range of uncertainty) over which each individual could be located. However, when choosing the Gaussian distribution (and the model is just identified), maximal uncertainty about the location of \theta is defined, and MML parameter estimates become identical to CML.

Thanks for the illustrations of simulations. I agree CML is a sort of ML estimation, but the algorithm of CML indeed does not require specifying prior distribution of persons, which does not mean the parameter estimators would be necessarily different from those of MML. In your simulation, I feel it is demonstrating the robustness of using a normal distribution but violated in reality. Somewhat similar to the Bayesian inference that we need to specify a prior distribution, although now we are talking about in the frequentist inference. We cannot say much about whether the prior distribution is correct, but sensitivity analysis could help here, as you demonstrated.

But I am still curious why so some studies claim the effect of nonnormal distribution on the parameter estimation. Do they contradict your simulation results? See

http://epm.sagepub.com/content/76/4/662
https://www.ncbi.nlm.nih.gov/pubmed/16953704
http://epm.sagepub.com/content/74/2/343.abstract

 

[Phil Chalmers]

On Thu, Oct 20, 2016 at 1:50 AM, Keith Lau <genw...@gmail.com> wrote:

Phil Chalmers於 2016年10月19日星期三 UTC+8下午10時58分40秒寫道：

On Wed, Oct 19, 2016 at 6:34 AM, Keith Lau <genw...@gmail.com> wrote:

Phil Chalmers於 2016年10月19日星期三 UTC+8上午11時08分28秒寫道：

Straight out of wikipedia: 

"The one-parameter model (1PL) assumes that guessing is a part of the ability and that all items that fit the model have equivalent discriminations, so that items are only described by a single parameter ( {\displaystyle b_{i}} b_{i}). This results in one-parameter models having the property of specific objectivity, meaning that the rank of the item difficulty is the same for all respondents independent of ability, and that the rank of the person ability is the same for items independently of difficulty. "

Of course, this is only true asymptotically. In practice you don't have the true intercepts parameters, so the rank-ordering of the individuals is only guaranteed to be true in larger samples (particularly when items with similar difficulty values exist, because they are likely to be exchanged due to sampling error). 

On Tue, Oct 18, 2016 at 9:40 AM, Balazs Klein <balazs...@gmail.com> wrote:

I keep reading that IRT (at least the one parameter model) is sample independent.

I thought it means that even if I had a non-representative, biased sample the model would reflect the properties of the whole population.

So I will not need to worry about representative samples to the same extent as I had to wit CTT.

Oh my no, that's never true. You ALWAYS need a representative sample for the population with which you are trying to make inferences to. Rasch/1PL are not magical, they are just models that need to be calibrated.

Using MML requires the assumption of person distribution, but not the case for conditional maximum likelihood method specifically for the Rasch model. See eRm package.

This is actually a common misinterpretation of CML, so I think I'll help clear it up here. CML is just a method to obtain ML estimates for item parameters through conditioning on a sufficient statistic supplied by the data. MML achieves the same goal, but specifies a latent distribution (i.e., a range of uncertainty) over which each individual could be located. However, when choosing the Gaussian distribution (and the model is just identified), maximal uncertainty about the location of \theta is defined, and MML parameter estimates become identical to CML.

Thanks for the illustrations of simulations. I agree CML is a sort of ML estimation, but the algorithm of CML indeed does not require specifying prior distribution of persons, which does not mean the parameter estimators would be necessarily different from those of MML. In your simulation, I feel it is demonstrating the robustness of using a normal distribution but violated in reality. Somewhat similar to the Bayesian inference that we need to specify a prior distribution, although now we are talking about in the frequentist inference. We cannot say much about whether the prior distribution is correct, but sensitivity analysis could help here, as you demonstrated.

But I am still curious why so some studies claim the effect of nonnormal distribution on the parameter estimation. Do they contradict your simulation results? See

http://epm.sagepub.com/content/76/4/662
https://www.ncbi.nlm.nih.gov/pubmed/16953704
http://epm.sagepub.com/content/74/2/343.abstract

No I wouldn't say these contradict my results at all, only that the MML approach can take on a more Bayesian adoption to help alleviate the problem. The prior used to marginalize the likelihood can be estimated from the data concurrently, so long as it doesn't consume too many degrees of freedom. Both MML with the normal prior and CML are negatively affected by non-normal latent traits (this wasn't shown in the simulation, but it's easy to add), and as I showed essentially to the same degree. But, MML and other Bayesian methods have a built-in mechanism to help improve the inferences about the item parameters by improving the prior. That's more what these authors are trying to get at. 

Phil

 

To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package+unsubscribe@googlegroups.com.

[Seongho Bae]

Phil was right. I learned what Phil noted who teaches Kang. (Kang developed the polytomous version of S_X2 with Chen in ACT; Kang & Chen, 2007) MML/EM is commonly available in current IRT software, and extension of MML/MCMC variants is available too for estimate accurate item parameters without effects of subject's ability parameter. 

Seongho

2016년 10월 20일 목요일 오후 10시 40분 12초 UTC+9, Phil Chalmers 님의 말:

[robi...@ipn.uni-kiel.de]

Just to put it in a different way. It is often shown in the literature that conditional maximum likelihood estimation is equivalent to a marginal maximum likelihood estimation with a nonparametrically estmated distribution (e.g. Formann or Verhelst). So, the statement that CML do not need distributional assumption contrary to MML is not correct. Furthermore, the true (nonparametric) theta distribution can be only identified asymptotically (number of items tending to infinity) for both (CML and MML) estimation approaches.

[Keith Lau]

Imposing and estimating prior distribution are different things. Nonparametric IRT like Rasmay method (as I know) uses sum scores to estimate person distribution. It does not mean CML needs a prior distribution assumption. I will thank if you can indicate which steps of CML algorithm require that assumption where I might misunderstand.

And sorry for this long off the track, Phil.

robi...@ipn.uni-kiel.de於 2016年10月23日星期日 UTC+8下午4時07分26秒寫道：

[Phil Chalmers]

On Sun, Oct 23, 2016 at 10:53 AM, Keith Lau <genw...@gmail.com> wrote:

Imposing and estimating prior distribution are different things.

Right. What Alexander is referring to is a kind of empirical Bayes approach to the problem. Either way, from my perspective such an issue is largely trivial. I was merely showing the MML and CML are basically identical in all practical respects, though I tend to favour MML for other more obvious reasons (the assumption of latent distributions never bothered me so much as it's not really of consequence with enough data....priors tend to disappear asymptotically, like they tend to in SEM and other latent variable models). 

 

Nonparametric IRT like Rasmay method (as I know) uses sum scores to estimate person distribution. It does not mean CML needs a prior distribution assumption. I will thank if you can indicate which steps of CML algorithm require that assumption where I might misunderstand.

I don't think Ramsay ever claimed the leave-one-out total score to be sufficient in any absolute sense, more that it's a reasonable and efficient mechanism to condition on to obtain kernal smoothed curves for exploratory/functional analyses. But yes I know what you are saying.

Although the total score is a sufficient statistic (in the sense of a mathematical fact) for CML estimation it is still only a stand-in to the true ability and requires the assumption that it is actually sufficient (as a philosophical/validity fact). E.g, a test with only 3 dichotomous items will have 4 total possible scores, but this hardly means that the unobserved abilities are literally sufficient to captured by the empirical information about the distribution of the abilities (surely there are more than 4 possible abilities in the population!). So, statistical sufficiency becomes an assumption about estimability of intercepts assuming the total score is "good enough", but not about validity sufficiency for representation of the underlying abilities (which requires nitems to tend to infty). Hence, why I call it just a trick, which requires an assumption much the same as MML. In the end, something has to be assumed for these models to work: either the assumption that the (potentially weighted) total score is sufficient to capture the ability parameters, or we assume something about the underlying distribution directly and attempt to average over this distribution. 

 

And sorry for this long off the track, Phil.

Not at all, it's why this forum is here. Cheers.

Phil

 

robi...@ipn.uni-kiel.de於 2016年10月23日星期日 UTC+8下午4時07分26秒寫道：

> Just to put it in a different way. It is often shown in the literature that conditional maximum likelihood estimation is equivalent to a marginal maximum likelihood estimation with a nonparametrically estmated distribution (e.g. Formann or Verhelst). So, the statement that CML do not need distributional assumption contrary to MML is not correct. Furthermore, the true (nonparametric) theta distribution can be only identified asymptotically (number of items tending to infinity) for both (CML and MML) estimation approaches.
>
> Am Dienstag, 18. Oktober 2016 15:40:17 UTC+2 schrieb Balazs Klein:
> I keep reading that IRT (at least the one parameter model) is sample independent.
> I thought it means that even if I had a non-representative, biased sample the model would reflect the properties of the whole population.
> So I will not need to worry about representative samples to the same extent as I had to wit CTT.
>
>
> My experience however is that whenever I create a mirt model from a sample (no matter how skewed it is) the sample ability mean will always be very close to 0.
>
> This suggests to me that if I oversampled clever people when creating the model I will still get a higher estimate for theta=0,
> and if I used that model I would still underestimate the ability of the population just like it used to be with CTT.
>
>
> I guess my bottomline question is: do I still need to have a representative sample if I used a simple Rasch or a 2PL model, or is an IRT model really sample independent?
> If it is sample independent, what does that mean and how could I show that?

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package+unsubscribe@googlegroups.com.

[robi...@ipn.uni-kiel.de]

Am Montag, 24. Oktober 2016 03:23:23 UTC+2 schrieb Phil Chalmers:

On Sun, Oct 23, 2016 at 10:53 AM, Keith Lau <genw...@gmail.com> wrote:

Imposing and estimating prior distribution are different things.

Right. What Alexander is referring to is a kind of empirical Bayes approach to the problem. Either way, from my perspective such an issue is largely trivial. I was merely showing the MML and CML are basically identical in all practical respects, though I tend to favour MML for other more obvious reasons (the assumption of latent distributions never bothered me so much as it's not really of consequence with enough data....priors tend to disappear asymptotically, like they tend to in SEM and other latent variable models). 

Yes, I agree. I only wanted to argue that MML and CML are statistically equivalent when MML is appropriately specified (semiparametric MML; estimate the theta distribution on a discrete grid of theta values). In addition, both MML and CML estimation can only identify a fully nonparametrically specified theta distribution in case of an infinite number of items. With a finite number of items, it is not clear whether CML has an advantage to MML.
 

 

Nonparametric IRT like Rasmay method (as I know) uses sum scores to estimate person distribution. It does not mean CML needs a prior distribution assumption. I will thank if you can indicate which steps of CML algorithm require that assumption where I might misunderstand.

I don't think Ramsay ever claimed the leave-one-out total score to be sufficient in any absolute sense, more that it's a reasonable and efficient mechanism to condition on to obtain kernal smoothed curves for exploratory/functional analyses. But yes I know what you are saying.

Although the total score is a sufficient statistic (in the sense of a mathematical fact) for CML estimation it is still only a stand-in to the true ability and requires the assumption that it is actually sufficient (as a philosophical/validity fact). E.g, a test with only 3 dichotomous items will have 4 total possible scores, but this hardly means that the unobserved abilities are literally sufficient to captured by the empirical information about the distribution of the abilities (surely there are more than 4 possible abilities in the population!). So, statistical sufficiency becomes an assumption about estimability of intercepts assuming the total score is "good enough", but not about validity sufficiency for representation of the underlying abilities (which requires nitems to tend to infty). Hence, why I call it just a trick, which requires an assumption much the same as MML. In the end, something has to be assumed for these models to work: either the assumption that the (potentially weighted) total score is sufficient to capture the ability parameters, or we assume something about the underlying distribution directly and attempt to average over this distribution. 

 

And sorry for this long off the track, Phil.

Not at all, it's why this forum is here. Cheers.

Phil

 

robi...@ipn.uni-kiel.de於 2016年10月23日星期日 UTC+8下午4時07分26秒寫道：

> Just to put it in a different way. It is often shown in the literature that conditional maximum likelihood estimation is equivalent to a marginal maximum likelihood estimation with a nonparametrically estmated distribution (e.g. Formann or Verhelst). So, the statement that CML do not need distributional assumption contrary to MML is not correct. Furthermore, the true (nonparametric) theta distribution can be only identified asymptotically (number of items tending to infinity) for both (CML and MML) estimation approaches.
>
> Am Dienstag, 18. Oktober 2016 15:40:17 UTC+2 schrieb Balazs Klein:
> I keep reading that IRT (at least the one parameter model) is sample independent.
> I thought it means that even if I had a non-representative, biased sample the model would reflect the properties of the whole population.
> So I will not need to worry about representative samples to the same extent as I had to wit CTT.
>
>
> My experience however is that whenever I create a mirt model from a sample (no matter how skewed it is) the sample ability mean will always be very close to 0.
>
> This suggests to me that if I oversampled clever people when creating the model I will still get a higher estimate for theta=0,
> and if I used that model I would still underestimate the ability of the population just like it used to be with CTT.
>
>
> I guess my bottomline question is: do I still need to have a representative sample if I used a simple Rasch or a 2PL model, or is an IRT model really sample independent?
> If it is sample independent, what does that mean and how could I show that?

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package...@googlegroups.com.