Rasch item difficulty parameters: continuous response in lavaan

[Simon Harmel]

Dear all,

I hope this email finds everyone safe and well. I want to estimate the Rasch item difficulty parameters for my test items (data below). However, I have two challenges:

(1) Item scores are continuous between 0 and 1 (item_score)

(2) Test is adaptive and thus, items (item_id) vary across persons (person_id)

I was wondering if it might be possible to take (1) and (2) into account using lavaan?

Many thanks, Simon

#===== Dataset in R

dat <- read.csv('https://raw.githubusercontent.com/izeh/n/master/g.csv', stringsAsFactors = F)

[Nickname]

Simon,
  Do you mean the continuous item responses in the unit interval to represent probabilities?  Could you dichotomize the responses at .5 and fit a conventional Rasch model?  Alternatively, could you gently rescale the responses to exclude values of 0 and 1, apply a logit transformation, and then fit a linear CFA to the transformed responses as an approximation to a logistic model?  Could you define the Rasch difficulty and descrimination as defined parameters and then use those to impose equal discrimination across items?  If so, it might be helpful to fit the unconstrained two-parameter model first, just to give yourself a reference point for evaluating the constrained solution.

  Is it safe to assume that you want to estimate parameters for specific items, not the the average across whatever item respondents answered in a particular ordinal position in their test?  If so, you probably need to arrange your data so that the columns represent unique items and the rows represent test takers.  You then have a standard data set up with a missing data issue represented by the blank cells in the data set where a given test taker was not presented with a given item.  That issue is hopefully manageable if you have a very large data set.  Lavaan offers both FIML and MI approaches to missing data.  The models you are fitting assume no order effects, the probability conditional on ability is independent of exposure to prior items.

Keith
------------------------
Keith A. Markus
John Jay College of Criminal Justice, CUNY
http://jjcweb.jjay.cuny.edu/kmarkus
Frontiers of Test Validity Theory: Measurement, Causation and Meaning.
http://www.routledge.com/books/details/9781841692203/

[Simon Harmel]

Dear Keith,

Thank you very much for your valuable perspectives. If you check the dataset, I have 16004 unique items (item_id) given to 2000 test takers (person_id). But if you look at how these items haven been assigned to the persons, you realize some items have been taken by as many as 347 persons, while some have been taken by only 1 person (overall, the median frequency of using a specific item by test takers is 3 times !red flag! see figure below, there are only 5 unique items taken by >= 200 persons).

So, to make the data wide would mean two things:

(1) I could widen the data such that I have 2000 rows (persons), and `16004` columns for each unique item.

(2) I could widen the data such that I have 2000 rows (persons), and `37` columns for each item_type attempt (e.g., person # 1, took 3 of X type item, 4 of Y type item).

Option 1 for widening the data, would lead, therefore, to the majority of items being barely estimable, right? (because only few people have taken them)

Option 2 for widening the data, would lead, however, to item_type attempts (e.g., each person taking x number of Y type item, w number of Z type item etc.)

If my reasoning regarding the data widening is correct, then it may seem more reasonable to estimate item difficulty parameter really by their type, and not unique items per se, right?

If so, I'm attaching the option 2 widened dataset and would be really interested to know how you would model it and whether you agree with dataset?

Thanks, Simon

#== New widened dataset (by item_type):

read.csv('https://raw.githubusercontent.com/izeh/n/master/w2.csv', stringsAsFactors = F)

 

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/5cd21e8a-a856-41fd-9140-093b6bb7972e%40googlegroups.com.

[Jeremy Miles]

Hi Simon,

As I mentioned on CrossValidated (https://chat.stackexchange.com/rooms/107527/discussion-between-jeremy-miles-and-user7148318) SEM is not going to work for this - you have (far!) too many variables, and too much missing data. 

A mixed model is equivalent to a single factor CFA - you can fit this in a mixed model - the item intercepts are the difficulty parameters.

The part I don't understand is where you say "The items come with their machine-learned difficulty parameters."

You either estimate them, or you don't. I don't think there's an in-between - unless you want to go Bayesian and use the machine learned difficulties as prior distributions of the parameters. I'd model that using Stan.

Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CACgv6yWpAc2S5n9S3Jof1PhhScOcVHWD76pt%2B%2BobX1_sRKGfGA%40mail.gmail.com.

[Simon Harmel]

Please disregard "the items come with their machine-learned difficulty parameters".

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAMtGSxm%3DL_cCirsRtKGDp10DzJVctnSt-xT8ogYtq6PyVZQWyQ%40mail.gmail.com.

[Jeremy Miles]

In that case, you might be able to fix the mixed model (if you have more RAM than me. :)  ).

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CACgv6yXZLpjto7iM68BCB2MsncxnSQr0TwtD9LWk0XXHKz9x6A%40mail.gmail.com.

[Simon Harmel]

I think the approach supported by this paper seems to exactly suite my type of data. It requires a dyadic expansion R function (called num2dyadic) to convert the continuous responses to binary responses. 

It seems after this conversion, a 2PL model with known discrimination parameters can be fit. But after the conversion, there is a significant number of 0s but few 1s. So, I'm not sure if I'm doing the conversion correctly.

dat <- read.csv('https://raw.githubusercontent.com/izeh/n/master/g.csv', stringsAsFactors = F)

dat$item_score2 <- num2dyadic(dat$item_score, nmax = length(dat$item_score)) # the conversion

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAMtGSxnn6K0N0Sz6BQ1qncWgfm7JoE%3DFGtkZ_nPWLjOsyLBHzQ%40mail.gmail.com.

[Nickname]

Simon,
  Option 1 is not an option for the reasons Jeremy gave.  You cannot have more columns than rows in your data set if you want to analyze covariance structures.

  It seems to me that the key issue for Option 2 is whether you expect the variations that distinguish different items of the same type to impact item parameters.  The approach appears to assume that they do not.

  I am have not thought this through, but I wonder if you might want to consider some form of explanatory IRT model.  If, instead of treating every new item as a unique value of a nominal variable with its own parameters, you instead could scale the items in a n-dimensional space of item characteristics, then you could model the item difficulties as functions of those characteristics.  I am wondering whether that might constrain the parameters sufficiently much that even your rare items could be useful and informative.  Just a thought.

[Simon Harmel]

Thanks, Keith. Is is possible to reduce the data below to make composite (synthetic) variables from the same-type items? For example, all `crests` columns become 1 composite? What do we need to make such composites?

Thanks, Simon

#== New widened dataset (by item_type):  

read.csv('https://raw.githubusercontent.com/izeh/n/master/w2.csv', stringsAsFactors = F)  

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/0dd6db85-c18a-44d3-ac5b-bf71bdd69e6e%40googlegroups.com.

[Nickname]

Simon,
  I do not want to speculate too much about the nature of the data or your goals in the analysis because that leads into a garden of forking paths.  This thread is also drifting further away from being a question about using lavaan.

  However, if the Rasch model is appropriate to your data, then you already have a workable measure of item difficulty in the probabilities themselves.  Estimating Rasch difficulty parameters would just translate those into the metric of latent ability.  However, unless I am missing something, the mean probabilities across respondents and the difficulty parameters should rank order the items in the same way.  If the Rasch model is not appropriate and the item characteristic curves cross, then this is not the case.

  It still strikes me that the key assumption in treating items of one type as all the same is that whatever distinguishes different items of the same type does not affect the item parameters.

  When you have multiple items of the same type for the same respondent, then the key assumption would seem to be that the mean probability from multiple items has the same statistical properties as a probability derived from a single item response.  Consider whether the CFA unique variances might be less for a composite than for a single item.  Given the size of your data, it seems reasonable, at least as a first pass, to randomly select one item of each non-missing type for each respondent and analyze the resulting 2000x37 data matrix.  Moreover, there is nothing wrong conducting more than one analysis of the same data, so long as you report the results in a transparent way.  You could at least partially test the above assumptions in ancillary analyses by imposing equality constraints and conducting chi-square difference tests.

[Simon Harmel]

Keith, Thank you!

In lavaan, when using logit transformed composite item scores, do I
need to do any kind of transformation on the intercepts to obtain the
item difficulty parameters?

I think I also should multiply the intercepts by -, right?

Thank you,
Simon

> --
> You received this message because you are subscribed to the Google Groups
> "lavaan" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lavaan+un...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/lavaan/d892b724-5c94-48fa-8a26-9deab36a97ff%40googlegroups.com.
>

[Nickname]

Simon,
  I have not thought through how the equivalencies would play out in your analysis.  The below reference might be helpful.  I believe that there is a more current reference that gives the expressions for each set of parameters in terms of the other set but it is escaping me at the moment.

Takane, Y. and de Leeuw, J. 1987. On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52: 393–408.

[Can Gürer]

Hi Simon,
this kind of date is perfectly for Müller's continuous Reach model. Check out the package pcIRT (https://www.jstatsoft.org/article/view/v084c02). Just make sure there's enough overlap between items over persons.
If there's not a lot overlap, then the LLTM might be a choice, although I don't know if that has been implemented for continuous data yet.

If you can't find Müller's article (side note: it's a bit technical, he was a physicist), then let me know. I hope I'll get a group update.
Cheers
Can