multipleGroup

[Steven Kalinowski]

Hi Phil,

I have a question about how to use the the multipleGroup function of mirt to best estimate item parameters for a test I am developing. I have given different versions of the test to my introductory biology class three semesters in a row (Sem1, Sem2, and Sem3). The student population should be similar each semester. I am still developing the test however, and searching for a combination of 18 questions that measure a single latent variable. Some of my questions seem to work well, and I have used them all three semesters. However, each semester, I have been testing out new questions. The table below shows my sample size for each semester for 18 question. You can see in the table that I used the control-of-variables question, "CONT_Corn" all three semesters, but started using the proportional reasoning question, "PROP_Wizard" in Semester 2. I want to measure the difficulty, slope, and guessing rate for all of the questions and figure the best way to do this is to use all of my data. It seems like it would be a shame to throw out the data from previous semesters. I first tried combining the data by pooling all of it in one datafame... with lots of missing data for questions that were not used in some semesters, but that had the effect of dramatically lowering the empirical reliability of the test. The ER for this test has always been ~0.80, but the ER for the pooled test was 0.70, so something didn't seem to be working. Next, I tried using the multipleGroup function, but I am not sure if I am setting my constraints incorrectly. I used the following command:

fit.pooled <- multipleGroup(data=pooled.data, model=1, itemtype= "3PL", group=group, invariance=c( 'free_means', 'free_var', c("CONT_Corn", "CONT_Pendulum", "CONT_RedFlies",

                                                                                                    "PROP_Photo", "PROP_Lemonade", "PROP_Wizard",

                                                                                                    "PROB_6Blocks", "PROB_24Blocks", "PROB_Spinner", 

                                                                                                    "HD_Wolves2", "HD_FC", "HD_Pigeons",

                                                                                                    "CORR_XYPlot", "CORR_ReefFish", "CORR_HeartAttacks",

                                                                                                    "LOGIC_Shapes", "LOGIC_CandyNotP", "LOGIC_TallDoctors")))

When I do this, I seem to get IRT parameters for questions that were given to a group (semester). Am I doing something wrong? And could you provide some advice on how best to estimate my item parameters given my sampling design. 

I appreciate the help you have given me in the past, and would appreciate any guidance you can offer now.

Steven Kalinowski

Dept. of Ecology

Montana State University

Table 1. Sample sizes for 18 questions given in different versions of a test on three semesters.

                      Sem1   Sem2   Sem3 Total

CONT_Corn             206    221    200   627

CONT_Pendulum         206    221    199   626

CONT_RedFlies         206    221    198   625

PROP_Photo            206    221    199   626

PROP_Lemonade          0     0      200   200

PROP_Wizard            0     221    200   421

PROB_6Blocks          206    221    200   627

PROB_24Blocks          0     221    199   420

PROB_Spinner          206    221    200   627

HD_Wolves2             0     0      200   200

HD_FC                  0     221    200   421

HD_Pigeons             0     0      200   200

CORR_XYPlot            0     221    200   421

CORR_ReefFish         206    221    200   627

CORR_HeartAttacks     206    220    200   626

LOGIC_Shapes           0     221    200   421

LOGIC_CandyNotP        0     221    200   421

LOGIC_TallDoctors      0     221    0     221

[Phil Chalmers]

Hi Steven,

Your setup sounds correct as far as I can tell. The parameters you are seeing for the semesters which did not actually take the item (complete NA's) are simply being ghosted-in, where the information is coming only from the other 1-2 semesters. In fact, you need these equality constraints, otherwise the model will spin out of control because it won't be identified. You should also be in a situation where you can loosen up some constraints (slope-intercept combinations) to determine whether the items are responding the same across semesters (DIF). 

The only thing I really worry about in your case is the estimation of the lower-bound parameters in the 3PL. These are notoriously difficult to obtain accurately (often needing N > 5000 for ML estimates), so you may want to consider either a) setting prior parameter distributions on the 'g' parameters to help keep the model stable, or b) imposing further equality constraints to make the guessing parameters equal within each semester or across all semesters. Cheers.

Phil

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.