model identification

[Jeanne Sinclair]

Hello again Phil and mirt forum,

I am attempting to fix my model identification problem and appreciate Phil's great, ongoing support for my clearly quite novice work! Previously Phil had suggested that I might use a 1-dimensional model, since the unidimensional model fit seemed pretty good. I feel that the dimensionality has not been resolved yet and my supervisor is keen to try the multidimensional approach. So here I am trying again. :)

Question 1: 

Can model fit indices (M2) be used/reported for mirt() analyses that are exploratory in nature?  E.g., mirt2d<- mirt(read_mirt, 2)

Question 2: 

Phil had suggested using summary() to see the loadings in the exploratory approach before moving to confirmatory approach. Attached is the output for 2-dim using summary(). The factors do make sense theoretically. I then tried to create a confirmatory model that does not allow F1 to load onto rmc 18, 19, and 26, and does not allow F2 to load onto rmc 1,7,10,14,15, 28, 30, 31. The two factors do correlate. This was my attempt:

[Please note that starting with rmc26 = variable 25, rmc27= variable 26, etc., but only for variables 25-30, which are rmc26-rmc31].

Two_dim <- '

F1 = 1-17, 20-24, 26-30

F2 = 2-6, 8-9, 11-13, 16-26, 28

MEAN=F1, F2

COV=F1*F2

'

xmirt<-mirt(read_mirt, Two_dim, SE=TRUE)

I got this warning:

Calculating information matrix...

Warning message:

Negative SEs set to NaN.

Just calling xmirt2 produced this:

Call:

mirt(data = read_mirt, model = Two_dim, SE = TRUE)

Full-information item factor analysis with 2 factor(s).

Converged within 1e-04 tolerance after 115 EM iterations.

mirt version: 1.25 

M-step optimizer: BFGS 

EM acceleration: Ramsay 

Number of rectangular quadrature: 31

Information matrix estimated with method: Oakes

Condition number of information matrix = 5004864

Second-order test: model is not a maximum, or the information matrix is too inaccurate

Log-likelihood = -63266.6

Estimated parameters: 82 

AIC = 126697.2; AICc = 126700

BIC = 127231.6; SABIC = 126971

G2 (1073741741) = 49248.56, p = 1

RMSEA = 0, CFI = NaN, TLI = NaN

As it appears that the model is not sufficiently specified, I would appreciate any tips about what sorts of changes I might be able to make  in order to correctly identify the IRT model.  For example, are CONSTRAIN, CONSTRAINB, and/or PRIOR necessary components for the confirmatory approach? If so, are there guidelines as to how these can be determined? Would examining the par values for commonalities be the first step?

Many thanks again!

Jeanne 

[Jeanne Sinclair]

Hello again mirt forum,

Update: I was able to designate dimensionality (2 factors) and constrain a1, a2,  and d parameters on some items by running mirt.model and then multipleGroup calling that model.  So my current question is this.... Building the constraints with CONSTRAIN in mirt.model and then running that model in multipleGroup then constrains the parameters within groups, across items. I can be fairly certain that that constraining parameters in mirt.model is producing a decent model with good fit indices. However, I am not sure that those constraints are applicable for BOTH reference and focal groups (note: I am not using CONSTRAINB, so I'm not talking about constraining across groups). 

However the M2 stats also look good for multipleGroup when CONSTRAIN holds parameters across items, within groups, so maybe this does make sense. However since 2-dim DIF is the goal, I'm not at all sure the constraining of parameters across items within groups make sense....For example one item (number 13) that had all three parameters constrained in the original mirt.model did not show any differences between groups at all. (p=1). 

I think I am pretty turned around here in terms of the steps with this so would love any advice the forum could offer! I am trying to use the help manual... but am still uncertain. Essentially I am unsure if I should ever place constraints on items that I will test (in mirt.model, below called Two_dim_c). 

Here is the code, and thanks!

-Jeanne

Two_dim_c <- '

F1 = 1-30

F2 = 1-29

CONSTRAIN=(18,4,19,26,a1), (2,3,5,17, a1), (21,8,13,10,11,9,a1), (24,6,22,7,25, a1),

(2,17,11,12,13,a2), (10,22,1,21,24,26,29,a2), (15,14,28,a2),

(22,8,11,d), (27,9,13,d), (25,30,20,d)

'

xmirt5<-mirt(read_mirt, Two_dim_c, SE=TRUE)

model_anchor1 <- multipleGroup(read_mirt, model = Two_dim_c, 

                               group = group,

                               invariance = c(itemnames[1:11], 'free_means', 'free_var'))

anchor <- DIF(model_anchor1, c('a1','a2', 'd'), items2test = 12:29)

[Phil Chalmers]

Hi Jeanne,

This model looks identified, but I'm not sure where you're trying to
go with this DIF analysis. It seems as though you want to test bias in
each multidimensional item, however you're uncertain of the loading
structure (hence, the use of CONSTRAIN and exploratory IRT models).
DIF testing is a fairly advanced topic in the measurement modelling
process, which occurs only after a suitable loading pattern has been
established (note that unidimensional IRT models don't have this
problem because the confirmatory pattern is automatically identified
in a restrictive way due to requiring all items to load on one
construct). So, it looks here like you're investigating DIF for and
exploratory IRT model....which feels a little on the peculiar side.

My observation last time about non-identifiability was because you
freed that latent mean and variance parameters for both groups instead
of just the focal group(s). Here it looks like your model is setup
better, though you could actually remove all the CONSTRAIN arguments
and it still would be identified (the anchor item constraints, via
CONSTAINB, are automatically being applied through the "invariance"
argument). HTH.

Phil

> --
> You received this message because you are subscribed to the Google Groups
> "mirt-package" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mirt-package...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Jeanne Sinclair]

Thank you, Phil! I feel I am making the same mistake over and over, so thanks very much for your patience.

I still have several problems in terms of my own understanding, but I’ll ask for now just the most foundational question…. if the code I used isn’t confirmatory, what could make it so? Is it a matter of identifying and ascribing parameters fixed/starting at certain values within mirt.model?

Regarding factor loadings, the 2 factors appear to be loading on all items except rmc30 (where F2 is 0.00). However... if they are loading in exploratory at less than .10 (or some other value? .30?) , can they be removed from that factor for confirmatory? As of now I’ve included in the confirmatory model items that in the previous exploratory model had demonstrated any loading greater/less than 0.00 on a given factor.

Grateful for your time and expertise,
Jeanne

[Jeanne Sinclair]

Oh sorry, I forgot to address your other point! Where am I headed with this DIF? Yeah, I’d like to know that, too. :) My sample is reading comprehension population data. If there are two dimensions to this assessment, then I’d like to find out if items are equally discriminating for students from two different home language backgrounds (I’ve reduced my six home language groups to 2 — monolingual English and multilingual), on each factor. Both home language groups have similar total scores.

It appears that the first dimension favors multilingual students, namely processing across text (summarizing). The second factor appears to be the skill of making inferences about characters’ feelings, which I argue is culturally situated — it theoretically makes sense that such items are favoring students from English-only home environments.

Maybe a bifactor is better, although I am not at all sure how to run bifactor DIF — three factors I guess? One general, one summarizing and one inferencing about characters’ feelings?

I also have not yet taken a more qualitative approach, which may be preferable — to identify a hypothesis based on analyzing each item's elicited skill (e.g., I read the items and apply a Q-matrix), and then using the confirmatory approach to test that hypothesis. Probably I should try that next.

This analysis is primarily for training purposes for me, just trying to wrap my head around the process. So once again, thanks very much for your ongoing support!

Jeanne



> On Oct 14, 2017, at 9:14 AM, Phil Chalmers <rphilip....@gmail.com> wrote:
>