Questions on validation of short scale

[Yuen Wan Ho]

Hi, I am working on development of short scale using IRT. I got some results but I was unsure which results I should use for validation of short scale. I am writing to see if anyone can provide suggestions to help interpret the results I got in validation.

I need to shorten an 80-item scale (in yes/no format), which measures a personality factor (named as Social Potency), this factor covers 8 facets (each facet has 10 items). I used a normative sample data set and split the data into half. The first half of dataset is used for item selection , and the remaining half is used for scale validation. For shortening the scale, I used the first half of the data set and tried two approaches. 

For the first approach, I used the bifactor model, treating Social Potency as a general factor and then 8 facets as specific 'factors'. The results showed  quite a number of items had slope parameters to general factor with .90 below. I also found LD for some of the paired items. Then I used the uni-dimensional model (2PL, using mirt package), treating all 80 items as a whole to shorten the scale, which finally resulted  in 15 items. The shortened scale includes items coming from 5 facets, not 8 facet. I used second half of normative data to use mirt function (full information item factor analysis) to examine the factor structure for short scale. The results showed a good model fit (RMSEA =.04, TLI and CFI = .97). 

For the second approach, I conducted irt (still using 2PL and mirt package) for each facet and removed items within each facet, resulting in 24 items (3 items for each facet * 8 facet =24 items).  I used second half of normative data to use mirt function (full information item factor analysis) to examine the factor structure for each facet. The model fit was good for each facet. However, the model fit for 24 item as whole was not satisfactory (RMSEA =.065, TLI  =.86 and CFI = .88). I used CFA approach to evaluate the factor structure for the 24 item short scale. The results showed a good model fit (RMSEA =.04, TLI  =.92 and CFI = .90).

As I got two results, which approach should I rely on?

Thank you very much!

[Seongho Bae]

1. Is facet is important to you? Social Potency score have to report each factor score instead general score?
   
2. Why are you split data in item response theory perspectives? Under the Model-based approach may not important split randomly. Because the IRT is not sample-based approach. If you had the representative sampling, you can apply item parameters to another sample.
3. Did you check item fit statistic in one-factor solution? In the IRT perspectives, validity can justification by item fit statistics. In this situation, the short scale may not need testlets.

Best,

Seongho

2016년 8월 12일 금요일 오후 4시 29분 16초 UTC+9, Yuen Wan Ho 님의 말:

[Yuen Wan Ho]

Dear Sheongho,

Thanks for your suggestions.

For your question 1:  The theoretical construct of Social Potency is the blending of extroversion and openness facets. After shortening, all items of extroversion facet are removed. It creates a problem when explaining the theoretical construct for  Social Potency short scale. 

For your question 2: When using the representative sampling (i.e., I used all data (N = 1500) for item selection, instead of using the first half of normative data (N =750)  for item selection), the remaining items for the short scale are a bit different from the results I obtained using split data. Which results (split data or representative sample) should I rely on? 

For your question 3: After checking the item fit statistic, one item of the short scale using the First approach (mentioned in previous message) gave a significant S-X2, 2 items of the short scale using the Second approach (mentioned in previous message) gave a significant S-X2. When using the representative sampling, the model fit for the short scale is (RMSEA =.03, TLI and CFI = .98), though 2 items of this short scale gave a significant S-X2. 

Thanks for your help or suggestions in advance!

Best,

Yuen Wan

Seongho Bae於 2016年8月13日星期六 UTC+8上午1時28分04秒寫道：

[Seongho Bae]

For your question 1:  The theoretical construct of Social Potency is the blending of extroversion and openness facets. After shortening, all items of extroversion facet removed. It creates a problem when explaining the theoretical construct for  Social Potency short scale.

Can I get files (code and raw data) for replication if that isn't sensitive? I can't reply anything without files.

For your question 2: When using the representative sampling (i.e., I used all data (N = 1500) for item selection, instead of using the first half of normative data (N =750)  for item selection), the remaining items for the short scale are a bit different from the results I obtained using split data. Which results (split data or representative sample) should I rely on? 

Don't split it. Use all cases.

For your question 3: After checking the item fit statistic, one item of the short scale using the First approach (mentioned in previous message) gave a significant S-X2, 2 items of the short scale using the Second approach (mentioned in previous message) gave a significant S-X2. When using the representative sampling, the model fit for the short scale is (RMSEA =.03, TLI and CFI = .98), though 2 items of this short scale gave a significant S-X2. 

I can't reply this without any files without replication.

2016년 8월 15일 월요일 오후 9시 31분 11초 UTC+9, Yuen Wan Ho 님의 말:

[Yuen Wan Ho]

Hi Seongho,

Thanks so much for your help ../__\ :::: It is a sensitive data set so I cannot share the data set with you. It is so kind of you to offer my guidance and support. Perhaps I can leave my code here and see if I have done anything from when shortening the scale:

I used the following syntax for testing the bifactor model of the Social Potency (8 facets, 1 factor)

dataset = read.spss(file.choose(), to.data.frame=TRUE)

specific <-c(1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3,3,4,4,4,4,4,4,4,4,4,4,5,5,5,5,5,5,5,5,5,5,6,6,6,6,6,6,6,6,6,6,7,7,7,7,7,7,7,7,7,7,8,8,8,8,8,8,8,8,8,8)

b_mod1<-bfactor(dataset,specific, SE=TRUE)

coef(b_mod1, printSE=TRUE)

summary(b_mod1)

residuals(b_mod1)

itemfit(b_mod1, QMC=TRUE)

M2(b_mod1, QMC=TRUE)

I used the following syntax for conducting IRT:

Dataset = read.spss(file.choose(), to.data.frame=TRUE)

x<-(mirt(Dataset,1,itemtype = c('2PL'), guess = 0, SE = TRUE))

coef(x, printSE=TRUE)

summary(x)

residuals(x)

M2(x)

The rule for item selection:

I selected items with slope values of .9 or higher, paired items with LD values (below 10) . I also checked the model fit. 

Thanks so much !

Yuen Wan

Seongho Bae於 2016年8月16日星期二 UTC+8下午4時50分58秒寫道：

[Seongho Bae]

Seems good. In my opinion, try to use itemtype = 'ideal'.

You may select items with LD values or Zh values, S-X2 in itemfit() function.

Seongho

2016년 8월 17일 수요일 오후 4시 53분 40초 UTC+9, Yuen Wan Ho 님의 말:

[Yuen Wan Ho]

Thank you very much very much. I have a question about model fit testing. My supervisor wishes to keep 8 dimensions (subscales) for the Social Potency factor. She is interested to know whether it is possible to use irt for selecting items within each subscale in our first split half sample and then use CFA to test the model fit of the Social Potency short scale. 

Thanks alot!

Seongho Bae於 2016年8月18日星期四 UTC+8下午7時07分51秒寫道：

[Phil Chalmers]

It's probably going to be faster to first obtain the exploratory model via something like psych::fa(), and then confirm the structure with mirt to see how well it fits (its exploratory anyway, so approximations are all fair game). When it comes to exploratory data analysis there really aren't any set rules, so do what makes sense for your project/data. Cheers. 

Phil

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yuen Wan Ho]

Dear Phil,

Thank you alot for your advice. I have a question on whether it is still okay to conduct IRT to shorten each dimension/facet for the original factor when the original factor structure does not have equal weights for the individual subscales/facets. 

Yuen Wan

the original factor structure did not have equal weights for the individual scales

Phil Chalmers於 2016年8月26日星期五 UTC+8上午7時47分03秒寫道：

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package...@googlegroups.com.