rwg and ICC

[V]

Hi Group!

My dataset consists about 50 organizations. In each organization, the
survey was answered by several employees. To justify aggregating the
individual level data to the organizational level, I want to compute
measures of agreement (rwg) and reliability ICC(1) and ICC(2).

Am I right in my understanding that I cannot compute rwg directly in SPSS?
So I would have to get the variances for each variable, and then plug into
the formula to get an rwg per variable per organization?

Also given that my data is arranged thus - rows are the employees, nested
within organizations and the columns are the various variables they rated -
can I compute ICC in SPSS directly? (My back up option is to get the SS
from ANOVA and plug it in the formula, but I have several variables and I
wanted to check if SPSS would do this).

Any tips welcome..Thank you for your help!!

[V]

Just to clarify,

I am interested in looking at ICC(1) and ICC (1,k) or ICC(2)

Thanks!

"V" <vaidyan...@osu.edu> wrote in message
news:%ZRNc.10546$09.13...@news4.srv.hcvlny.cv.net...

[Bruce Weaver]

I did not know what "rwg" was. In fact, it is r_wg (i.e., the wg is a
subscript), and is touted as a measure of interrater agreement. Here are
two articles about it, for those who are interested:

James, L.R., Demaree, R.G., & Wolf, G. (1984). Estimating within-group
interrater reliability with and without response bias. Journal of Applied
Psychology, 69(1), 85-98.

James, L.R., Demaree, R.G., & Wolf, G. (1993). rwg: An assessment of
within-group interrater agreement. Journal of Applied Psychology, 78(2),
306-309.

Note the use of "reliability" in the first article, and "agreement" in the
second. I don't know the details of this literature, but I think this
change may be due to criticisms leveled in this paper by Schmidt & Hunter:

Schmidt, FL, & Hunter, JE. (1989). Interrater reliability coefficients
cannot be computed when only one stimulus is rated. J of Applied Psych,
74, 368-370.

SPSS has no direct means of computing this. You will have to do it
yourself, plugging into the formula, as you say. But according to the 1993
article, you will first have to determine what distributional form to use
for sigma-squared_E (the theoretical errors).

HINT: Since you will be computing an r_wg for each group, you might want
to "sort cases by group", then use SPLIT FILE.

For info about the various ICCs available in SPSS, see:

http://www.nyu.edu/its/socsci/Docs/intracls.html

For lots of info on agreement in general:

http://ourworld.compuserve.com/homepages/jsuebersax/agree.htm

HTH.
--
Bruce Weaver
wea...@mcmaster.ca
www.angelfire.com/wv/bwhomedir/