Multiple-choice questions with one correct answer
arisai...@yahoo.com
It has 23 items. Each item has 3-4 answers and only 1 answer is
allowed.
In the SPSS, I have chosen 'scale' as the measure type and enter '0'
for wrong answers and '1' for correct answers. The answers of each
participant then have been computed to get the total correct answer.
I have run a reliability test on the 23 items. However, the cronbach
alpha reading was low .591. It also shows a warning as below:
"Warnings
The determinant of the covariance matrix is zero or approximately
zero. Statistics based on its inverse matrix cannot be computed and
they are displayed as system missing values".
What does this mean? Am I entering the data correctly?
Thank you
Aisha
arisai...@yahoo.com
Rich Ulrich
wrote:
0.591 is not necessarily a terrible alpha for 23 binary items, but
it depends mightily on what the subject area is.
I take it that you do not know what a determinant is, so that
makes detailed discussion slower. When you do a principal
component-type of decomposition of a matrix, you get a
set of components of varying sizes; the determinant may be
thought of as product of the sizes. The maximum number of
non-zero components is the minimum of the number of variables
and the number of rows. The message usually indicates "zero"
and not merely small.
The determinant will be zero if
- some item is a constant (either 0 or 1 in all rows for 0/1);
- there are fewer rows than there are variables;
- there is some linear dependency among rows.
The simplest linear dependency is when two items have
exactly the same (or exactly opposite) responses.
The second simplest may be when you have included a total along
with the items that make it up.
However, there is a linear dependency whenever any combination
of rows can be "added" (with multipliers allowed) to create some other
row. This is guaranteed to be possible when there are fewer subjects
than variables.
--
Rich Ulrich
arisai...@yahoo.com
Many thanks for the explanation
I now understand what is determinant and how the determinant be zero.
However, what does it mean by 'there are fewer rows than there are
variables'?. My research subject is 159. I also did not include the
total score in the reliability test.
I am aware that .60 is acceptable. Nevertheless, the original author
reported the alpha reading > .70. This is somewhat worrying me.
Thanks again
Aisha
Rich Ulrich
wrote:
>Hi Rich,
>
>Many thanks for the explanation
>
>I now understand what is determinant and how the determinant be zero.
>
>However, what does it mean by 'there are fewer rows than there are
>variables'?. My research subject is 159. I also did not include the
>total score in the reliability test.
Yes, rows=subjects. You haven't been forthcoming about
what the data actually are, so I wasn't sure they were Subjects.
People? Questionnaire data?
So you don't have that problem.
Are any of the means really low/high? Are any of the correlations
especially high? I'd look at that information before I ever looked at
the alpha. As I mentioned, any item being constant gives a zero
determinant, or any two items being identical in answers.
Any time that I have a set of questionnaire items, I ordinarily
perform a factor analysis, just to see what structure is there -
and to make sure of stuff like, "right variables", "right labels",
and so on. There will be factors, and they should make have some
face validity as factors for this scale. - I think that the factor
procedure might do something to flag zero determinants, and I
know that it would pick out any "duplex" or pair that correlate
almost perfectly.
Also -
If I recall right, you can locate more subtle dependencies by a clever
misuse of the regression procedure. Take *any* variable as the
outcome for the regression -- "ID" -- and use all the items as
predictors. The regression will drop out some variable(s) as being
"redundant".
>
>I am aware that .60 is acceptable. Nevertheless, the original author
>reported the alpha reading > .70. This is somewhat worrying me.
>
--
Rich Ulrich
arisai...@yahoo.com
I did respond to you last night. But it did not appear. I am new to
this group anyway. So, I am not sure whether I posted it correctly or
not.
Anyway, my response was:
My research samples are patients and I used a questionnaire.
None of the correlations are especially high. Which mean are you
referring to? Is it the inter-item correlation mean? If it is, the
mean is low (.065)
The original author has also advised me to perform the factor
analysis. However, I am not really familiar with factor analysis. I
will consider it but need to read about it first.
If I do the regression procedure, what should I do with the redundant
items? Should I exclude it when performing cronbach's alpha test?
Thank you
Rich Ulrich
wrote:
>Hi Rich,
>
>I did respond to you last night. But it did not appear. I am new to
>this group anyway. So, I am not sure whether I posted it correctly or
>not.
A reply came to me in email, but I prefer to post to the group...
>
>Anyway, my response was:
>
>My research samples are patients and I used a questionnaire.
>
>None of the correlations are especially high. Which mean are you
>referring to? Is it the inter-item correlation mean? If it is, the
>mean is low (.065)
That seems unreasonably low. Are you sure that 0.065 is the
mean of the correlations? And nothing negative? I've never had
an attempted scale look so uncorrelated. (And if the correlations
are so near zero, you do not have a coherent scale of any latent
factor; and factor analysis would not be especially useful, either.)
However, I mean the items -- my point was this. You can't have
an item with zero variance (i.e., a constant) in the reliability.
You can't have an item (or items that total up) equal to another
item. An item is more apt to match something if it has only one
or two endorsements. - Another thought: Proofread your
variable list for Reliability. I once saw such messages when I
accidentally listed the same item twice.
>
>The original author has also advised me to perform the factor
>analysis. However, I am not really familiar with factor analysis. I
>will consider it but need to read about it first.
>
>If I do the regression procedure, what should I do with the redundant
>items? Should I exclude it when performing cronbach's alpha test?
Yes, that is the end-result after you identify the variable, if
there isn't anything else but accident and randomness as the
reason why.
--
Rich Ulrich
Art Kendall
error prone. As a rule of thumb, it is better to let the software do
tedious things like recoding. I usually suggest entering the data "as
is" i.e., a,b,c,d or 1,2,3,4.
Did you double check your manual recoding?
Did you double check your data entry?
How many cases do you have?
Do you have any missing data?
What N is the reliability based on?
What is the construct you are trying to measure? Some kind of knowledge
or ability? How id you decide what is a "correct" answer?
Is this a scale that has been used in prior research or did you put it
together for this study?
From the printout what are the low and high inter-item correlations?
What do the "corrected item-total correlations" look like?
Is there any pattern of negative correlations in the inter-item
correlations which might hint that there was an error in scoring or that
there is more than one construct the respondents might be using to answer.
What the error message is telling you is that in the remaining data
after listwise deletion there are more items than cases, or if there are
more cases than items, one or more items is (close to) perfectly
predictable from one or more other items.
Art Kendall
Social Research Consultants
arisai...@yahoo.com
Items in the questionnaire: 23 items
Missing data: none
N=159
Construct: Knowledge about diabetes. I give 1 mark to a correct answer
(each item). Each item has multiple choices of answer ranging from
a,b,c,d and a,b,c. However, only 1 correct answer for each item. For
example;
1. Which of the following is highest in carbohydrate?
a. Chicken
b. Milk
c. Rice (correct answer)
d. Peanut butter
1 mark is given if the research participant answer 'C', and zero mark
will be given if they answer either a,b or d.
In the SPSS, I only enter either zero or one. In the Label, I code it
as '0=wrong and 1=correct'.
What do you mean by entering the data as it is? Is it, I need to code
the data as it is (in the questionnaire)? e.g. a,b,c,d and then give
the value either zero or one?
This scale has been used in previous researches.
The Alpha reading for research was low with many negative correlation.
Thank you
Aishairma
arisai...@yahoo.com
Bruce Weaver
You could use a DO-REPEAT, as in this example:
www.angelfire.com/wv/bwhomedir/spss/do_repeat_example01.txt
HTH.
--
Bruce Weaver
bwe...@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/Home
"When all else fails, RTFM."
arisai...@yahoo.com
Rich Ulrich
Bruce,
that's nice.
Aishairma,
That solution implies that you could enter your data as the
alphabetic response (or numbers if you want), and check against
a 'key' - which is easy to create.
Now that you have explained that your data are not attitudes or
symptoms (which would be more highly correlated), I suspect that your
intercorrelations are not suspicious, after all. You are asking for
items that to many people are rather like random trivia. Your total
of Right is a measure of knowledge.
What is always true about reliability, but is often irrelevant when
it is not simply forgotten, is that any reliaibility is also a measure
of the sample. So if your sample includes a wide range of people
in terms of how much they know, you would get a higher Reliability
score than (say) if you intentially excluded dieticians and chefs
from a sample of "ordinary people". Or if you excluded ordinary
people and looked at professionals-only. A wider range implies
a higher reliability.
Consider your reference source and what it says about its sample.
That is the baseline sample for considering the variability of your
own sample.
--
Rich Ulrich
PS - I read elsewhere that reading from Google Groups was
disrupted for a couple of days. I assume that accounts for the
re-post.
RU.
arisai...@yahoo.com
>
> > Hi all! I am using a set of multiple-choice questions in y reaserch.
> > It has 23 items. Each item has 3-4 answers and only 1 answer is
> > allowed.
>
> > In the SPSS, I have chosen 'scale' as the measure type and enter '0'
> > for wrong answers and '1' for correct answers. The answers of each
> > participant then have been computed to get the total correct answer.
>
> > I have run a reliability test on the 23 items. However, the cronbach
> > alpha reading was low .591. It also shows a warning as below:
>
> > "Warnings
> > The determinant of the covariance matrix is zero or approximately
> > zero. Statistics based on its inverse matrix cannot be computed and
> > they are displayed as system missing values".
>
> > What does this mean? Am I entering the data correctly?
>
> > Thank you
> > Aisha
Hi, many thanks for the response
arisai...@yahoo.com
Many thanks for the response.
I was trying a few times to response to you. However, the text did not
appear. I do not know why.
arisai...@yahoo.com
Items in the questionnaire: 23 items
Missing data: none
N=159
Construct: Knowledge about diabetes. I give 1 mark to a correct answer
(each item). Each item has multiple choices of answer ranging from
a,b,c,d and a,b,c. However, only 1 correct answer for each item. For
example;
1. Which of the following is highest in carbohydrate?
a. Chicken
b. Milk
c. Rice (correct answer)
d. Peanut butter
1 mark is given if the research participant answer 'C', and zero mark
will be given if they answer either a,b or d.
In the SPSS, I only enter either zero or one. In the Label, I code it
as '0=wrong and 1=correct'.
What do you mean by entering the data as it is? Is it, I need to code
the data as it is (in the questionnaire)? e.g. a,b,c,d and then give
the value either zero or one?
This scale has been used in previous researches.
The Alpha reading for research was low with many negative correlation.
Thank you
Aishairma
arisai...@yahoo.com
Rich Ulrich
wrote:
Your original message posted, which you have now posted several
more times. Both Bruce Weaver and I responded to the initial post.
From the headers of this message,
From: arisai...@yahoo.com
Newsgroups: comp.soft-sys.stat.spss
Subject: Re: Multiple-choice questions with one correct answer
Date: Fri, 5 Aug 2011 20:05:41 -0700 (PDT)
Organization: http://groups.google.com
Lines: 6
- It appears that users who read posts at Google are still
not seeing new posts. Or it is happening again. I heard about
that in another group, a few days ago. It seemed to take a
couple of days for anything to show up in the Google archive,
whether you read all your posts there or not.
Unless you are willing to wait a few days, or hope that this
is not going to be a regular thing, the best advice must be,
"Use a real news-reader" and a separate ISP.
--
Rich Ulrich
- also to be sent by e-mail to the OP.
Art Kendall
Are you _very_ sure that the scoring key was accurately applied to the
original data?
Are any of the corrected item-total correlations negative?
Are any of the interitem correlations exactly -1.00 or +1.00?
Are any of the squared multiple correlations 1.00?
Is there any pattern to the negative inter-item correlations e.g., many
on a row or or column?
Art Kendall
Social Research Consultants
xiuh...@gmail.com
Art Kendall
E.g., there is more than one answer that could be correct or you chose
the wrong response as correct etc.
I suggest that you enter the data exactly as the students answered.
Then you should either enter the data a second time and do a file
compare or have someone proofread it.
Then recode the data into 23 new variables.
Crosstab the raw responses by the keyed responses. Is the value you
chose as correct the most often chosen response? Is one of the
distractors highly chosen? If so, go back and read the substance of
the questions.
Before looking at the internal consistency, use the inter-item
correlations and corrected item total correlations to see if you recoded
the items correctly.
When you are sure the items were keyed correctly see if any items have a
pairwise correlation that is close to one.
One thing that can cause message is that some item(s) are (close to)
perfectly correlated with other items. Another is that there are not
enough cases. A zero determinant will happen when there are fewer cases
than there are items.
Is it possible that the items measure more than one underlying
construct? Examples might be knowing specific facts, and understanding
the process of science.
Art Kendall
Social Research Consultants
Rich Ulrich
I, for one, wish that you would ask your question without
posting it as a Reply to a question from years ago.
Or, if you use such a question, *do* say what is relevant
about it.
Yes. Cronbach alpha gives the internal reliability of a scale.
For a set of answers to a physics test, I expect that there
may be answers that nobody missed. Including those in the
list of items for Reliability will give a determinant of zero,
showing the matrix is not full rank.
For the matrix to be full rank, you also need to have more
test subjects than the number of items in the scale, for the
reasons that apply to multiple regression.
- In terms of "internal reliability" for a sample, an item with
zero variation contributes nothing; it does nothing to select
the high or low score for the total-correct.
--
Rich Ulrich
leekwan...@gmail.com
>
>
>
>
>
> Poster:
>
>
>
> I, for one, wish that you would ask your question without
>
> posting it as a Reply to a question from years ago.
>
> Or, if you use such a question, *do* say what is relevant
>
> about it.
>
>
>
> Yes. Cronbach alpha gives the internal reliability of a scale.
>
>
>
> For a set of answers to a physics test, I expect that there
>
> may be answers that nobody missed. Including those in the
>
> list of items for Reliability will give a determinant of zero,
>
> showing the matrix is not full rank.
>
>
>
> For the matrix to be full rank, you also need to have more
>
> test subjects than the number of items in the scale, for the
>
> reasons that apply to multiple regression.
>
>
>
> - In terms of "internal reliability" for a sample, an item with
>
> zero variation contributes nothing; it does nothing to select
>
> the high or low score for the total-correct.
>
>
>
> --
>
> Rich Ulrich
Rich Ulrich
wrote:
>how to check reliability of my multiple choices questions of physics exam. Since every answer is a, b,c, d that allocated randomly, that maybe the answer for the item 1 is c, item 2 is a and so on... can i use cronbach alpha to determine the reliability of my test paper?
>>
>Hi, i am also facing the same problem. i want to test reliability of my physics exam item. I want to test their previous knowledge about force, so , before i teach them, i let them answer 12 questions. So, how to determine the reliability of that 12 questions? Is it that i have to make correct answer as "1", wrong answer as "0",then key in to SPSS? Thank you.
students gave, and let the computer do the scoring. That eliminates
human errors of scoring, and gives you an unambiguous record
of what was what. You can also re-score immediately if you learn
that your key was wrong.
Here is one way to do the scoring.
DO REPEAT ans= var1 to var12/
key= 1,4,4,2, 3,2,1,4, 2,2,3,3 /
correct= good1 to good12 .
COMPUTE correct= (ans EQ key).
END REPEAT.
The COMPUTE statement creates a Bolean result, so that
the correct/wrong get coded as 1/0 respectively.
MISSING for an answer will result in MISSING in the
new variables, good1 to good12.
--
Rich Ulrich
dhenr...@gmail.com
Bruce Weaver
> On Friday, 11 April 2014 12:08:25 UTC-4, Rich Ulrich wrote:
>> The usual procedure is to make a file with the answers that the
>>
>> students gave, and let the computer do the scoring. That eliminates
>>
>> human errors of scoring, and gives you an unambiguous record
>>
>> of what was what. You can also re-score immediately if you learn
>>
>> that your key was wrong.
>>
>>
>>
>> Here is one way to do the scoring.
>>
>>
>>
>> DO REPEAT ans= var1 to var12/
>>
>> key= 1,4,4,2, 3,2,1,4, 2,2,3,3 /
>>
>> correct= good1 to good12 .
>>
>> COMPUTE correct= (ans EQ key).
>>
>> END REPEAT.
>>
>>
>>
>> The COMPUTE statement creates a Bolean result, so that
>>
>> the correct/wrong get coded as 1/0 respectively.
>>
>> MISSING for an answer will result in MISSING in the
>>
>> new variables, good1 to good12.
>>
>>
>>
>> --
>>
>> Rich Ulrich
>
> I am a beginner and have a similar problem, although I only need to score the questions, not determine reliability. Is there a menu option to do this, or is my best option to use the DO REPEAT code above?
>
DO-REPEAT method is your best bet.
