Theory & Measurement
Guenter Trendler
I’ve just finished reading Michell’s (2009) “Invalidity in Validity”
where he emphasizes again that: “what is really required are
instruments tailored to true theories about the psychological
processes involved in the attributes we wish to assess. In the absence
of such theories we cannot determine whether our attributes are
quantitative, no matter how well data fit our models.” (p. 120)
The importance of theory has also been emphasized in this forum (e.g.
by Andrew), but I must confess that I never really understood the
point. Can someone please explain? If I think for instance about
temperature measurement by means of gas thermometry (= instruments)
not much theory is needed. In analogy, what are the true theories
about the physical processes involved in temperature measurement?
Thanks!
Guenter
Michell, J., Invalidity in Validity, in R.W. Lissitz (Ed.), The
Concept of Validity, 2009, IAP, Charlotte, NC.
Denny Borsboom
define ambient temperature as (a suitable function of) average kinetic
energy of particles in a given space; particles in a fixed column
(i.e., a thermometer setup, so say mercury) will align their energy to
those in the ambient space (thermal equilibrium); the mercury
expands/contracts as a result; hence the indicated level on the
thermometer rises or falls depending on the difference between their
energy and that of the ambient space; so the process instantiates a
functional relation between average kinetic energy and mercury ticks.
This is why you can say that the measurement instrument (here the
thermometer) indeed measures the attribute referred to by the
theoretical term "temperature" (average kinetic energy).
At least this is how it seems to me (Joel never indicated he disagrees).
Best
Denny
> --
> You received this message because you are subscribed to the Google Groups
> "Talking Measurement" group.
> To post to this group, send email to talking-m...@googlegroups.com.
> To unsubscribe from this group, send email to
> talking-measure...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/talking-measurement?hl=en.
>
>
--
Denny Borsboom
Department of Psychology
University of Amsterdam
Roetersstraat 15
1018 WB Amsterdam
The Netherlands
phone: +31 20 525 6882
email: d.bor...@uva.nl
homepage: http://users.fmg.uva.nl/dborsboom
Nick Connolly
that some understanding of the underlying physical theory is
neccesary.
A simple example is that the basic arithemtic mean is not suitable for
finding an 'average' temperature (except in a very restricted
circumstance) precisely because of what a 'temperature' is.
For example if you follow any of the on-going discussions about
climate-science and global warming then you may have come across
assorted arguments about the validity of the methods used to establish
the extent of changes in global average surface temperature and even
some challenges to the validity of the notion of such a thing.
[Perhaps not wise to bring up climate science and its assorted
controversies but what-the-heck :) ]
Trendler, Guenter
Does Michell himself use this example somewhere? Let me start with a quote from Schooley (1986):
"Throughout the history of scientific thermometry, there has been no other method for thermodynamic temperature determination to compare in effectiveness with gas thermometry. All of the international temperature scales have been based upon thermodynamic measurements of temperature by gas thermometry. The ideal gas law
Pv = nRT (I)
is the basis for gas thermometry . After the reciprocal relationship between pressure and volume in an isothermal gas was elucidated by Boyle and his contemporaries and the temperature dependence of the volume of a gas at constant pressure was shown to be essentially linear by Charles, Gay-Lussac, and others, the ideal gas law was given a firm theoretical basis by the development of the sciences of thermodynamics and statistical mechanics."(p. 115)
As Schooley explains the gas laws were discovered previous to thermodynamics and statistical mechanics. Hence these sophisticated theories were not necessary for the discovery of the measurability of temperature. Furthermore, since no theory about underlying physical processes (e.g. the molecular behaviour of gas particles) was involved the instruments were not tailored to any such processes. Maybe Michell means something else?
How would such theory look with regard to Rasch models in their simplest form, i.e. response as a function of person ability and item difficulty? Do we have to go into postulating certain brain structures etc.?
Regards,
Guenter
Schooley, James F., Thermometry, CRC Press, Boca Raton, Florida (1986).
________________________________
Trendler, Guenter
But the topic of climatic change could be used as an example of another branch of science plagued by pathologies (1). :)
Best,
Guenter
(1) http://www.spiegel.de/international/world/0,1518,686697,00.html
________________________________
--
Denny Borsboom
I can't speak for Joel; but you should surely ask him.
In case of the Rasch/2PL model, the only known candidate for filling
the process level void is the general diffusion model for two-choice
decisions. The source here is: Tuerlinckx, F., & De Boeck, P. (2005).
Two interpretations of the discrimination parameter. Psychometrika,
70(4), 629-650.
Roughly this paper says that if subjects use an evidence accumulation
process well described by the Ratcliff diffusion model to answer the
items, then you'll find a 2PL to hold in the data; the IRT parameters
then translate neatly into the process level parameters of the
diffusion model, with ability-difficulty equal to diffusion drift
rate, and discrimination equal to diffusion boundary separation. By
the way, I think that this is the most important psychometric paper to
appear in years (it has received a total of 4 citations, all of which
are by me ;-). Although don't really see why this is relevant, you may
indeed search for a neural basis of the relevant processes if you like
(e.g., see Forstmann, B. U., Anwander, A., Schäfer, A., Neumann, J.,
Brown, S., Wagenmakers, E.-J.,Bogacz, R., & Turner, R. (2010).
Cortico-striatal connections predict control over speed and accuracy
in perceptual decision making. Proceedings of the National Academy of
Sciences, 107, 15916-15920.)
With regard to the temperature example: clearly you don't need to
understand what a thermometer does at a process level to use it; as
long as you have the phenomenal laws like the ideal gas laws you'll
get by fine. Just like you don't need to understand what distance is
or how a tape measure works (pretty complicated if you consider the
details) in order to use it as a measurement instrument. But if you
want to understand *why* a thermometer in fact measures temperature
(i.e., address the validity problem), a theory that deals with the
process level (i.e., that describes how the measurement instrument
works) is quite helpful - in my personal view, essential.
Best
Denny
Denny Borsboom
Stephen Humphry
Steve
________________________________
From: talking-m...@googlegroups.com [talking-m...@googlegroups.com] On Behalf Of Trendler, Guenter [guenter....@zi-mannheim.de]
Sent: Wednesday, 13 October 2010 5:22 PM
To: talking-m...@googlegroups.com
Subject: AW: [talking-measurement] Theory & Measurement
Denny Borsboom
not entirely sure what happened here... Decoded it probably means something ;-)
denny
Trendler, Guenter
D. With regard to the temperature example: clearly you don't need to understand what a thermometer does at a process level to use it; as long as you have the phenomenal laws like the ideal gas laws you'll get by fine. Just like you don't need to understand what distance is or how a tape measure works (pretty complicated if you consider the details) in order to use it as a measurement instrument. But if you want to understand *why* a thermometer in fact measures temperature (i.e., address the validity problem), a theory that deals with the process level (i.e., that describes how the measurement instrument works) is quite helpful - in my personal view, essential.
S. I haven't tuned into the whole conversation here but a comment. The ideal gas law is fine for measuring the temperature of gases; it's not much use for measuring the temperature of solids. Given this, the application of measurable temperature is not nearly so general. Nevertheless, the ideal gas law is theory in what is effectively the simplest case that leads into the more general theory (statistical mechanics) upon which the measurement of temperature is based.
G. Well, this is all true (though, Denny, I don't think it's possible to use a tape measure without understanding what distance means), but does it hit the point? Michell seems to be saying that in absence of theories about underlying processes (e.g. kinetic theory) it is not possible to determine if an attribute is quantitative. To quote him again: "In the absence of such theories we cannot determine whether our attributes are quantitative" (op. cit., p. 120). My question: why is not possible? In my view the case of temperature measurement demonstrates the contrary. Or, are theories on the phenomenal level (e.g. gas laws) sufficient? (Of course, I could ask Joel directly, and probably I will, but people here also expressed this view and maybe they can also help elucidate the matter.)
Guenter
PS: Thanks for the references Denny!
________________________________
Trendler, Guenter
not entirely sure what happened here... Decoded it probably means something ;-)
denny
Ahaaa!...coded messages, so there is a conspiracy after all... ;))
Denny Borsboom
I didn't say "what distance means" but "what distance is".
best
denny
On 10/13/10, Trendler, Guenter <guenter....@zi-mannheim.de> wrote:
Tom Bramley
Dear Guenter,
I think that Michell's views that are relevant to your question are expressed on p206-7 (see quote below) of his 1999 book Measurement in Psychology (and doubtless in other places too).
"In this instance [Spearman's 2-factor theory of ability] there are 3 further requirements necessary to apply conjoint measurement theory: a theory of problem solving capable of distinguishing homogenous from non-homogenous tests; some way of identifying values of general ability that is independent of test scores, some way of identifying values of specific ability, also independently of test scores, and …identifying and controlling other relevant causes…".
Michell elsewhere (1990 p69) gives the example of performance on a task (like a rat running a maze) being a function of motivation and prior practice, again with the requirement that different levels of these attributes can be identified. He suggests food or water deprivation could be used to identify levels of motivation, and number of previous practice trials could identify levels of prior practice.
It is this idea of independently identifying levels of an attribute that is (to me) somewhat confusing. Mike Maraun (1998) has argued that whether something is measureable is 'internally related' to the concept itself. For example, to understand what length means is to understand what it is to measure length. Psychological concepts like 'motivation' on the other hand do not involve measurement concepts as part of their definition. They have what Maraun calls 'complex grammars', in contrast to the clean, simple, necessary/sufficient grammars of physical properties like length. Using levels of food or water deprivation is in my view an operational definition of motivation for the purpose of a particular experimental study. I'm not quite sure what identifying 'real' levels of motivation would involve, but this seems to be what Michell requires of psychological theories.
Regards,
Tom.
Maraun, M.D. (1998). Measurement as a normative practice: implications of Wittgenstein's philosophy for measurement in psychology. Theory & Psychology, 8(4), 435-461.
Michell, J. (1990). An introduction to the logic of psychological measurement. Hillsdale, New Jersey: Lawrence Erlbaum Associates.
Michell, J. (1999). Measurement in psychology: A critical history of a methodological concept. Cambridge: Cambridge University Press
From: talking-m...@googlegroups.com [mailto:talking-m...@googlegroups.com] On Behalf Of Trendler, Guenter
Sent: 13 October 2010 13:18
If you are not the intended recipient, employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination or copying of this communication and its attachments is strictly prohibited. If you have received this communication and its attachments in error, please return the original message and attachments to the sender using the reply facility on e-mail. Internet communications are not secure and therefore Cambridge Assessment (the brand name for the University of Cambridge Local Examinations Syndicate, the constituent elements of which are CIE, ESOL and OCR [Oxford Cambridge and RSA Examinations is a Company Limited by Guarantee Registered in England. Registered office: 1 Hills Road, Cambridge CB1 2EU. Company number: 3484466]) does not accept legal responsibility for the contents of this message. Any views or opinions presented are solely those of the author and do not necessarily represent those of Cambridge Assessment unless otherwise specifically stated. The information contained in this email may be subject to public disclosure under the Freedom of Information Act 2000. Unless the information is legally exempt from disclosure, the confidentiality of this email and your reply cannot be guaranteed.
This message has been scanned for viruses by BlackSpider MailControl
Nick Connolly
to use a tape measure without understanding what distance means" as it
has some relevance. Clealry you can use a tape measure without
understanding the physics of length but you can't use a tape measure
without some intutive understanding of length.
With temperature, going back to the issue of combining temperatures,
give school age students a numerical problem involving two jugs of
water of different temps being put in the same container you'll get,
on the whole, nonsensical answers (eg a jug of 10 C and a jug of 40 C
making a combined temp of 50 C). Ask a similar question in terms of
more vague descriptors (eg a jug of cold water and a jug of warm
water) you get physically correct answers (luke warm water).
Stephen Humphry
Hi Tom. You said:
It is this idea of independently identifying levels of an attribute that is (to me) somewhat confusing. Mike Maraun (1998) has argued that whether something is measureable is 'internally related' to the concept itself. For example, to understand what length means is to understand what it is to measure length.
I disagree with this. People would have understood what length is just fine before they ever devised even crude procedures for measuring it. We have to understand length to negotiate our surroundings and to do just about anything. One way to put it is that spatial extension, direction, and length in a direction exist whether or not we devise procedures to measure it.
Psychological concepts like 'motivation' on the other hand do not involve measurement concepts as part of their definition. They have what Maraun calls 'complex grammars', in contrast to the clean, simple, necessary/sufficient grammars of physical properties like length. Using levels of food or water deprivation is in my view an operational definition of motivation for the purpose of a particular experimental study. I'm not quite sure what identifying 'real' levels of motivation would involve, but this seems to be what Michell requires of psychological theories.
What are the clean, simple, necessary/sufficient ‘grammars’ of length?
Tom Bramley
Hi Steve,
I've only recently come across Maraun's work, so I would rather point people towards it than try to expound it myself. I don't find it easy to understand, but I think that the members of this group would find a lot to agree with, and find some things provocative. His main targets are latent variable theory (especially factor analysis) and construct validation theory. Like Michell, he is critical of the enterprise of psychometrics, but comes from a different angle. Chapter 9 (Latent variable interpretation)
in the book on his website
http://www.sfu.ca/~maraun/Mikes%20page-%20Myths%20and%20Confusions.html ends with a brief comment about representational (axiomatic) measurement, and this is preceded by an entertaining demolition of an account of the 'g' of general intelligence. A more sanitised version is given in his 2009 paper cited below.Just out of curiosity, I typed 'length' and then 'motivation' into the search box on
www.dictionary.com The first two definitions of 'length' did indeed have measure as part of their definition, and the list of so-called synonyms included 'magnitude' and 'quantity'. The same was not the case for 'motivation'!Regards,
Tom.
Maraun, M.D., Slaney, K.L., & Gabriel, S.M. (2009). The Augustinian methodological family of psychology. New Ideas in Psychology, 27, 148-162.
From: talking-m...@googlegroups.com [mailto:talking-m...@googlegroups.com] On Behalf Of Stephen Humphry
Sent: 14 October 2010 03:04
To: talking-m...@googlegroups.com
Subject: RE: [talking-measurement] Theory & Measurement
Click here to report this email as spam.
Trendler, Guenter
Thanks for referring me to those passages! Mentioning Spearman reminded me of a section in Michell 1990 book which reads:
"An example of what I mean is Hunt's (1980) suggestion that Spearman's g may be identified with attentional resources, and his s with problem solving strategies. If each of these variables can be experimentally manipulated it may be possible to test Spearman's theory. Of course, the value of such a test would be contingent on the truth of Hunt's suggestion, and that may be false. However, this kind of approach is the only one that is going to show whether or not abilities are quantities and how they combine to determine performance. Progress can only be made in this area of psychology by linking the hypothetical quantitative variables of the factor analytic theories (the abilities), to experimentally controllable variables. This requires bold hypotheses similar to Hunt's and while such hypotheses may be false, it is better to show an hypothesis to be false than not to be able to test it at all." (p. 156)
If "bold hypotheses similar to Hunt's" is what Michell means by the necessity of theories about underlying processes then this basically answers my question. Of course I agree: if we want progress we must find a way to experimentally control the independent (usually called latent or hypothetical) variables and not just register the observable (or manifest variables). But ultimately we will have to identify the material substrate of psychological variables (e.g. of what Hunt calls attentional resources and problem solving strategies) in order to manipulate and/or control them.
I do not quite follow Maraun. Everyday physical concepts also have a complex grammar. The grammar of physical concepts is simplified in experiment where complexity is controlled. However, to return to the Rasch model: in order to test it we will have to identify the material substrate of ability. The only workable route is in my view to find out if certain items always require the same ability by observing if always the same brain region is active. If this is the case, we can assume to have identified the material substrate. Then we will have to find a way to identify equal levels of ability (e.g. volume of brain mass active in that region). We will have to verify if the same item always is associative with the same level of ability per person, etc. The same applies in my view to Michell's example you mention. This is all quite venturous, I know...but I fear this is the only way...
Regards,
Guenter
________________________________
Von: talking-m...@googlegroups.com im Auftrag von Tom Bramley
Gesendet: Mi 13.10.2010 18:09
An: 'talking-m...@googlegroups.com'
Betreff: RE: [talking-measurement] Theory & Measurement
Dear Guenter,
I think that Michell's views that are relevant to your question are expressed on p206-7 (see quote below) of his 1999 book Measurement in Psychology (and doubtless in other places too).
"In this instance [Spearman's 2-factor theory of ability] there are 3 further requirements necessary to apply conjoint measurement theory: a theory of problem solving capable of distinguishing homogenous from non-homogenous tests; some way of identifying values of general ability that is independent of test scores, some way of identifying values of specific ability, also independently of test scores, and ...identifying and controlling other relevant causes...".
Regards,
Tom.
________________________________
Guenter
________________________________
Steve
Hi Denny,
Regards,
Guenter
Hi Guenter,
Best
Denny
This message has been scanned for viruses by BlackSpider MailControl <http://www.blackspider.com/>
Trendler, Guenter
________________________________
Von: talking-m...@googlegroups.com im Auftrag von Stephen Humphry
Gesendet: Do 14.10.2010 04:04
An: talking-m...@googlegroups.com
Betreff: RE: [talking-measurement] Theory & Measurement
Hi Tom. You said:
It is this idea of independently identifying levels of an attribute that is (to me) somewhat confusing. Mike Maraun (1998) has argued that whether something is measureable is 'internally related' to the concept itself. For example, to understand what length means is to understand what it is to measure length.
S. I disagree with this. People would have understood what length is just fine before they ever devised even crude procedures for measuring it. We have to understand length to negotiate our surroundings and to do just about anything. One way to put it is that spatial extension, direction, and length in a direction
G. Also, consider this: to understand what temperature means is to understand what it is to measure temperature. Hence, prior to the establishment of temperature measurement the notion of temperature was meaningless?
Denny Borsboom
I have never understood the linguistic analysis that much of Maraun's
work presumes. I am somewhat heartened by the fact that others too
apparently find it difficult to comprehend. However it would be much
more interesting if somebody could in fact explain the position in
terms that ordinary non-Wittgensteinians could also understand.
Anyone?
Best
Denny