fundamentals before numbers
Dan
physical sciences seem to have utilized extensively, is qualitative
analysis. Before we ever figured out a measurement for something, we
tend to describe it in detail, examine its intrinsic properties, look
for ways that it can be measured, then test it against observable
measures. If something has no consistent length or width, we cannot
apply standard forms of measurement against it. Putting a ruler to a
puddle of water on a table-top, for example will yield inaccurate
measures as the puddle moves and changes shape. You would need to
understand how to measure a volume of water. Dr. Kyngdon stated
"Systems of derived measurements, as far as I can see, require base
quantities." We have not sufficiently found, defined, or qualified the
base quantities of psychology. I agree that in psychology we have no
analogs to length, angle, mass, etc., largely due to our application
of S.S. Stevens' operationalistic work-around.
We ignore many of the intrinsic properties of a psychological
phenomenon and take what seems to fit our pet theories, assume
measurement, construct a scale, test the scale and call it good.
Meanwhile everyone else is doing the same thing so we wind up with
multiple scales of measurement for things like intelligence, or
personality. In the end, when the scales do not equate to one another,
we account for the differences by pointing out that the kind of
intelligence that you measured is different from the intelligence that
I measured. It all comes down to a semantic exercise to excuse the
fact that we are really measuring someone's idea, or construct of what
matters and not the thing itself. There is little agreement on what
constitutes intelligence, personality, attitude, and so on. These
concepts are complex and likely based on smaller base operations. What
things make up intelligence and what things make up the things that
constitute intelligence? We don't know in any real sense because we
haven't taken the time to look thoroughly for all of the base drivers
for the construct in a coherent, sequential, constructive manner. My
thought is that we need to launch from the biological underpinnings
and then sequentially out to the higher-order constructs in order for
anything to make sense. I think that cognitive neuroscience, and a
stronger inter-disciplinary focus is the next wave.
~ Dan
Andrew Kyngdon
Welcome to Talking Measurement. Our forum has been quiet for a few months
now; and a new contribution is timely.
I really couldn't agree with you more in what you say. I have recently had
a paper accepted by Theory & Psychology where I have made arguments
consistent with what you conclude - that psychometrics needs to focus on
developing descriptive, behavioural theories of individual differences in
cognitive ability. This will mean identifying what cognitive processes are
involved in responding to test items; and how these relate to identifiable
features or components of the items themselves. Once we have such a
theory, we might be in a position to write an equation that predicts item
difficulty without any regard to empirical data.
So far I only have a very small list of things that could be tentatively
considered "base psychological quantities". The first one is physical -
time. It is difficult to see how human behaviour could occur at all
without time. Another is a simple count - working memory capacity -
although the problem here is that the "chunk" unit (Miller, 1956) is
ambiguous to say the least. Another count is vocabulary size, as word
frequency in continuous prose text is quite prominent in a lot of
cognitive studies of reading behaviour, especially the eye fixation
research conducted by Keith Rayner. Another base quantity would be media
of exchange (fiat money). Money used to be a simple count but now appears
to be continuous, as our banks now tell us that we can have an amount of
money equal to USD$1000.7333 in our accounts, for example.
I am currently working on a paper which uses quantity calculus of physics
to define the utility of risky incremental gains and losses as a derived
quantity - an amount of probabilistic risk attitude per unit of money. I
did this in the context of cumulative prospect theory (Tversky & Kahneman,
1992) to define a putative unit of measurement for utility.
Cheers,
Andrew
Andrew Kyngdon, Ph.D
MetaMetrics, Inc.
Associate Professor
Graduate School of Education
University of Western Australia
You received this message because you are subscribed to the Google Groups
"Talking Measurement" group.
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to
talking-measure...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/talking-measurement?hl=en.
Dan
It is funny that you bring up time when even physicists have difficulty defining time. Are you talking about actual time or perception of time? Time itself appears to change depending on gravity, speed and perhaps dimension. Perception of time seems far more elastic. Sorry, a bit of an aside. I imagine you are referring to common time measured in seconds, minutes, hours, etc. and I agree that our behavior seems to function within the context some sort of sense of time.
The only truly quantitative measure in psychology that I was aware of is Weber's law concerning just noticeable differences. I find it disappointing that we only have one well known major quantifiable theory in the entire field and it is quite old. I think we could build quite a bit by integrating the findings from psychophysics and perception studies in a bottom-up manner, adding qualitative work along the way. The field of psychology has needed integration for a long time.
I am also interested in decision making strategies, but I am more interested in sub-optimal real time decision making under natural conditions. Most people are horrible at thinking in terms of probabilities and optimization. My questions are "what is really going on when we choose?", and "is there a way to learn to improve natural decision making?" I am also interested in using fuzzy logic based models to examine utility.
My other interests are incorporating chaos and catastrophe theory into examining and describing psychological phenomena. I think we are a few decades away from finding anything meaningful in this regard, but it is interesting to think of the possibilities.
Thanks,
Dan
To post to this group, send email to talking-measurement@googlegroups.com.
To unsubscribe from this group, send email to
talking-measurement+unsub...@googlegroups.com.
Stephen Humphry
Hi Dan,
Welcome to the group. I’m not going to have time to take part for a little while, but just want to briefly comment on something you said. I mostly agree with what you have said. I do not follow in what sense Weber’s law is a truly quantitative measure. Weber’s law concerns the comparisons of physical magnitudes. It is debatable whether any ‘sensory magnitudes’ are involved or indeed even necessary to state the “law”. Perhaps more on this another time. I certainly agree that this is one of the best shots for getting a handle on how to approach psychology quantitatively.
Where it comes to time, I think it will be most profitable to try to use real/actual time. To me the key thing is that time is required for physiological and psychological processes. All modern definitions of physical quantities (base or derived) refer explicitly or implicitly to physical relations between two or more physical kinds-of-quantity. A reasonable question is whether we can identify psychological quantities that are related to time in an analogous fashion, which could give a connection between quantities that we refer to as physical and quantities we refer to as psychological.
Regards, Stteve
Stephen Humphry | Associate Professor
Graduate School of Education
The University of Western Australia
M428, 35 Stirling Highway, Crawley, WA, 6009
Telephone: +61 8 6488 7008
Fax: +61 8 6488 1052
www.gse.uwa.edu.au
CRICOS Code: 00126G
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/talking-measurement?hl=en.
--
You received this message because you are subscribed to the Google Groups "Talking Measurement" group.
To view this discussion on the web visit https://groups.google.com/d/msg/talking-measurement/-/NxjYvx3A9d4J.
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to talking-measure...@googlegroups.com.
Andrew Kyngdon
Dan,
I was referring to actual physical time. As Steve remarked, behaviour and cognitive processes take time to happen. Of course, it is entirely possible that the human perception of time may involve some kind of weighting. But the circumstances under which this may occur, or what specific behaviours such weighting is relevant to, have not been identified. Or at least I’m not aware of any relevant literature.
I am not sure as to what you mean by “sub-optimal real time decision making under natural conditions”. My interest lies in decision making under conditions of risk and uncertainty. I am not that interested in riskless choice. You state that most people are horrible in terms of thinking about probabilities. I would add that they do not seem to make decisions on the basis of probabilities, but the subjective weighting of event probabilities (at least under conditions of risk). These “decision weights”, which are an inverse-S shape function of event probabilities, are at the core of the rank dependent utility theories such as cumulative prospect theory. These decision weights rather accurately predict risk attitude, can account for paradoxical choice behaviour such as the Allais (1953) common consequence and common ratio effects, and can explain why people purchase both insurance policies and lottery tickets. Decision makers attach far greater weight to a change in event probability from certainty (1) to .99 far more than a commensurate change between probabilities (such as between .84 and .85). Likewise from impossibility (0) to .01. This seems to be a fundamental process of risky decision making, as the phenomenon has been quite experimentally robust over the past three decades or so of research.
Chaos theory seems to come in and out of fashion within psychology, but no real progress ever seems to come from it.
Cheers,
Andrew
From: talking-m...@googlegroups.com [mailto:talking-m...@googlegroups.com] On Behalf Of Dan
Sent: Tuesday, 17 July 2012 1:53 AM
To: talking-m...@googlegroups.com
Subject: Re: [talking-measurement] fundamentals before numbers
Andrew,
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/talking-measurement?hl=en.
--
You received this message because you are subscribed to the Google Groups "Talking Measurement" group.
To view this discussion on the web visit https://groups.google.com/d/msg/talking-measurement/-/NxjYvx3A9d4J.
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to talking-measure...@googlegroups.com.
Dan
I was referring to decision field theory when talking about sub-optimal decision making. Decision field theory deals with how people actually make decisions rather than how we should make (optimal) decisions. The mathematics are based on a probability diffusion process, using a continuous-time Markov chain. There is a growing line of research by Busemeyer and others regarding decision field theory (Busemeyer & Townsend, 1993). I like how DFT accounts for person-in-context effects and affect as a part of decision making. I am also more familiar with linear/matrix algebra so it makes more sense to me when put in terms of a Markov chain. I have to admit that I am still learning the ropes and that I have a math deficit that I am working hard to correct.
Part of the problem of applying chaos theory to psychology is that it requires real math skill, not just a rudimentary understanding of statistics. Math competency is something sorely lacking in psychology. The ground work has never been put in place to incorporate chaos theory into psychology in the first place in terms of quantitative theory building and base functions. There are some nice theoretical ideas around chaos theory and how it could apply to psychology, but few have attempted to actually do the math to prove anything. The majority of chaos theory is built around nonlinear dynamics, which requires understanding a great deal of advanced calculus. Most psychologists in the U.S. seem to struggle with getting through graduate statistics, let alone trying to grasp partial differential equations. My personal belief is that the whole field of psychology is suffering from a lack of mathematical understanding that permeates everything. We should have the same basic math skills as psychological scientists as a physicist, or biologist, or any other natural scientist if we ever hope to shore up the inconsistencies in psychology.
There is a small sense of growing urgency here because most of the quantitative psychologists are either retired, retiring soon, or holding on through emeritus status, and there are very few programs producing quantitative psychologists to replace those we are losing. The field of psychology needs more quantitative folks to deal with the complexity of the phenomena in psychology and we produce fewer each year. Makes me wonder about the future of the field.
Thanks,
Dan
Busemeyer, J. R., & Townsend, J. T. (1993) Decision Field Theory: A dynamic cognition approach to decision making. Psychological Review, 100, 432-459.
To post to this group, send email to talking-measurement@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/talking-measurement?hl=en.
--
You received this message because you are subscribed to the Google Groups "Talking Measurement" group.
To view this discussion on the web visit https://groups.google.com/d/msg/talking-measurement/-/NxjYvx3A9d4J.
To post to this group, send email to talking-measurement@googlegroups.com.
To unsubscribe from this group, send email to talking-measurement+unsub...@googlegroups.com.
Dan
Thanks for the welcome, and the response. I tend to think of the Weber-Fechner law and its many extension as a quantitative measure. Weber was relating stimuli magnitude to just noticeable differences between levels of the stimulus; describing the relationships mathematically. The law has held across a number of contexts, although not all. The point being that Weber got it right and started from the ground-up.
It would be nice to see more psychological phenomena tied to physically observable events and measures. Relating psychological phenomena to time would be tricky though. There are so many potential confounds. There is a whole host of reaction time experiments that show just how variable time-response can be even under controlled conditions. We also have to be careful in our inferences resulting from the correlation of time to psychological event. I think of physicists working on finding sub-atomic particles. They formulate hypotheses and make observations, narrowing the possibilities as they go until they have excluded any other explanation beyond a reasonable doubt. The level of reasonable doubt is far "tighter" in physics, mostly because they are dealing with things that are detectible or directly observable. I would like to see psychology tighten up in a similar manner. I think time is only one possible way of attacking the problem.
Thanks,
Dan
To post to this group, send email to talking-measurement@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/talking-measurement?hl=en.
--
You received this message because you are subscribed to the Google Groups "Talking Measurement" group.
To view this discussion on the web visit https://groups.google.com/d/msg/talking-measurement/-/NxjYvx3A9d4J.
To post to this group, send email to talking-measurement@googlegroups.com.
To unsubscribe from this group, send email to talking-measurement+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/talking-measurement?hl=en.
To post to this group, send email to talking-measurement@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/talking-measurement?hl=en.
--
You received this message because you are subscribed to the Google Groups "Talking Measurement" group.
To view this discussion on the web visit https://groups.google.com/d/msg/talking-measurement/-/NxjYvx3A9d4J.
To post to this group, send email to talking-measurement@googlegroups.com.
To unsubscribe from this group, send email to talking-measurement+unsub...@googlegroups.com.
Paul Barrett
Hmmm ... I think a close reading of:
Schonemann, P. (1994) Measurement: the Reasonable Ineffectiveness of Mathematics in the Social Sciences. In I. Borg and P.Mohler (Eds.). Trends and Perspectives in Empirical Social Research. Walter de Gruyter ... might reduce the energetic enthusiasm for ‘more mathematics’ in the social sciences ..
I’m happy to forward a copy (pfd) to anyone ...
Two other articles very relevant here – just out ...
Markus, K., & Borsboom, D. (2012) The cat came back: Evaluating arguments against psychological measurement. Theory and Psychology, 22, 4, 452-466.
Abstract
The possibility or impossibility of quantitative measurement in psychology has important ramifications for the nature of psychology as a discipline. Trendler’s (2009) argument for the impossibility of psychological measurement suggests a general and potentially fruitful strategy for further research on this question. However, the specific argument offered by Trendler appears flawed in several respects. It seems to conflate what must hold true with what one must know and also equivocate on the necessary evidence. Moreover, if the argument supported its conclusion, it would rule out qualitative discourse on psychology as well as psychological measurement. Taking Trendler’s argument as an example, one can formulate a general structure to arguments adopting the same basic strategy. An overview of the requirements that such arguments should meet provides a metatheoretical perspective that can assist authors in constructing such arguments and readers in critically evaluating them.
This one is perhaps the most relevant to the current conversation ..
Saint-Mont, U. (2012) What measurement is all about. Theory and Psychology, 22, 4, 467-485.
Abstract
The nature of psychological measurement is still the subject of fierce controversy. A rather philosophical debate has been going on in this journal; therefore a closer look at physicists’ ideas on measurement may be helpful. In particular, we will try to clarify matters with the help of the crucial concepts of access (validity), precision (reliability), and invariance.
There is a place for mathematics in many areas of psychological investigation, but I think this comes AFTER careful observations and some basic (even if limited) understanding gained of a phenomenon and its occurrence. Not least how it may be replicated in order to yield very nearly the same observations (even if just humble counts, or two-class-membership frequencies).
I’m not a fan of statistical aggregates/inferential and probability models because these are excellent ways of hiding empirical anomalies as ‘unreliability’ and ‘measurement error’. But, they do have pragmatic value and can perhaps suggest areas for more careful observational work. I think we just have to be very careful about the why we adopt precise mathematical models which avoid explaining individual, single-person behaviors in favour of asserting some statistically-generated mathematical entity for which there is no evidence any individual actually ‘possesses’ it (as specified).
These days I just use OOM (Grice, J. (2011) Observation Oriented Modeling: Analysis of cause in the behavioral sciences. New York: Academic Press. ISBN: 978-012-385194-9, and Grice, J.W., Barrett, P.T., Schlimgen, L.A., & Abramson, C.I. (2012) Toward a brighter future for psychology as an observation oriented science. Behavioral Sciences, 2, , 1-22 .. http://www.mdpi.com/2076-328X/2/1/1 )
Or in many cases, I’ll work with data models/analyses which may be custom hybrids of arithmetic/quantitative math + non-quantitative production-rules, with verification using cross-validated predictive accuracy of specific observational outcomes.
And look within some areas of economics/politics and prediction modeling via regression models .. a move away from ‘precise beta-weights’ as these proves to be less accurate than using simple ‘index’ weighting systems ...
Armstrong, J.S., Graefe, A. (2010) Predicting elections from biographical information about candidates. Presented at the Symposium on Leadership and Individual Differences, Lausanne, Switzerland, November 30 - December 1, 2009, , , 1-20.
Soyer, E., Hogarth, R.M. (2012) The illusion of predictability: How regression statistics mislead experts. International Journal of Forecasting (In Press), , , 1-39.
Armstrong, J.S. (2012) Illusions in regression analysis. International Journal of Forecasting (In Press), , , 1-10.
There is also something very profound in Joel Michell’s latest article:
Michell, J. (2012-in press). Alfred Binet and the concept of heterogeneous orders. Frontiers in Quantitative Psychology and Measurement. Available free-to-download online at:
http://www.frontiersin.org/Quantitative_Psychology_and_Measurement/10.3389/fpsyg.2012.00261/abstract
In a comment, hitherto unremarked upon, Alfred Binet, well known for constructing the first intelligence scale, claimed that his scale did not measure intelligence, but only enabled classification with respect to a hierarchy of intellectual qualities. Attempting to understand the reasoning behind this comment leads to an historical excursion, beginning with the ancient mathematician, Euclid and ending with the modern French philosopher, Henri Bergson. As Euclid explained (Heath, 1908), magnitudes constituting a given quantitative attribute are all of the same kind (i.e., homogeneous), but his criterion covered only extensive magnitudes. Duns Scotus (Cross, 1998) included intensive magnitudes by considering differences, which raised the possibility (later considered by Kant (Sutherland, 2004)) of ordered attributes with heterogeneous differences between degrees (“heterogeneous orders”). Of necessity, such attributes are non-measurable. Subsequently, this became a basis for the “quantity objection” to psychological measurement, as developed first by Tannery (1875) and then by Bergson (1889). It follows that for attributes investigated in science, there are three structural possibilities:
(1) classificatory attributes (with heterogeneous differences between categories);
(2) heterogeneous orders (with heterogeneous differences between degrees); and
(3) quantitative attributes (with thoroughly homogeneous differences between magnitudes).
Measurement is possible only with attributes of kind (3) and, as far as we know, psychological attributes are exclusively of kinds (1) or (2). However, contrary to the known facts, psychometricians, for their own special reasons insist that test scores provide measurements.
Don’t get me wrong Dan, this is not a call to ignore mathematics or axiomatization, but as Schonemann and Saint-Mont argue, I would argue that it comes last, not first, in a sequence of empirical investigation – exactly as it did for physics. And, there may be some areas in psychology (such as decision-theory) where sufficient reliable observations about phenomena exist, and can now be more fruitfully modeled and tested via explanatory mathematic models.
And it is here I am still impressed with Andrew’s exposition on the lexile .. because that is how the math was eventually applied, to robust observations for which a theory seemed to apply which could sustain a quantitative (mathematical) model which could subsequently be tested against new ‘manipulation’ experiments and observations.
I must say I also like Borsboom and colleague’s work with network theory .. as applied to symptom comorbidity ... again, a different approach altogether which avoids ‘quantity’ measurement assumptions’. They are now applying this modeling approach to personality attributes.
Borsboom, D., Cramer, A.O.J., Schmittman, V.D., Espkamp, S., Waldorp, L.J. (2011) The small world of psychopathology. PLoS One (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0027407 ), 6, 11, 1-11.
Background: Mental disorders are highly comorbid: people having one disorder are likely to have another as well. We explain empirical comorbidity patterns based on a network model of psychiatric symptoms, derived from an analysis of symptom overlap in the Diagnostic and Statistical Manual of Mental Disorders-IV (DSM-IV).
Principal Findings: We show that
a) half of the symptoms in the DSM-IV network are connected,
b) the architecture of these
connections conforms to a small world structure, featuring a high degree of clustering but a short average path length, and
c) distances between disorders in this structure predict empirical comorbidity rates. Network simulations of Major Depressive Episode and Generalized Anxiety Disorder show that the model faithfully reproduces empirical population statistics for these disorders.
Conclusions: In the network model, mental disorders are inherently complex. This explains the limited successes of genetic, neuroscientific, and etiological approaches to unravel their causes. We outline a psychosystems approach to investigate the structure and dynamics of mental disorders.
Anyway, just my ‘take’ on some issues ..
Regards .. Paul
Advanced Projects R&D Ltd.
__________________________________________________________________________________
From: talking-m...@googlegroups.com [mailto:talking-m...@googlegroups.com] On Behalf Of Dan
Sent: Thursday, July 19, 2012 6:14 AM
To: talking-m...@googlegroups.com
Subject: Re: [talking-measurement] fundamentals before numbers
Andrew,
I was referring to decision field theory when talking about sub-optimal decision making. Decision field theory deals with how people actually make decisions rather than how we should make (optimal) decisions. The mathematics are based on a probability diffusion process, using a continuous-time Markov chain. There is a growing line of research by Busemeyer and others regarding decision field theory (Busemeyer & Townsend, 1993). I like how DFT accounts for person-in-context effects and affect as a part of decision making. I am also more familiar with linear/matrix algebra so it makes more sense to me when put in terms of a Markov chain. I have to admit that I am still learning the ropes and that I have a math deficit that I am working hard to correct.
Part of the problem of applying chaos theory to psychology is that it requires real math skill, not just a rudimentary understanding of statistics. Math competency is something sorely lacking in psychology. The ground work has never been put in place to incorporate chaos theory into psychology in the first place in terms of quantitative theory building and base functions. There are some nice theoretical ideas around chaos theory and how it could apply to psychology, but few have attempted to actually do the math to prove anything. The majority of chaos theory is built around nonlinear dynamics, which requires understanding a great deal of advanced calculus. Most psychologists in the U.S. seem to struggle with getting through graduate statistics, let alone trying to grasp partial differential equations. My personal belief is that the whole field of psychology is suffering from a lack of mathematical understanding that permeates everything. We should have the same basic math skills as psychological scientists as a physicist, or biologist, or any other natural scientist if we ever hope to shore up the inconsistencies in psychology.
There is a small sense of growing urgency here because most of the quantitative psychologists are either retired, retiring soon, or holding on through emeritus status, and there are very few programs producing quantitative psychologists to replace those we are losing. The field of psychology needs more quantitative folks to deal with the complexity of the phenomena in psychology and we produce fewer each year. Makes me wonder about the future of the field.
Thanks,
Dan
Busemeyer, J. R., & Townsend, J. T. (1993) Decision Field Theory: A dynamic cognition approach to decision making. Psychological Review, 100, 432-459.
On Tuesday, July 17, 2012 1:03:20 AM UTC-7, Andrew Kyngdon wrote:
Dan,
I was referring to actual physical time. As Steve remarked, behaviour and cognitive processes take time to happen. Of course, it is entirely possible that the human perception of time may involve some kind of weighting. But the circumstances under which this may occur, or what specific behaviours such weighting is relevant to, have not been identified. Or at least I’m not aware of any relevant literature.
I am not sure as to what you mean by “sub-optimal real time decision making under natural conditions”. My interest lies in decision making under conditions of risk and uncertainty. I am not that interested in riskless choice. You state that most people are horrible in terms of thinking about probabilities. I would add that they do not seem to make decisions on the basis of probabilities, but the subjective weighting of event probabilities (at least under conditions of risk). These “decision weights”, which are an inverse-S shape function of event probabilities, are at the core of the rank dependent utility theories such as cumulative prospect theory. These decision weights rather accurately predict risk attitude, can account for paradoxical choice behaviour such as the Allais (1953) common consequence and common ratio effects, and can explain why people purchase both insurance policies and lottery tickets. Decision makers attach far greater weight to a change in event probability from certainty (1) to .99 far more than a commensurate change between probabilities (such as between .84 and .85). Likewise from impossibility (0) to .01. This seems to be a fundamental process of risky decision making, as the phenomenon has been quite experimentally robust over the past three decades or so of research.
Chaos theory seems to come in and out of fashion within psychology, but no real progress ever seems to come from it.
Cheers,
Andrew
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/talking-measurement?hl=en.
--
You received this message because you are subscribed to the Google Groups "Talking Measurement" group.
To view this discussion on the web visit https://groups.google.com/d/msg/talking-measurement/-/NxjYvx3A9d4J.
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to talking-measure...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/talking-measurement?hl=en.
--
You received this message because you are subscribed to the Google Groups "Talking Measurement" group.
To view this discussion on the web visit https://groups.google.com/d/msg/talking-measurement/-/NLMhN6Iqs7UJ.
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to talking-measure...@googlegroups.com.
Andrew Kyngdon
Dan,
I’m not familiar with sub-optimal decision making, so I cannot meaningfully comment on it. Decision making under risk and uncertainty is also concerned with descriptive theories (i.e., how people make risky choices). Examples included prospect theory and configural weighting theories such as transfer of attention exchange. The field of behavioural economics is largely concerned with these theories. Mainstream economics still advocates Expected Utility Theory (EUT) as a normative theory of risky choice (i.e., as a theory of how people should make choices), but grudgingly concedes that it is simply not a true account of human choice behaviour under risk.
I can well understand how there has been insufficient theoretical and experimental work done in psychology to allow a serious attempt at applying chaos theory. I also share your belief about increasing the mathematical skills of psychologists; and firmly believe that university syllabus and curricula need to be overhauled. As Borsboom (2006) noted, psychologists tend to only use procedures that are in SPSS. There is a completely unwarranted, discipline wide belief that Fisher – Neyman – Pearson null hypothesis significance testing is the sole route to scientific knowledge. Calls for reform have be made for decades, but like the heavy smoker trying to give up the cancer sticks, psychology just can’t seem to quit NHST. There also seems to me, although I’m probably wrong, a strong element of sheer intellectual laziness amongst behavioural scientists with respect to quantitative methods – “…it’s all too hard so why bother. It’s not like we need to do learn any of that ultra hard stuff to publish”.
However, I am hesitant in calling for more mathematicians into the behavioural sciences. I believe we need people more like experimental physicists, more Faradays if you like, than statisticians. There are a heck of a lot of highly mathematically trained scholars in psychology and psychometrics, but they tend, in my experience, to be not that concerned with developing quantitative, psychological theories, particularly in psychometrics. Most of them are representationalists and feel that one can simply pluck numbers out of the thin air, and that measurements are just numbers, and if you have any set of numbers (like test scores), you have measurement. They recoil in particular at the idea that psychology should develop systems of units; and are usually completely silent on the criticisms made by Joel Michell over the past 20 years. As Paul Barrett noted in 2008, raising Michell’s concerns with highly mathematically trained folk will tend to elicit a whole range of responses, none of which are scientific.
As an anecdotal example of needing less statisticians and more experimentalists, I found a paper on the web a few weeks ago in which the author had developed a probabilistic variant of EUT, in which the important Allais Paradox could be interpreted merely as “error”. He argued that this was a good thing, which I found astounding. We know why decision makers behave in a manner consistent with this paradox – people prefer sure consequences rather than risky gambles that have a good probability of the decision maker receiving more money than the sure thing. Whilst not strictly rational behaviour, it is not entirely irrational at all to prefer a sure thing over a gamble. Furthermore, the paradox has been replicated in virtually every study of it done since Allais (1953) first discovered it (most recently by Huck & Mueller (2012), who found that the paradox is even stronger in samples of decision makers that do not consist of the highly educated). Only in the quantitative behavioural sciences will people try to banish a highly robust and well understood psychological phenomenon to the purgatory of “error”. It is basically arguing that ignorance is preferable to knowledge.
Yes, a lot of quantitative psychology’s stars are either retiring or way past it already. Duncan Luce is 87 and still publishing. The “greying” of quantitative psychology was spoken about when I first got into the field a decade ago, but students of behavioural science seem more than ever unwilling to enter the field.
Cheers,
Andrew
From: talking-m...@googlegroups.com [mailto:talking-m...@googlegroups.com] On Behalf Of Dan
Sent: Thursday, 19 July 2012 4:14 AM
To: talking-m...@googlegroups.com
Subject: Re: [talking-measurement] fundamentals before numbers
Andrew,
I was referring to decision field theory when talking about sub-optimal decision making. Decision field theory deals with how people actually make decisions rather than how we should make (optimal) decisions. The mathematics are based on a probability diffusion process, using a continuous-time Markov chain. There is a growing line of research by Busemeyer and others regarding decision field theory (Busemeyer & Townsend, 1993). I like how DFT accounts for person-in-context effects and affect as a part of decision making. I am also more familiar with linear/matrix algebra so it makes more sense to me when put in terms of a Markov chain. I have to admit that I am still learning the ropes and that I have a math deficit that I am working hard to correct.
Part of the problem of applying chaos theory to psychology is that it requires real math skill, not just a rudimentary understanding of statistics. Math competency is something sorely lacking in psychology. The ground work has never been put in place to incorporate chaos theory into psychology in the first place in terms of quantitative theory building and base functions. There are some nice theoretical ideas around chaos theory and how it could apply to psychology, but few have attempted to actually do the math to prove anything. The majority of chaos theory is built around nonlinear dynamics, which requires understanding a great deal of advanced calculus. Most psychologists in the U.S. seem to struggle with getting through graduate statistics, let alone trying to grasp partial differential equations. My personal belief is that the whole field of psychology is suffering from a lack of mathematical understanding that permeates everything. We should have the same basic math skills as psychological scientists as a physicist, or biologist, or any other natural scientist if we ever hope to shore up the inconsistencies in psychology.
There is a small sense of growing urgency here because most of the quantitative psychologists are either retired, retiring soon, or holding on through emeritus status, and there are very few programs producing quantitative psychologists to replace those we are losing. The field of psychology needs more quantitative folks to deal with the complexity of the phenomena in psychology and we produce fewer each year. Makes me wonder about the future of the field.
Thanks,
Dan
Busemeyer, J. R., & Townsend, J. T. (1993) Decision Field Theory: A dynamic cognition approach to decision making. Psychological Review, 100, 432-459.
On Tuesday, July 17, 2012 1:03:20 AM UTC-7, Andrew Kyngdon wrote:
Dan,
I was referring to actual physical time. As Steve remarked, behaviour and cognitive processes take time to happen. Of course, it is entirely possible that the human perception of time may involve some kind of weighting. But the circumstances under which this may occur, or what specific behaviours such weighting is relevant to, have not been identified. Or at least I’m not aware of any relevant literature.
I am not sure as to what you mean by “sub-optimal real time decision making under natural conditions”. My interest lies in decision making under conditions of risk and uncertainty. I am not that interested in riskless choice. You state that most people are horrible in terms of thinking about probabilities. I would add that they do not seem to make decisions on the basis of probabilities, but the subjective weighting of event probabilities (at least under conditions of risk). These “decision weights”, which are an inverse-S shape function of event probabilities, are at the core of the rank dependent utility theories such as cumulative prospect theory. These decision weights rather accurately predict risk attitude, can account for paradoxical choice behaviour such as the Allais (1953) common consequence and common ratio effects, and can explain why people purchase both insurance policies and lottery tickets. Decision makers attach far greater weight to a change in event probability from certainty (1) to .99 far more than a commensurate change between probabilities (such as between .84 and .85). Likewise from impossibility (0) to .01. This seems to be a fundamental process of risky decision making, as the phenomenon has been quite experimentally robust over the past three decades or so of research.
Chaos theory seems to come in and out of fashion within psychology, but no real progress ever seems to come from it.
Cheers,
Andrew
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/talking-measurement?hl=en.
--
You received this message because you are subscribed to the Google Groups "Talking Measurement" group.
To view this discussion on the web visit https://groups.google.com/d/msg/talking-measurement/-/NxjYvx3A9d4J.
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to talking-measure...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/talking-measurement?hl=en.
--
You received this message because you are subscribed to the Google Groups "Talking Measurement" group.
To view this discussion on the web visit https://groups.google.com/d/msg/talking-measurement/-/NLMhN6Iqs7UJ.
To post to this group, send email to talking-m...@googlegroups.com.
To unsubscribe from this group, send email to talking-measure...@googlegroups.com.
Stephen Humphry
Stephen Humphry | Associate Professor
Graduate School of Education
The University of Western Australia
M428, 35 Stirling Highway, Crawley, WA, 6009
Telephone: +61 8 6488 7008
Fax: +61 8 6488 1052
www.gse.uwa.edu.au
CRICOS Code: 00126G
From: talking-m...@googlegroups.com [mailto:talking-m...@googlegroups.com] On Behalf Of Paul Barrett
Sent: Thursday, 19 July 2012 6:06 AM
To: talking-m...@googlegroups.com
Subject: RE: [talking-measurement] fundamentals before numbers
Paul, thanks for the list of sources, especially the latest Theory and Psych one.
You say:
I must say I also like Borsboom and colleague’s work with network theory .. as applied to symptom comorbidity ... again, a different approach altogether which avoids ‘quantity’ measurement assumptions’. They are now applying this modeling approach to personality attributes.
Borsboom, D., Cramer, A.O.J., Schmittman, V.D., Espkamp, S., Waldorp, L.J. (2011) The small world of psychopathology. PLoS One (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0027407 ), 6, 11, 1-11.
Background: Mental disorders are highly comorbid: people having one disorder are likely to have another as well. We explain empirical comorbidity patterns based on a network model of psychiatric symptoms, derived from an analysis of symptom overlap in the Diagnostic and Statistical Manual of Mental Disorders-IV (DSM-IV).
Principal Findings: We show that
a) half of the symptoms in the DSM-IV network are connected,
b) the architecture of these
connections conforms to a small world structure, featuring a high degree of clustering but a short average path length, and
c) distances between disorders in this structure predict empirical comorbidity rates. Network simulations of Major Depressive Episode and Generalized Anxiety Disorder show that the model faithfully reproduces empirical population statistics for these disorders.
Conclusions: In the network model, mental disorders are inherently complex. This explains the limited successes of genetic, neuroscientific, and etiological approaches to unravel their causes. We outline a psychosystems approach to investigate the structure and dynamics of mental disorders.
You seriously liked this? I followed complex adaptive systems stuff closely in the 90s and beyond for a while. I see yet another case of people taking an approach and all but totally ignoring its origins and the basis for its rationale.
Steve
Andrew Kyngdon
Hello Paul,
As I argued in my response to Dan, psychology needs more Faradays than Einsteins (apparently the latter kept a portrait of the former on his mantelpiece). At least at the current stage of development. Thanks for the Sain-Mont reference. Hank Stam should be commended for his sustained interest and support in allowing Theory & Psychology to play host to the measurement debate. It really shows up the timidity of the psychometrics journals to seriously engage with the debate. If Michell and others are so horribly wrong, what have they got to fear?
The Lexile Framework for Reading is quite an impressive psychometric system, but , again, it is a system which mainstream psychometricians seem unwilling to discuss or critique. There are some key similarities that it shares with utility theories, which is something I hope to present on at the next NCME conference. At the core of the system is the idea that if a passage of prose text were used to create sets of reading items, and that passage was accurately targeted to the reading ability of the examinee, then the examinee should get about 75% of those items correct. Of course, there is absolutely no a priori reason that this should happen – readers may make more correct responses or give fewer. But as it turns out, what we have found is that out of 4,300,000 administered reading items, readers have gotten 74.45% of these items correct. It is difficult to see how the Lexile Framework is false given this observation. It may well be false, but this I think is pretty strong evidence in support of it.
In regards to psychopathology and clinical psychology, there was a recent article in one of Australia’s quality broadsheet newspapers discussing how disconcerting it is to learn that the DSM keeps growing with each new edition, and that there is a tendency now to “pathologise” the everyday vicissitudes of human life.
I mean look at this http://www.livescience.com/10679-psychology-darth-vader-revealed.html Apparently Darth Vader had “Borderline Personality Disorder”. So psychiatrists are now pathologising Star Wars. Where will it end? Whilst I can understand Lord Vader strangling to death an annoyingly incompetent Admiral Ozzel after remarking that he was “as clumsy as he is stupid” (The Empire Strikes Back), in my view, someone who willingly takes a lightsaber to a group of kids under five (The Revenge of the Sith) is just plain evil.
Could you imagine Vader on the psychiatrist’s couch, or on Dr Phil?
Cheers,
Andrew
From: talking-m...@googlegroups.com [mailto:talking-m...@googlegroups.com] On Behalf Of Paul Barrett
Sent: Thursday, 19 July 2012 8:06 AM
To: talking-m...@googlegroups.com
Subject: RE: [talking-measurement] fundamentals before numbers
Hmmm ... I think a close reading of:
David Torres Irribarra
The Lexile Framework for Reading is quite an impressive psychometric system, but , again, it is a system which mainstream psychometricians seem unwilling to discuss or critique.
Paul Barrett
Hello Steve!!
What I liked about the work was the novel approach to representing data relationships without attempting to do so via the usual ‘latent variable’ approaches. I’m no fan of the DSM but the comorbidity issue is a separate feature of mental health practice – which has normally been addressed using psychometric questionnaire/checklist analysis, correlations, and other forms of covariance and cluster analysis.
Ultimately I think the approach is more ‘data representation’ than anything to do with quantitative measurement per se .. but I did find it interesting as a way of representing comorbidity among attributes, with a neat test of some features of the representative model against observed epidemiological data.
Personally, I didn’t see anything that might be regarded a complex ‘adaptive’ systems approach in the work .. as there is nothing ‘adaptive’ that I can visualize in a set of static DSM symptoms/features. Dunno .. I just saw it as a representation model – much in the way you might look at non-metric MDS, a Kohonen, or other form of inductive classifier.
To me, that word ‘adaptive’ implies ‘some kind of interaction between inputs/outputs evolved over time’ ..
Regards .. Paul
Advanced Projects R&D Ltd.
__________________________________________________________________________________
From: talking-m...@googlegroups.com [mailto:talking-m...@googlegroups.com] On Behalf Of Paul Barrett
Sent: Thursday, 19 July 2012 6:06 AM
To: talking-m...@googlegroups.com
Subject: RE: [talking-measurement] fundamentals before numbers
Paul Barrett
Hello Andrew ...
“psychology needs more Faradays than Einsteins”
Absolutely agree.
And I’m with you all the way with the DSM crap and the urge to diagnose anything that moves.
That piece of mindless stupidity from the article ...
“it was during my residency in psychiatry while trying to explain borderline personality disorder to medical students that I thought of Anakin," said Eric Bui, a psychiatrist at Toulouse University Hospital in France.
Bui and his colleagues first presented their diagnosis at the annual convention of the American Psychiatric Association in 2007. Now, their letter to the editor titled "Is Anakin Skywalker suffering from borderline personality disorder?" is slated to appear in an upcoming issue of the journal Psychiatry Research.”
is why both psychologists and psychiatrists remain figures of ridicule in the minds of many outside the discipline.
Instead of being publicly ridiculed for spouting such tosh, they are given conference time and a brief publication ... I wouldn’t mind if they meant it to be a humorous take on some of the wally-brained psychologists and psychiatrists out there – such as the recent wonderful article by Bones, A.K. (2012) We knew the future all along: Scientific hypothesizing is much more accurate than other forms of precognition- A satire in one part. Perspectives on Psychological Science, 7, 3, 307-309, but these idiots abuse their professional status by spouting their clinical judgments as ‘substantive’ knowledge claims.
Ah well ... I leave it to Arina K. Bones (a pseudonym!) ..
“With a near 100% accuracy rate, psychological scientists have clearly demonstrated that psychological scientists already know what is going to occur. This makes the subsequent empirical confirmation superfluous. Once predicted, there is no logical justification for expending the resources to actually conduct the data collection and analysis. ”
I look forward to the new journal:
“Impressively, a soon-to-be published journal, Bite-Size Psychology, is pursuing a new reporting format to facilitate these practices: the 15-word-limit “all-headline” article format. Easy-to-read. Flashy. No data or methods. Perfect for fostering real impact—media mentions. ”
Denny Borsboom
Although the simulation models we use can be adaptive, the ones in the PLoS paper you cite are not. The big network is just a network representation of the DSM. It's simple, yet many substantive people have found it to be of considerable interest.
The simulation models we use in the PLoS paper are dynamic of course, in the sense that they have a function that propagates them through time. (However, they aren't adaptive in that they don't change like, e.g., neural networks typically would. Nevertheless extending the models in that direction is an interesting approach to deal e.g. with the phenomenon of 'kindling' - the finding that depression appears to set in easier when a person has had a prior episode, an effect that remains after you control for genetics etc. The system somehow seems to 'learn' how to become depressed. I think we could accommodate that theoretically)
We've implemented one of these models in a NetLOGO simulation model that you can play around with online:
http://ccl.northwestern.edu/netlogo/models/community/run.cgi?Symptom%20Spread%20Model.1341.551.0
Also we have just constructed methodology, paper now in production, to assess people's personal networks by analyzing time series of symptom reports that are becoming more frequent, so hopefully in time we'd be able to construct these model for individual people's network structures. This turned out really nicely, and I hope that it will be of use to practicing clinicians.
Finally, we have constructed visualization techniques for psychometric data based on the network idea:
Epskamp, S., Cramer, A.O.J., Waldorp, L.J., Schmittmann, V.D. and Borsboom, D. (2012) qgraph: Network Visualizations of Relationships in Psychometric Data. Journal of Statistical Software, 48(4), 1-18.
Even if all our other stuff ultimately crashes, then at least we have produced some pretty pictures.
Best
D
> You received this message because you are subscribed to the Google Groups
> "Talking Measurement" group.
> To post to this group, send email to talking-m...@googlegroups.com.
> To unsubscribe from this group, send email to
> talking-measure...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/talking-measurement?hl=en.
--
Department of Psychology
University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands
+31 20 525 6882
d.bor...@uva.nl
http://sites.google.com/site/borsboomdenny/dennyborsboom
Andrew Kyngdon
Hello David,
Welcome to Talking Measurement.
In Kyngdon (2011) I applied the theory of conjoint measurement to the Lexile Framework using the probabilistic order restricted inference methodology created by Karabatsos (2011). I found that in order to satisfy the axioms, one had to permute the columns of the conjoint array, thus contradicting the Lexile difficulty measures of the items. Ben Domingue, whom I had the pleasure of having a few drinks with at the Top of Hill pub in Chapel Hill, NC last August (Derek Briggs was there as well), addressed one conceptual flaw in Karabatsos’ methodology which involved the order upon cells entailed by the double cancellation axiom. He managed to analyse the whole dataset rather than the 3 x 4 matrix I looked at and found systematic axiom violation.
As I concluded in the paper, it was not a definitive study on the descriptive adequacy of the Lexile Framework. The key problem may not have been due to the theory, but to the type of stimulus employed. The data I used were obtained from a traditional pencil and paper based test using what is known as an “embedded sentence cloze” or an “inter-sentential cloze” reading item type. This item type consists of a stem of professionally edited continuous prose text, in which a test constructor has inserted at the end of the passage a sentence with a missing word. The examinee “closes” the sentence by selecting a word from a list of four, much like a multiple choice item. Until recently this kind of reading item was the empirical backbone of the Lexile framework, and was known within MetaMetrics as a “native” item type.
An example of this kind of item is as follows:
Thus did he pray, and Apollo heard his prayer. He came down furious from the summits of Olympus, with his bow and his quiver upon his shoulder, and the arrows rattled on his back with the rage that trembled within him. He sat himself down away from the ships with a face as dark as night, and his silver bow rang death as he shot his arrow in the midst of them. First he smote their mules and their hounds, but presently he aimed his shafts at the people themselves. He was _______.
a) merciless b) qualified c) accommodating d) depressing
This is an “imbedded sentence cloze” reading item type created from text in Homer’s Iliad. Lexile measure of the item stem is 1220L. The problem is, as is pretty obvious, that an item writer could construct a large of number of “imbedded sentences” to use with the same text passage, which of course affects examinees’ responses to the items. One could create easier and harder imbedded sentences, thus creating harder and easier items from the one stem. This is an intractable limitation of this kind of reading item type; and something that probably contributed to axiom failure in the Kyngdon and Domingue studies.
Stenner, et al (2006) investigated this experimentally and proposed the “ensemble mean” hypothesis – that the Lexile text complexity measure is the mean difficulty of all possible items that could be constructed from the one passage of text. One example they used in their study was as follows:
1. She disappeared through the trees. “Fine with me,” I thought angrily. It would be fine with me if I never saw her again. I am glad she is _______.
2. She disappeared through the trees. “Fine with me,” I thought angrily. It would be fine with me if I never saw her again. I was _______.
3. She disappeared through the trees. “Fine with me,” I thought angrily. It would be fine with me if I never saw her again. I ______ her.
The Lexile text complexity of the stem was 430L. Items 1, 2 and 3 had “empirical Lexile difficulties” (transformed Rasch item difficulties) of 269L, 632L and 740L, respectively. Now the arithmetic mean of these empirical Lexiles is 547L, which is a value closer to the theoretical 430L given by the Lexile Framework than any of the empirical Lexiles. This was the first evidence in support of the ensemble mean hypothesis.
Of course, this study was quite limited given the amount of work that needs to go into creating a rather small number of items. So information technology was exploited to produce another reading item type – the autogenerated item cloze. Here a technology, called Learning Oasis, scans a passage of continuous prose text, selects words to act as “closes” and creates three foils/distracters based on part of speech and difficulty. For example,
The study and interpretation of myth and the body of myths of a particular culture. Myth is a complex cultural phenomenon that can be approached from a number of viewpoints. As generally understood, a myth is a story or narrative that is traditional in a certain culture, having been passed down from early times and regarded as true. It may be said to 1 symbolically the origin of the basic elements and assumptions of a culture. Mythic narratives frequently revolve around the doings of gods or heroes, and may relate, for example, how the world began, how humans and animals came into being, or how certain customs, gestures, or forms of human activities 2. Almost all cultures possess or at one time possessed and lived in terms of myths.
1 immerse belittle portray contradict
2 originated adorned handicapped entwined
The Lexile text complexity of this passage was 1300L and the empirical Lexile was 1357L. This type of item, by the sheer number of individual cloze items that can be created, can enable a more rigorous test of the Lexile Framework. From June 2007 to June 2011, Learning Oasis created over 4,300,000 of such items from almost 400,000 student – text encounters. The Lexile theory predicts that 75% of such items should be responded to correctly, IF the Learning Oasis program is accurately targeting reader ability to text difficulty. What was found was that 74.45% of these items were responded to correctly. So the Lexile Framework was only 0.55% off in its prediction. What this means is that we could trade off reader ability and text complexity to keep the success rate constant, or keep reader ability constant and manipulate text complexity to produce a pattern of success rates consistent with the cancellation axioms of conjoint measurement.
Another thing is that conjoint measurement itself recently got a revamp by Luce & Steingrimmson (2011), in which a condition they called “conjoint commutativity” replaces double cancellation. It essentially states that we manipulate two variables so as to produce equal magnitudes of the third. Interestingly, Duncan Luce got in contact with me after Kyngdon (2011) was published as he reviewed my paper; and I conveyed to him my ideas of testing “conjoint commutativity” with the Learning Oasis item type. He was quiet interested to hear how it would turn out.
So the story with Lexiles and conjoint measurement has a fair bit more to run. The imbedded sentence cloze item type was critical to the early success of the Lexile Framework as a psychometric system, but it has been superseded by the auto-generated item cloze type and information technology. I personally feel that one of the key problems that psychometrics faces is the continued reliance on the pencil and paper based test. It is a very crude, very old and very limited observational methodology for individual differences in cognitive abilities. Information technology may greatly aid the development of any descriptive theories of the cognitive processes involved in responding to items.
Cheers,
Andrew