A Response To The Adventure Game Industry Markey Research Summary
David Wendt
Professor of Statistics
Statistical Consultant
Permissions: This file is based on data presented in the Adventure Game
Industry Market Research Summary. The file in question is Copyright 2000,
Wizards of the Coast. The file in question had permission granted to be
freely redistributed in or quoted in whole or in part, provided that this
attribution remained intact.
With regard to Mr. Dancey's Market Research Summary:
While I applaud Wizards of the Coast, and Mr. Dancey in particular, for
their openness with their market data, I feel that must express some
concerns with the manner in which they presented their information. While
Market Research
methodologies are commonly accepted for internal corporate use, one must be
careful when presenting them in a public forum. In this document, I plan
to address my concerns with this particular summary one at a time. I
encourage my colleagues to comment on my discussion and in particular on a
number of open issues I point out in the text.
With regard to the 'Pokemon Effect': While I respect the decision not
present information on the trading card game (henceforth referred to as
TCG) component of the data, your survey was intended to "determine
information about [TCGs], [RPGs] and [MWGs]". This sends up an immediate
flag with respect to my training. Any survey or test is set up to gather
specific information. When one changes the use of the data, one is in
danger of invalidating the information. Thus, we must already consider the
rest of your conclusions with a grain of salt.
My second major concern is related to your decision to exclude population
based on age. Your acknowledgement of the impact of this exclusion was a
good first step. However, a great many of your conclusions are age- and
time- based.
As a result, you have potentially introduced a great deal of bias to your
data. This is a second reason to be wary of the conclusions presented in
your document.
Additionally, you claim that the survey "accurately represents the US
population as a whole". I will assume this is a misstatement. The US
population includes a large population in the over 35 group. You might
argue that your survey accurately represents the 12 to 35 age group in the
United States. This is also possibly incorrect, however, as the screener
and follow-up survey may have produced further bias based on the
information provided to the respondents. Were the surveys identified as
being sponsored by WOTC or some related organization? In addition,
non-compliance also tends to bias such surveys (who is the sort of person
who would take the time to fill them out?). This non-compliance bias is
well documented and is part of the recent Census versus Sampling debates,
for those who wish to read more on this issue.
I will address further concerns related to the relationship between age and
time further later in this document.
Let us consider your conclusions in Section3: Basic Demographics. You
either have been careless in your explanation of your conclusions, or have
omitted some of the results you base those conclusions on. While you
provide single level crossover values, you do not provide the probability
of multiple crossovers (defined as, for instance, a TRPG gamer plays two or
more of the other categories.) Comparing the crossover values you did
provide to the percent of women in gaming (which you call significant), the
values seem comparable. This would imply that by your definitions, these
crossover percentages are significant. (As an aside, 'significant' has a
very specific statistical meaning, so one should be careful when using it
in a public document.)
Otherwise, I am a bit concerned with the conclusion you draw. I am not at
all convinced that the fact there is little crossover means that extending
a brand can be successful. I would imagine there are psychological factors
to be accounted for, and would welcome the comments of anyone trained in
such matters on this issue. Further, I suspect there may be gender buying
trends that you have not accounted for when you say that "females, as a
group, spend less than males". Again, I choose not to comment further, but
encourage those with appropriate training to do so.
Finally, for this section, while your conclusion that gaming is an adult
hobby is well supported by your data, I remind the reader again that a
significant portion of the populace has not been included. I suspect that
the percentages of gamers in the lower age brackets has been unduly
inflated as a result. Further, your final sentence in this section,
"...the existing group of players is aging and not being refreshed by
younger players at the same rate as in previous years" is completely
unfounded. First, without previous years market data, we have no benchmark
to compare these number to. Secondly, later in your own document, you
observe that gamers join the hobby at various stages of life. This will
directly increase percentages as age increases by definition.
In sections 4 and 5, I again encourage the reader to be aware that all of
these percentages as biased by the exclusion of respondents over 35. In
particular, levels of education and marital status have direct
relationships to age. A 12 year old is much less likely to have done
post-graduate work or to be partnered than a 35 year old. This must be
considered when reviewing the results.
I have additional concerns about the results presented in section 5.
First, I must revisit my concerns about potential bias due to the attributions
of the survey. If response were biased by perceived ownership (that is if
the respondent believed he was responding a specific company), then all the
percentages are in doubt, but in particular those about "past game
experiences". Different game companies have historically had different
attitudes about a number of the experiences the respondents were queried on.
This should be kept in mind when considering the results.
Secondly, the amount of spending is likely correlated to age. Due to the
impact of education and employment, older gamers are likely to have more
expendable income. Thus increasing their total spending. Further, the
longer a particular gamer has been in the hobby, the older they are, thus
taking us back to the earlier argument. As a result, subdividing spending by
age is of minimal information, and likewise with the D&D specific spending.
Regarding your monthly spending comments, I will simply encourage you to
look to hypothesis testing. I suspect that the perceived differences are
not significant. That is, due to variations in the data, there is not
sufficient evidence to say that there are spending differences. Such tests
are well documented in basic statistics texts, and as a result, I will
comment no further on them.
I do not find the results on the spending of current versus lapsed players
overly interesting. I suggest an alternate measure to you. That is the
Mean Total non-D&D Spending per non-D&D game played. Based on your
numbers, Mean Total non-D&D spending is ($1273 - $895 =) $378 for current
players and $1068 for lapsed players. Dividing these by the Number of
non-D&D games played (1.2 and 2.3 respectively), gives the following:
Mean Total non-D&D Spending
per non-D&D RPG Played: $315 / $464
These numbers are much closer indicating that perhaps there is little to no
difference in non-D&D playing. It also suggests that perhaps RPGs other
than D&D represent less of an investment for players. I also suggest that
there may be a bias inflicted by the age truncation that may explain the
difference between current and lapses spending. Further, lapsed players
may be overestimating their spending and current players underestimating
theirs. This is a psychological factor that I would encourage
professional or academic comment on.
I am a little concerned about your percentages regarding "what games TRPG
players play" and "retail type". First, does the 35 or older truncation
bias these results? Do any of the other potential sources of bias
discussed in this document affect these numbers? These are points for
consideration. Relating more specifically to what games are being played, the
choice of games over companies is disturbing and introduces a newer source
for error. I suggest that for future surveys, this sort of question be
replaced by "Do you play TRPGs from the following companies? Wizards of
the Coast, White Wolf, ..., Other, Unknown." A number of solid games were
excluded from your list and not all companies were included equally. As a
result, this data, as it stands, is useful for little more than trivia
(ignoring any bias issues). Percentages for games from the same company
cannot be added with any meaning, due to overlap, nor can this be used for
any measure of marketshare.
I am curious about how you moved from "on a scale from 1 to 5" to
percentages in the question where players where asked to describe aspects
of their games. While the data points are potentially of great interest
with respect to game design, without more information on how the data was
gathered and the results computed, I believe this section tells us very
little.
In conclusion, I again applaud Wizards of the Coast for their openness
with their market data. I encourage other members of the gaming community
to be equally forthcoming. However, due to decisions made in the data
collection phase and in the results development phase, I caution anyone who
reads this survey to be cautious in drawing any conclusions. There is not
only a danger of significant bias, but important correlations in the data
have been seemingly ignored. Market research is a powerful internal
corporate tool, but must be taken as such when released to the public.
David Wendt, Ph.D.
Bruce Baugh
>Permissions: This file is based on data presented in the Adventure Game
>Industry Market Research Summary. The file in question is Copyright 2000,
>Wizards of the Coast. The file in question had permission granted to be
>freely redistributed in or quoted in whole or in part, provided that this
>attribution remained intact.
Hi, Dave! Good to see you posting here. Fascinating reading - it
confirms some of my initial concerns, but from a much better-informed
perspective.
--
Bruce Baugh / bruce...@sff.net
"Never let it be be said, especially by large men with big guns,
that I failed to help." - Dave Weinstein