Reason Magazine
The Prediction Markets Stumble
The "wisdom of crowds" crapped out in Election 2006.
Katherine Mangu-Ward | November 13, 2006
It's not often one finds oneself in a position to eat crow on behalf of
more than 50,000 people-but here I am. My beloved election prediction
markets have failed me this cycle, and so I'm here to apologize for the
inaccuracy of the predictions I stole from those markets, and for the
failings of the tens of thousands of traders who were supposed to
illuminate the truth for us all.
Usually, when people-especially pollsters-predict election outcomes
wrong, they say something like "No one could have guessed that the
turnout would be so high in the southwestern part of the state" and
leave it at that. They don't get fired, and they only occasionally even
concede that they might have something to apologize for. And maybe they
don't. After all, they're in the business of entertaining and
informing, but perfect accuracy isn't required.
For a brief, shining moment last week, politics junkies had lots of
facts at their disposal, and it made them giddy. Judging by the
frequency with which polling data was released, Americans spent the
entire first week of November answering pollsters questions about their
voting habits, opinion on the issues, and preferences for the more
handsome candidate. Oceans of data poured forth from every television,
every political blog, every newspaper. With all that information,
talking heads swelled. Already confident in their predictive abilities,
the presence of such a profusion of information sent election season
seers into a frenzy of confident explication and prediction.
Conversations about politics inevitably degenerated into poll
swapping-"Zogby says Allen is ahead." "Yes, but CNN's poll favors
Webb." And when things got really desperate (either on air or over
drinks) surprising fervent discussion of margins of error would flame
into existence.
Thorough all this chaos, I calmly cited online election prediction
markets. Standing zen-like above the fray in that chaotic week, I'd
casually drop a mention that "InTrade has the likelihood of Republicans
holding the Senate at 70 percent." I delivered tiny, smug lectures on
the superior ability of markets to aggregate information, name-checking
Hayek. I sat aloof, murmuring the old TradeSports motto to myself "Put
your money where your mind is."
This strategy has served me well in the past: In 2004, InTrade traders
correctly called all 50 states the weekend before the election. The
Iowa Electronic Market, a highfaultin', academic futures market in the
same vein, has frequently boasted a lower margin of error than polls
since its creation in 1988.
But this time around, InTrade and its other electronic market cousins
didn't acquit themselves very impressively. Iowa Electronic Markets had
Republicans holding the Senate, with an all-Republican Congress trading
high for most of the history of this cycle's market, and an all
Democratic Congress trading very low-below 20 percent probability for
most of the market's history. And while markets devoted to individual
races in the Senate tended to be correct, the overall prediction
markets for the balance of power in the upper house miscalled the race.
These outcomes were still better than a heck of a lot of pundits, but
not good enough to justify my serene pre-election confidence.
Weirdly, the McLaughlin Group, the fustiest of all the talking head
shows, had one of the best records this cycle, with Eleanor Clift,
Lawrence O'Donnell, and John McLaughlin all predicting Democratic
takeover of the Senate and calling nearly all of the close races
correctly. Still, that old line about stopped clocks comes to mind.
Now my favorite moment of the political cycle is upon us: Total
ignorance. The whole ecosystem of pundits, pollsters, and
politics-obsessed bloggers has been floating in a lush primordial soup
of political data. But information-rich environments are not the
natural habitats of such creatures. As soon as the marathon election
night coverage ends ("We are calling the Senate for the Democrats") the
punditry returns to a state of Eden-like ignorant bliss, unburdened by
data.
And for now, I'll go back to my old strategy. After all, markets still
have a better track record than pundits. Sen. John McCain (R-Ariz.) is
trading at just over 53 cents on the dollar at InTrade today as the
prospective 2008 Republican nominee. After a lull during the
congressional election when investors were in wait-and-see mode,
McCain's chances spiked up over 50 percent, as Allen contracts
plummeted to less than one cent on the dollar.
And InTrade has Sen. Hillary Clinton (D-NY) on a steady upward
trajectory since early September. After virtually no trading in the
first week of November, she's up over the 50 percent mark for the
Democratic 2008 nominee. So, McCain versus Hillary it is.
Fortunately, even if I'm wrong this time around, I don't have to do
more than offer a casual apology. I'll know not to be quite so cocky,
and all those stupid traders I had to apologize for this time around
have lost all their money, so they'll be sufficiently chastened (and
impoverished) when considering their bets for 2008.
Katherine Mangu-Ward is an associate editor at Reason.
I thought this election season was a bit fuzzy, particularly the Senate
majority and Joe Lieberman. Joe was an independent candidate, supported by
the GOP, and the Dems campaigned hard against him. If the markets are driven
by party affiliation, where does an independent-democrat or
democrat-independent that will caucus (?) with the Dems to get a committee
chair fit? Also, what happens if Joe joins the GOP, which is possible, then
there is a 50/50 split. IT would be good to get more opinions of this season
and the markets.
Cheers,
John
thoughts?
tom abeles
Q
_________________________________________________________________
All-in-one security and maintenance for your PC. Get a free 90-day trial!
http://clk.atdmt.com/MSN/go/msnnkwlo0050000002msn/direct/01/?href=http://www.windowsonecare.com/?sc_cid=msn_hotmail
At 02:05 AM 11/15/2006, Dave Snowden wrote:
>but hardly matches some of the excessive claims on this list serve
>... prediction markets, like VNA are a useful tool but not a panacia
>
>>While it's true that the markets were "wrong" on control of the Senate,
>>70-30 is far from a sure thing. 30% probability of Dem control was an
>>underdog, but not prohibitive, IMO.
>
>
>Robin Hanson rha...@gmu.edu http://hanson.gmu.edu
>Associate Professor of Economics, George Mason University
>MSN 1D3, Carow Hall, Fairfax VA 22030-4444
>703-993-2326 FAX: 703-993-2323
Perhaps there are "technical" improvements that this group can debate,
but I sense a larger and perhaps more ominous problem that is brewing,
namely how prediction markets may be misunderstood by those outside
this Google group, just as the awareness of prediction markets is
dramatically growing.
Most people, we should remember, either don't care about the details or
can't quite grasp them, and instead just want to know what prediction
markets can do and how well they can do them (as a marketing person
might say, the users want to know what the benefits are, and what it'll
cost to get them).
If this seems a bit fluffy for a board full of financial engineers,
let's just bear in mind that more than a few great ideas and emerging
industries have been stymied (or strangled in the crib) because they
either over-promised/under-delivered or were misunderstood by the
general market, i.e., the media, consumers, regulators, etc.
Perception is indeed reality. And as one who had skin in previous
emerging industries, I can assure you that misperception is a nightmare
we all can do without.
Even the term "prediction market" is loaded with expectation, because a
prediction, in the minds of most people, is either right or wrong; you
either nailed it or you didn't; there are no shades of gray. They
expect 100%. Period.
Perhaps we should all think about this, and even revisit the suggestion
posed here by Hubertus a number of weeks ago that a nomenclature for
this otherwise broad, ill-defined world of ours is needed.
As my own PR counsel reminds me often, you either define yourself for
the world or the world will define you.
-- Peter Leitner
This poster is a simple Internet troll.
http://en.wikipedia.org/wiki/Internet_troll
i.e., In Internet terminology, a troll is a person who enters an established
community such as an online discussion forum and intentionally tries to
cause disruption, most often in the form of posting inflammatory messages.
(Wikipedia)
He just spews provocative bait. Don't expect any facts or reply.
Bio: http://www.cognitive-edge.com/files/David-Snowden.pdf
Occasionally, there are constructive comments.
Better just to ignore until he himself decides to behave responsibly. He has
a lot of reputation at stake. It make little sense for him to be mendacious
or predatory.
If it continues the group moderators are free and privileged to take care of
the problem.
Cordially,
-j
-----Original Message-----
From: Predictio...@googlegroups.com
[mailto:Predictio...@googlegroups.com] On Behalf Of Robin Hanson
Sent: Wednesday, November 15, 2006 3:27 AM
To: Predictio...@googlegroups.com
Subject: Re: The Prediction Markets Stumble
Jed –
We do ‘adjust’ MovieStocks based on historical performance - which I believe is what you are referring to by ‘modifying a market’s predictions based on past performance’.
When a film goes into its initial release weekend (typically on a Friday), we halt trading of that film’s stock for that opening weekend. We then ‘adjust’ the share price on Monday reflecting that weekend’s box office performance, and then restart trading for the duration of the stock’s life on the exchange (e.g. approximately four weeks).
Alex -
From:
Predictio...@googlegroups.com
[mailto:Predictio...@googlegroups.com] On
Behalf Of Jed D. Christiansen
Sent: Wednesday, November 15, 2006
6:44 AM
To:
Predictio...@googlegroups.com
Subject: Re: The Prediction
Markets Stumble
It's true that the only way to assess if a market is accurately capturing probabilities is through multiple independant trials. (The calibration of a linear market, such as HSX's movie stocks, can be easily measured.)
In the research I conducted this summer, I ran a number of markets of the
course of several weeks, each one being liquidated as a new set of markets
opened. I found no significant calibration difference from the first
markets to the final markets; calibration depended much more on the number of
participants. That said, I'm sure individual traders "learned"
based on the profit/loss feedback provided by each set of predictions, even
though the data didn't show it specifically.
I don't know of any markets that modify a market's predictions based on past
performance. The only thing even potentially
related is some work that HP has done, where each trader's risk preference is
measured before participating in the market, and their preferences are then
incorporated into the market prediction. But that has nothing to do with
historic performance.
Regards,
Jed
On 11/15/06, tom abeles < tab...@hotmail.com> wrote:
Question: Prediction Markets such as this election are singularities. Many
of the examples of prediction markets are similarly singular events. Are
there prediction markets that are adjusted by other, probably historic,
performances by the same group of predictors- i.e. either adjustments by an
outside group or a vehicle to allow for learning to affect outcomes of the
predicting group?
thoughts?
tom abeles
>From: "ahawkman" < ahaw...@yahoo.com >
>Reply-To: Predictio...@googlegroups.com
>To: "Prediction Markets" < Predictio...@googlegroups.com
>
>Subject: Re: The Prediction Markets Stumble
>Date: Wed, 15 Nov 2006 04:38:38 -0000
>
>
>
>While it's true that the markets were "wrong" on control of the
Senate,
>70-30 is far from a sure thing. 30% probability of Dem control
was an
>underdog, but not prohibitive, IMO.
>
>
>>
Jed – we do make adjustments to outcomes as part of our propriety model for long-lead forecasts we provide in our research service. We can discuss off-line if you are interested in learning more.
Bravo, a thin response from David Snowden. No facts, just more bait,
now disguised as being 'fairly accurate.'
Here are some facts for M. Snowden --
1.) The specific reason the 'list serve' (sic) exists is to debunk
myth, hype and 'excessive claims.'
2.) No one here, particularly me, ever said prediction markets are a
'panacia' (sic).
Here is what 'Dave' wrote --
"excessive claims on this list serve ... prediction markets... are not
a panacia"
Of course 'Dave' knows this is false, but he is just too cowardly to
admit he is wrong, confused, that he made a mistake.
The discussions here are authentic.
Trolls often say they are 'too busy' to retrieve facts or substantiate
their inflammatory, wild claims. Note: This is a Usenet-style group,
from Google's purchased of Deja. Archives are open, available back to
1981.
When challenged, they always couch their bait as legitimate
'disagreement.' Not so.
Anyway, no biggie, just beware, this poster has a record of being a
troll, and not the kind that lives under bridges and have shillelaghs.
Cordially,
-j
"More generically, the term "arms race" is also used to describe any competition where there is no absolute goal, only the relative goal of staying ahead of the other competitors. Evolutionary arms races are common occurrences, e.g. predators evolving more effective means to catch prey while their prey evolves more effective means of evasion. This is sometimes called the Red Queen effect. In addition to predators, parasites can force their hosts into an arms race.
In technology, there are close analogues to the arms races between parasites and hosts, such as the arms race between computer virus writers and anti-virus software writers, or spammers against Internet Service Providers and E-mail software writers."
<snipped>
"However, they can provide real, measurable improvements compared to current forecasting methods. Now, 0.2% may not be much, but if prediction markets improve forecasting by 3-5% . . . "
Thanks for your message.
The concern is that the prediction markets community already has one
bomb-thrower. There is perhaps an over-sensitivity to keeping the
discourse authentic, cordial, open, agnostic and genuine.
Passion and opinion is welcome, expected and encouraged.
It is worth emphasizing the prediction markets group, consortium,
libraries, Websites, events and so forth are open, non-commercial,
not-for-profit. Furthermore, unlike other groups, announcement of new
offerings, events, people, libraries, blogs, Websites, networks,
methods, tools, groups, etc., are also open, welcome, expected and
encouraged!
There is substantial pull from Asia to ramp the conversation and
establish prediction markets. We can pick it up offline as usual.
Cordially,
-j
> The concern is that the prediction markets community already has one
> bomb-thrower.
What does the above statement mean?
> It is worth emphasizing the prediction markets group, consortium,
> libraries, Websites, events and so forth are open, non-commercial,
> not-for-profit.
So the prediction markets cluster and related events are non-profit?
- jill
Reply in CAPS.
-----Original Message-----
From: Predictio...@googlegroups.com
[mailto:Predictio...@googlegroups.com] On Behalf Of ajbaer5
Sent: Thursday, November 16, 2006 9:29 PM
To: Prediction Markets
Subject: Re: The Prediction Markets Stumble
Thanks for the info and clarification John.
> The concern is that the prediction markets community already has one
> bomb-thrower.
What does the above statement mean?
THERE EXISTS A ONE-TRICK-PONY THAT COVERS ONLINE GAMBLING COMPANIES MOSTLY
AND CALLS IT PREDICTION MARKETS. THE BLOG ONLY OFFERS TIGHTLY-WOUND PERSONAL
OPINION ON PREFERRED VENDORS. IT USES INVECTIVE AND A BOMBASTIC STYLE. THERE
IS A REJECTION OF OPEN COLLABORATION, COMMUNITY, THEORY, NON-GAMING PRACTICE
AND ALTERNATIVE PREDICTION MARKET OPINIONS.
> It is worth emphasizing the prediction markets group, consortium,
> libraries, Websites, events and so forth are open, non-commercial,
> not-for-profit.
So the prediction markets cluster and related events are non-profit?
YES, OF COURSE, CLUSTERS ARE NOT-FOR-PROFIT. THEY ARE OPEN, AGNOSTIC,
DISTRIBUTED AND POPULAR. NOTE: THE PREDICTION MARKETS RESEARCH NETWORK AND
CLUSTERS ARE DRIVEN BY PARTICIPANTS ONLY. AGENDA, THEMES, VENUES, ETC. ARE
OWNED BY PARTICIPANTS. GOVERNANCE ORIGINATIONS FROM COLLABORATION AND
COMMUNITY. THE PREDICTION MARKETS CLUSTERS ARE THE OPEN INDUSTRY NETWORK FOR
PREDICTION MARKETS. THEY ARE AN OPEN ACTION/RESEARCH COLLABORATION AMONG
SCHOLARS, VENDORS, ACADEMIA, TRADERS, USERS, DEVELOPERS, MARKETS AND
STAKEHOLDERS. THE GOAL IS TO PROVIDE AWARENESS, DIFFUSION AND ADOPTION OF
INFORMATION AND PREDICTION MARKETS PRACTICES, TOOLS AND THEORIES. SEE:
http://www.pmcluster.com/
- jill
Cheers,
-j
>From the day after the 2004 elections, Delaney & Cie. have boasted ad
nauseum that TradeSports "called" every race in that the market's
favorite won. That the lion's share of midterm betting and coverage
thereof has gone to TradeSports is a tribute to that boast.
It should be obvious to anyone whose dictionary contains the word
'probability' (or who can measure the distance between 75% and 100%)
that a favorite does not a winner make, but that misses the point.
TradeSports and lazy reporters have repeatedly defined success as the
market's favorite prevailing, and the public now reasonably follows the
same logic to declare the market a failure.
As with the North Korea missile fiasco, this has little to do with
theory of Prediction Markets and much to do with TradeSports shooting
themselves in the foot whenever they open their mouths.
-- Vancheeswaran
PS, I am eager to hear about the "independent" arbitration body that
TradeSports promised in this very forum three months ago. I would like
to chat with them about TradeSports' new hobby of closing contracts
before events have concluded.
Your remarks are spot-on.
Innovation and the diffusion of innovation takes a lot more that top experts
or folks with strong opinions. Innovation travels on common narrative, on
ordinary vocabulary, on building constructive relationships, creating value
and above all, authentic conversation.
Multiformity of thinking is critical to transformation. Tipping points
happen when there exists a broad, continuous, dynamic range of knowledge,
know-how and value, not just deep or arcane expertise.
Thus, for prediction markets, it is important to have the high-octane
experts, like we do today, as well as interpreters for the crowds, the
mainstream. That's the motivation for the remediation of the prediction
market Wikipedia page, for example.
The prediction markets community is fortunate to have many engaged, vocal
experts. Translation and collaboration may seem pedestrian to them, but the
open collaboration is essential to all stakeholders. It creates the network
patterns of uplift essential to prediction markets innovation and diffusion.
Cordially,
-j
-----Original Message-----
From: Predictio...@googlegroups.com
[mailto:Predictio...@googlegroups.com] On Behalf Of Peter
Sent: Wednesday, November 15, 2006 8:22 AM
To: Prediction Markets
Subject: Re: The Prediction Markets Stumble
In the last few years, a great deal of excitement has been generated by
"prediction markets." In these markets, people bet on future events,
such as the outcome of an election, the success of a movie, the capture
of Osama bin Laden, the unemployment rate, the success of a product, or
the likelihood that Iraq will be unified on January 1, 2009.
Prediction markets have been performing uncannily well, apparently
because they successfully aggregate the knowledge of widely dispersed
people. For example, the Iowa Electronic Markets have outperformed
polls in forecasting the outcomes of presidential elections. The
Hollywood Stock Exchange has taken some of the fun out of Oscar night,
because it usually predicts the winners (and it does well in
forecasting box office success as well). At high tech, pharmaceutical,
and finance companies around the globe, prediction markets are giving
accurate forecasts for product usage, dates of product availability,
office openings, and much more.
It is true that prediction markets must be structured to comply with
new legal restrictions on gambling and futures trading. But, with their
remarkable track record, prediction markets seem to be exceptionally
promising tools for predicting the performance of the economy, the
stability of nations, changes in environmental quality, and much more.
Many observers expected them to nail the outcomes of the midterm
elections as well.
It didn't happen. Days before the election, most prediction markets put
the probability of a Republican Senate at around 75 percent. Worse, the
market on Donald Rumsfeld's departure was trading at $1.52 out of a
maximum of $10.00 at TradeSports.com, an especially prominent
prediction market. Someone anticipating Rumsfeld's departure the Monday
a week before the election could have earned almost 600 percent in a
day.
As a result, many critics are now doubting the predictive capacity of
these new markets in the political domain and elsewhere. For example,
Reason's Katherine Mangu-Ward offered an apology to readers for
championing the markets in her reporting. At DailyKos, Markos Moulitsas
called the markets "b.s." and said they "blew this one." Atrios, too,
weighed in, with scorn for those who think "there's something magical
about market aggregated preferences."
But the doubts are unwarranted; they are based on serious confusion.
Even for the latest elections, the performance of prediction markets
has been truly impressive. On July 31, long before the pundits reached
a consensus, the prices on the Iowa Electronic Markets suggested that
the Democrats were likely to take the House of Representatives. The
Republicans had a brief market surge on September 16, but the Democrats
were seen as increasingly probable winners from October 9 until
Election Day. Even better, in every single Senate race at
TradeSports.com, the markets' overall favorite defeated his or her
opponent.
Still, the predictions markets did forecast a Republican Senate. Should
we dismiss them for getting that one so wrong?
If you think so, then you're probably forgetting the principles of
probability. Prediction markets do not make absolute predictions about
electoral outcomes, economic developments, product success, or anything
else. Instead, their predictions are mere probabilities. That's one
reason why the markets expected the Senate to stay in Republican hands,
even though so many Democratic candidates were favored. Senator-elect
Jim Webb had roughly a 60 percent probability of success, and
Senator-elect Bob Casey Jr. had roughly a 70 percent chance of success;
but these numbers do not suggest that both candidates were likely to
win simultaneously.
Prediction markets cannot be judged by asking whether a single
prediction turned out to be right. The real question is how their
probability judgments compare with the actual outcomes. And on this
count, prediction markets have a fantastic track record in many
domains, including elections, company performance, and economic
developments. If you look at the set of outcomes estimated to be 80
percent likely, about 80 percent of them happened; events estimated to
be 70 percent likely happen about 70 percent of the time; and so on.
This is what it means to say that prediction markets supply accurate
probabilities. No one thinks that all events whose probabilities are
above 50 percent should come true. That's most unlikely.
It's too early to say whether and when prediction markets will perform
poorly. For example, the markets said that John Roberts was unlikely to
be nominated for the Supreme Court and that Karl Rove would probably be
indicted--apparently because there was not a lot of dispersed
information out there about the likely judgments of President Bush and
Special Prosecutor Patrick Fitzgerald. But, for both the public and the
private sector, prediction markets remain an exceedingly valuable tool.
We're willing to bet on it.
Cass R. Sunstein is a contributing editor at The New Republic and a
professor at the University of Chicago. He is the author of Infotopia:
How Many Minds Produce Knowledge. Bo Cowgill is a researcher at Google
studying prediction markets.
http://www.tnr.com/doc.mhtml?i=w061120&s=cowgillsunstein112106
Perhaps we should add 'probability markets' to the taxonomy...
Cheers,
-j
-----Original Message-----
From: Predictio...@googlegroups.com
[mailto:Predictio...@googlegroups.com] On Behalf Of vancheeswaran
Sent: Tuesday, November 21, 2006 2:45 PM
To: Prediction Markets
Subject: Re: The Prediction Markets Stumble