A good portion of the PM summit tomorrow in NY is likely to address the fact that the industry is now exploring and deploying many kinds of collective intelligence systems besides regular "trading" prediction markets. This shift has started years ago -- Hewlett-Packard started experimenting with * BRAIN* in 2003, and in 2005 NewsFutures was already fielding the first *Competitive Forecasting* applications with enterprise customers -- but one can feel while surveying the marketing pitches of various vendors that we've now reached a tipping point of sorts.
While unfortunately I won't be unable to join you in NY tomorrow -- Co-founder Maurice Balick will be NewsFutures incarnate and present some of our finest, latest work -- allow me to be part of the conversation by sharing a new paper that I wrote as a contribution for a forthcoming book on "Collective Wisdom". *This paper recaps the evidence for prediction market accuracy and examines the economic, mathematical, and neurological foundations of this form of collective wisdom. It concludes that the ultimate driver of accuracy is the betting proposition itself rather than any particular trading mechanism*: on the one hand, a wager attracts contrarians, which enhances the diversity of opinions that can be aggregated. On the other hand, the mere prospect of reward or loss promotes more objective, less passionate thinking, thereby enhancing the quality of the opinions that can be aggregated. *The take-away is that a prediction market is just one of many betting-based methods for aggregating forecasts.*In some contexts, for some purposes, it is an elegant solution, while in other situations it may be cumbersome. In any case, the ability of betting crowds to predict the future is a robust phenomenon that doesn’t seem to be bottled up in any one particular contraption.
> [My] paper ... concludes that the ultimate driver of accuracy is the > betting proposition itself rather than any particular trading > mechanism: on the one hand, a wager attracts contrarians, which > enhances the diversity of opinions that can be aggregated. On the > other hand, the mere prospect of reward or loss promotes more > objective, less passionate thinking, thereby enhancing the quality > of the opinions that can be aggregated. The take-away is that a > prediction market is just one of many betting-based methods for > aggregating forecasts.
I've said (http://www.overcomingbias.com/2006/12/ the_80_forecast.html): "I'd guess you can get 80% of the improvement that predict markets offer by using a much simpler solution: collect [forecast] track records." So, yes, the mere prospect that someone may later check your forecasts can encourage you to think more objectively.
And yes, we can do even better if people can be selective about what they forecast how confidently. Yes, contrarians can better correct consensus errors when they can identify where they most disagree with a consensus, and can then choose to emphasize this disagreement just when they feel the most confident in it.
In my talk Friday (http://hanson.gmu.edu/ppt/Combo.ppt) I emphasized the issue of transparency. Even when users have nice simple and graphical tools to help them express their opinions, it is important that users can see clearly both how those opinions will influence the consensus, and how those opinions will influence their final score.
Decades of theory and experiment in economic mechanism design has shown that one can't usually just assume that users facing a complex mechanism will just tell the truth if asked. So I worry that it is not viable long term to just tell users, "tell us what you think and we'll do a good job of combining everyone's opinion into a consensus, and of evaluating everyone's contributions; no need to worry your pretty little head about that."
When this field was dominated more by academics, they tended more to publish suggested mechanisms for all to see, and to argue for or against mechanisms on the basic of formal proofs or controlled lab experiments. Now that the field is dominated more by private vendors, we see more propriety mechanisms, unavailable for study by critics, and offered without supporting formal proofs or controlled experiments. We instead see more proof by demo and icons; customers are asked to believe that a nice pretty screen, or a long list of prior clients, implies their mechanisms are good.
Perhaps this is an inevitable progression, but it still saddens me.
Robin Hanson http://hanson.gmu.edu Res. Assoc., Future of Humanity Inst., Oxford Univ. Assoc. Professor, George Mason University MSN 1D3, Carow Hall, Fairfax VA 22030 703-993-2326 FAX: 703-993-2323
[Seems my last message was mangled; trying again.]
On Apr 23, 2009, Emile Servan-Schreiber wrote:
> [My] paper ... concludes that the ultimate driver of accuracy is the > betting proposition itself rather than any particular trading > mechanism: on the one hand, a wager attracts contrarians, which > enhances the diversity of opinions that can be aggregated. On the > other hand, the mere prospect of reward or loss promotes more > objective, less passionate thinking, thereby enhancing the quality > of the opinions that can be aggregated. The take-away is that a > prediction market is just one of many betting-based methods for > aggregating forecasts.
I've said (http://www.overcomingbias.com/2006/12/ the_80_forecast.html ) "I'd guess you can get 80% of the improvement prediction markets offer by using a much simpler solution: collect [forecast] track records." So, yes, the mere prospect that someone may later check your forecasts can encourage you to think more objectively.
And yes, we can do even better if people can be selective about what they forecast how confidently. Yes, contrarians can better correct consensus errors when they can identify where they most disagree with a consensus, and can then choose to emphasize this disagreement just when they feel the most confident in it.
In my talk Friday (http://hanson.gmu.edu/ppt/Combo.ppt) I emphasized the issue of transparency. Even when users have nice simple and graphical tools to help them express their opinions, it is important that users can see clearly both how those opinions will influence the consensus, and how those opinions will influence their final score.
Decades of theory and experiment in economic mechanism design has shown that one can't usually just assume that users facing a complex mechanism will just tell the truth if asked. So I worry that it is not viable long term to just tell users, "tell us what you think and we'll do a good job of combining everyone's opinion into a consensus, and of evaluating everyone's contributions; no need to worry your pretty little head about that."
When this field was dominated more by academics, they tended more to publish suggested mechanisms for all to see, and to argue for or against mechanisms on the basic of formal proofs or controlled lab experiments. Now that the field is dominated more by private vendors, we see more propriety mechanisms, unavailable for study by critics, and offered without supporting formal proofs or controlled experiments. We instead see more proof by demo and icons; customers are asked to believe that a pretty screen or a long list of prior clients, implies their mechanisms are good.
Perhaps this is an inevitable progression, but it still saddens me.
Robin Hanson http://hanson.gmu.edu Res. Assoc., Future of Humanity Inst., Oxford Univ. Assoc. Professor, George Mason University MSN 1D3, Carow Hall, Fairfax VA 22030 703-993-2326 FAX: 703-993-2323
On Mon, Apr 27, 2009 at 8:02 PM, Robin Hanson <rhan...@gmu.edu> wrote:
> On Apr 23, 2009, Emile Servan-Schreiber wrote:
> *[My] paper ... concludes that the ultimate driver of accuracy is the > betting proposition itself rather than any particular trading mechanism*: > on the one hand, a wager attracts contrarians, which enhances the diversity > of opinions that can be aggregated. On the other hand, the mere prospect of > reward or loss promotes more objective, less passionate thinking, thereby > enhancing the quality of the opinions that can be aggregated. *The > take-away is that a prediction market is just one of many betting-based > methods for aggregating forecasts.*
It has never been clear how you derive that 80% figure. In my paper, I cite evidence that some non-trading betting schemes can fully match the accuracy of a prediction market. Here is the relevant paragraph:
"Several non-market betting schemes have also been shown to perform just as well as prediction markets when matched head to head. For instance, Chen et al (2005) compared the prediction accuracy of NewsFutures and Tradesports to the *simple averages of predictions* elicited from 1966 individuals regarding the outcomes of 210 American football games. Those predictions were scored by the quadratic scoring rule, one of the so-called “proper” scoring rules designed elicit honest forecasts. Although the collective intelligence emerged forcefully in each case – outperforming all but a few individuals – there was no advantage for the markets over a simple arithmetic average of the individual predictions. Limited laboratory experiments have shown that when the number of people in a crowd is particularly small, say around a dozen, some simple betting schemes may even beat the market (Chen et al, 2003)."
References: 1. Chen, K.Y., Fine, L.R., & Huberman, B.A. (2003). Predicting the future. Information Systems Frontiers 5(1) 2. Chen, Y., Chu, C.H., Mullen, T., & Pennock, D.M (2005). Information Markets vs. Opinion Pools: An Empirical Comparison. Proc. of the 6th ACM Conference on Electronic Commerce (EC), Vancouver, BC Canada, June 2005.
> When this field was dominated more by academics, they tended more to > publish suggested mechanisms for all to see, and to argue for or against > mechanisms on the basic of formal proofs or controlled lab experiments. Now > that the field is dominated more by private vendors, we see more propriety > mechanisms, unavailable for study by critics, and offered without supporting > formal proofs or controlled experiments. We instead see more proof by demo > and icons; customers are asked to believe that a long list of prior clients, > implies their mechanisms are good.
> Perhaps this is an inevitable progression, but it still saddens me.
You are right that the proliferation of un-tested mechanisms is bothersome. It leaves an open door to snake oil salesmen... That said, selling PMs as easy to design and run in a corporate context also comes close to selling snake oil: "Set it up in minutes! Add a few fun markets! Guaranteed liquidity with our market maker! Free pilot!" Quite a few have been seduced into failure by such simplistic marketese that ignores the complex reality of engaging actual people.
What those of us in the field who don't live by formal proof and controlled experiments have learned is that no matter how formally excellent your market mechanisms is, if not enough people participate then the project is considered a failure. It is considered a failure even if the predictions are accurate, simply because no one will care. Coming up with simpler input mechanisms has helped lower the barrier to participation, which has led to greater acceptance within organizations, and it's really not clear that we've lost any accuracy in the process.
What is needed now is not so much more formal proofs or laboratory experiments on a few undergrads. These are largely irrelevant to what will or won't work in an enterprise or mass-market setting. What is needed is more field studies that embrace reality. When we did just that a few years back comparing real-money and play-money *in the field*, the results took the formal-proof-and-lab-experiments gang by surprise. The Chen et al (2005) study cited above is also surprising. What other surprises await us out there in the real world?
-- Emile Servan-Schreiber CEO, NewsFutures Inc. http://www.newsfutures.com Tel US: +1 (443) 321-2700 Tel EU: +336 1804 3404 Fax: +1 (978) 383-1065 Email: e...@newsfutures.com
On Apr 27, 2009, at 4:56 PM, Emile Servan-Schreiber wrote:
> I've said (http://www.overcomingbias.com/2006/12/ > the_80_forecast.html): "I'd guess you can get 80% of the improvement > prediction markets offer by using a much simpler solution: collect > [forecast] track records." So, yes, the mere prospect that someone > may later check your forecasts can encourage you to think more > objectively.
> It has never been clear how you derive that 80% figure. In my paper, > I cite evidence that some non-trading betting schemes can fully > match the accuracy of a prediction market.
Sure simple opinion averages often work very well. But they can do badly when info is very unequally distributed among participants. See: An Experimental Test of Combinatorial Information Markets, with John Ledyard, Takashi Ishikida, Journal of Economic Behavior and Organization 69:182-189, 2009.
> When this field was dominated more by academics, they tended more to > publish suggested mechanisms for all to see, and to argue for or > against mechanisms on the basic of formal proofs or controlled lab > experiments. Now that the field is dominated more by private > vendors, we see more propriety mechanisms, unavailable for study by > critics, and offered without supporting formal proofs or controlled > experiments. We instead see more proof by demo and icons; customers > are asked to believe that a long list of prior clients, implies > their mechanisms are good.
> Perhaps this is an inevitable progression, but it still saddens me.
> You are right that the proliferation of un-tested mechanisms is > bothersome. It leaves an open door to snake oil salesmen... That > said, selling PMs as easy to design and run in a corporate context > also comes close to selling snake oil: "Set it up in minutes! Add a > few fun markets! Guaranteed liquidity with our market maker! Free > pilot!" Quite a few have been seduced into failure by such > simplistic marketese that ignores the complex reality of engaging > actual people.
Yes, creating useful markets (or reasonable substitute mechanisms) takes a lot more work than setting up some software, and it is deceptive to imply otherwise.
> What those of us in the field who don't live by formal proof and > controlled experiments have learned is that no matter how formally > excellent your market mechanisms is, if not enough people > participate then the project is considered a failure. It is > considered a failure even if the predictions are accurate, simply > because no one will care. Coming up with simpler input mechanisms > has helped lower the barrier to participation, which has led to > greater acceptance within organizations, and it's really not clear > that we've lost any accuracy in the process.
As I've said, I have no complaint with simple ways to express opinions; my complaint above was about opaque mechanisms to score and combine those opinions. How much participation is needed for success depends in part on whether the customer sees themselves as having been sold a "wisdom of crowds" morale booster or an info aggregator estimator.
> What is needed now is not so much more formal proofs or laboratory > experiments on a few undergrads. These are largely irrelevant to > what will or won't work in an enterprise or mass-market setting. > What is needed is more field studies that embrace reality.
Well field studies are great when we can get them, but we also need models and lab tests too, as those are usually cheaper and offer more control.
Robin Hanson http://hanson.gmu.edu Res. Assoc., Future of Humanity Inst., Oxford Univ. Assoc. Professor, George Mason University MSN 1D3, Carow Hall, Fairfax VA 22030 703-993-2326 FAX: 703-993-2323
[Mac Mail is killing me - keeps clipping my text. Trying again.]
On Apr 27, 2009, at 4:56 PM, Emile Servan-Schreiber wrote:
> I've said (http://www.overcomingbias.com/2006/12/ > the_80_forecast.html): "I'd guess you can get 80% of the improvement > prediction markets offer by using a much simpler solution: collect > [forecast] track records." So, yes, the mere prospect that someone > may later check your forecasts can encourage you to think more > objectively.
> It has never been clear how you derive that 80% figure. In my paper, > I cite evidence that some non-trading betting schemes can fully > match the accuracy of a fully match the accuracy of a prediction > market.
Sure simple opinion averages often work very well. But they can do badly when info is very unequally distributed among participants. See: An Experimental Test of Combinatorial Information Markets, with John Ledyard, Takashi Ishikida, Journal of Economic Behavior and Organization 69:182-189, 2009.
> You are right that the proliferation of un-tested mechanisms is > bothersome. It leaves an open door to snake oil salesmen... That > said, selling PMs as easy to design and run in a corporate context > also comes close to selling snake oil: "Set it up in minutes! Add a > few fun markets! Guaranteed liquidity with our market maker! Free > pilot!" Quite a few have been seduced into failure by such > simplistic marketese that ignores the complex reality of engaging > actual people.
Yes, creating useful markets (or reasonable substitute mechanisms) takes a lot more work than setting up some software, and it is deceptive to imply otherwise.
> What those of us in the field who don't live by formal proof and > controlled experiments have learned is that no matter how formally > excellent your market mechanisms is, if not enough people > participate then the project is considered a failure. It is > considered a failure even if the predictions are accurate, simply > because no one will care. Coming up with simpler input mechanisms > has helped lower the barrier to participation, which has led to > greater acceptance within organizations, and it's really not clear > that we've lost any accuracy in the process.
As I've said, I have no complaint with simple ways to express opinions; my complaint above was about opaque mechanisms to score and combine those opinions. How much participation is needed for success depends in part on whether the customer sees themselves as having been sold a "wisdom of crowds" morale booster or an info aggregator estimator.
> What is needed now is not so much more formal proofs or laboratory > experiments on a few undergrads. These are largely irrelevant to > what will or won't work in an enterprise or mass-market setting. > What is needed is more field studies that embrace reality.
Well field studies are great when we can get them, but we also need models and lab tests too, as those are usually cheaper and offer more control.
Robin Hanson http://hanson.gmu.edu Res. Assoc., Future of Humanity Inst., Oxford Univ. Assoc. Professor, Georg 1D3, Carow Hall, Fairfax VA 22030 703-993-2326 FAX: 703-993-2323
