The Challenge with Challenges
John Maloney
Hi –
Noteworthy blog.
http://workingwikily.net/?p=1015
Q: I’m curious to know if you’ve had any experience with prediction markets. Do you think they have applications for social-sector organizations?
I look at prediction markets as a kind of cognition example of collective intelligence; how does one aggregate the analytical pieces of knowledge that need to contribute to a conclusion of some kind? Do we just vote on it? Do we take a poll? Do we average across everybody’s responses or find the midpoint? I happen to be more intrigued than opposed — I think [this approach] aggregates knowledge in a very clever way, more accurate than just taking the average.
One of the areas I’ve seen [prediction markets] being used is in predicting legislative outcomes, so there could be political applications. They’re quite adaptable.
The challenge with them is the challenge with challenges: articulating the “security” to be traded. I’ve found that until people have been through it a couple times and are trained to think that way, they keep writing polls, which won’t work if you’re trying to get a true prediction out of it.
What characterizes a good security is that there’s an unambiguous strike price — which has two components: the pricing mechanism and the act or event that determines when the pricing kicks in. It’s not trivial, although I think there are some pretty good rules for writing good securities or deciding that this isn’t a good candidate for a prediction market.
We ran a closed study one time in which we predicted scientific outcomes that were complex, such as FDA approval at the end or in the earlier stages of the process. Even though that might sound like a black and white technical question, at the time you’re predicting it you’re asking a lot of questions about which you have a lot of knowledge but where the knowledge in each of those areas isn’t perfect and the way in which all of those areas fit together to determine an outcome isn’t clear.
Written by Noah Flower
-j