Unscientific fivethirtyeight.com / Intrade et al divergence thread

[Jason Ruspini]

Since sites like fivethirtyeight.com don't necessarily make their
historical estimates available in any detailed form, I would like to
record any interesting divergences over the next week here with the
help of others.

One divergence I noticed was in West Virginia, where 538 gave roughly
50% odds for Obama while Intrade had 40% from 10/16-10/17. Over that
weekend the incorporation of some new polls swung 538 all the way down
to 0-10% for Obama, agreeing with Intrade's direction. This morning
Intrade had Indiana as a toss-up while 538 had Obama near 60%. The
prices have now converged to 50% even though no new Indiana poll is
listed on 538. Maybe it was the result of an updated trend estimate,
or dare I say even a model tweak at this late stage.. I can't tell.
As I said, this is unscientific and wouldn't be necessary if we had
the historical data for all the poll-based models as we have for
markets. That opacity could enable modelers to cherry-pick success
stories and ignore their failures if no-one is watching closely.

[Jed Christiansen]

I think some of what you're seeing is due to two factors:

1- fivethirtyeight uses nearest-neighbor analysis, so a new poll in
demographically similar state to WV will affect the WV forecast.
From his FAQ:

"In plain English, we look at movement in the polling in recently-
polled states and in national polls to predict movement in other
states. For example, if there are new polls conducted in
Massachusetts and Connecticut showing the Democratic candidate
gaining 5 points, we can probably also infer that the candidate’s
numbers have improved by about 5 points in Rhode Island."

2 - It might not look like a new poll, but it very well might be.
Again, from the FAQ:

"What precisely is indicated by the 'date' reported in association
with the poll? It will indicate the median date of interviewing for
that poll -- not when that poll was reported or posted to the site.
For example, a poll which conducted interviews on July 1, July 2 and
July 3, and was reported to the media on July 5, would be listed with
a date of July 2."

I agree, it would be very interesting to compare the two methods of
forecasting by using detailed data from both fivethirtyeight and
InTrade.

Best regards,
Jed

----------
Jed D. Christiansen
Managing Director
Mercury Research and Consulting Ltd
http://www.mercury-rac.com
---
jed.chri...@mercury-rac.com
+44 (0) 796 358 3663
http://blog.mercury-rac.com
http://www.linkedin.com/in/jedchristiansen
---

[Jason Ruspini]

Yes, I saw the parts about nearest neighbors and regression
estimates.. that's part of what I meant by trend adjustment, both the
time-series and the cross-section. If we had access to historicals
the meaning of the date would have been more clear. Actually on
google cache you can see that it was the Zogby poll (10/25) released
today that did it.

On Oct 27, 8:41 pm, Jed Christiansen <jed.christian...@mercury-

> jed.christian...@mercury-rac.com+44 (0) 796 358 3663http://blog.mercury-rac.comhttp://www.linkedin.com/in/jedchristiansen
> ---

[Jed Christiansen]

I've been reading Nate's blog for a while now, and he appears to be
pretty explicit when he makes even the most minor "tweaks" in the
model. Since things settled down after the conventions I don't think
he's made one since. (I think he wrote that having conventions and
their associated "bounces" so close together made the model go a
little wacky.)

Again, I completely agree that more data would be ideal.

Best regards,
Jed

----------
Jed D. Christiansen
Managing Director

Mercury Research and Consulting Ltd
http://www.mercury-rac.com
---
jed.chri...@mercury-rac.com
+44 (0) 796 358 3663
http://blog.mercury-rac.com
http://www.linkedin.com/in/jedchristiansen
---

[Jason Ruspini]

As of now there are large divergences in Arizona (Intrade: 19%, 538:
3%) and Georgia (Intrade: 26%, 538: 7%). Is this the result of 538
not being updated? If this is a fight over a couple points of error,
always being behind seems to matter. Tempting shorts though...

[Joe Seither]

538 vs PMs - perhaps the two predictive models decouple as confidence increases around extreme odds because the PMs require extreme liquidity?

for example, 538's algorithm could return 97% for McCain in AZ while the Betfair shows 80% because it takes 2 real opposing bettors to match at 90-10 odds, for example, and how many players do you need in the market at this point in order to find a 1-9 bettor against McCain in AZ?

[Jason Ruspini]

Certainly the main difference was that the Intrade odds were less
confident near the extremes.

Using 9pm EST probabilities from Monday, for the presidential races by
state I get a mean absolute error of 5.6% for 538 and 10.4% for
Intrade. Even accounting for commissions, Nate Silver had a good
showing.

When considering the five states within a 30/70% band, the error is
44% for 538 and 42% for Intrade.

Using root mean square error instead does close the error gap, as that
method effectively removes the markets' penalty for the favorite/
longshot bias. I'm not sure that is justified though.

Going forward, capturing the public imagination with a range of
contracts, which won't necessarily have rich underlying datasets,
should be more important than a few percentage points of error.