Since sites like fivethirtyeight.com don't necessarily make their
historical estimates available in any detailed form, I would like to
record any interesting divergences over the next week here with the
help of others.
One divergence I noticed was in West Virginia, where 538 gave roughly
50% odds for Obama while Intrade had 40% from 10/16-10/17. Over that
weekend the incorporation of some new polls swung 538 all the way down
to 0-10% for Obama, agreeing with Intrade's direction. This morning
Intrade had Indiana as a toss-up while 538 had Obama near 60%. The
prices have now converged to 50% even though no new Indiana poll is
listed on 538. Maybe it was the result of an updated trend estimate,
or dare I say even a model tweak at this late stage.. I can't tell.
As I said, this is unscientific and wouldn't be necessary if we had
the historical data for all the poll-based models as we have for
markets. That opacity could enable modelers to cherry-pick success
stories and ignore their failures if no-one is watching closely.
> Since sites like fivethirtyeight.com don't necessarily make their > historical estimates available in any detailed form, I would like to > record any interesting divergences over the next week here with the > help of others.
> One divergence I noticed was in West Virginia, where 538 gave roughly > 50% odds for Obama while Intrade had 40% from 10/16-10/17. Over that > weekend the incorporation of some new polls swung 538 all the way down > to 0-10% for Obama, agreeing with Intrade's direction. This morning > Intrade had Indiana as a toss-up while 538 had Obama near 60%. The > prices have now converged to 50% even though no new Indiana poll is > listed on 538. Maybe it was the result of an updated trend estimate, > or dare I say even a model tweak at this late stage.. I can't tell. > As I said, this is unscientific and wouldn't be necessary if we had > the historical data for all the poll-based models as we have for > markets. That opacity could enable modelers to cherry-pick success > stories and ignore their failures if no-one is watching closely.
I think some of what you're seeing is due to two factors:
1- fivethirtyeight uses nearest-neighbor analysis, so a new poll in demographically similar state to WV will affect the WV forecast. From his FAQ:
"In plain English, we look at movement in the polling in recently- polled states and in national polls to predict movement in other states. For example, if there are new polls conducted in Massachusetts and Connecticut showing the Democratic candidate gaining 5 points, we can probably also infer that the candidate’s numbers have improved by about 5 points in Rhode Island."
2 - It might not look like a new poll, but it very well might be. Again, from the FAQ:
"What precisely is indicated by the 'date' reported in association with the poll? It will indicate the median date of interviewing for that poll -- not when that poll was reported or posted to the site. For example, a poll which conducted interviews on July 1, July 2 and July 3, and was reported to the media on July 5, would be listed with a date of July 2."
I agree, it would be very interesting to compare the two methods of forecasting by using detailed data from both fivethirtyeight and InTrade.
Yes, I saw the parts about nearest neighbors and regression
estimates.. that's part of what I meant by trend adjustment, both the
time-series and the cross-section. If we had access to historicals
the meaning of the date would have been more clear. Actually on
google cache you can see that it was the Zogby poll (10/25) released
today that did it.
On Oct 27, 8:41 pm, Jed Christiansen <jed.christian...@mercury-
rac.com> wrote:
> > Since sites like fivethirtyeight.com don't necessarily make their
> > historical estimates available in any detailed form, I would like to
> > record any interesting divergences over the next week here with the
> > help of others.
> > One divergence I noticed was in West Virginia, where 538 gave roughly
> > 50% odds for Obama while Intrade had 40% from 10/16-10/17. Over that
> > weekend the incorporation of some new polls swung 538 all the way down
> > to 0-10% for Obama, agreeing with Intrade's direction. This morning
> > Intrade had Indiana as a toss-up while 538 had Obama near 60%. The
> > prices have now converged to 50% even though no new Indiana poll is
> > listed on 538. Maybe it was the result of an updated trend estimate,
> > or dare I say even a model tweak at this late stage.. I can't tell.
> > As I said, this is unscientific and wouldn't be necessary if we had
> > the historical data for all the poll-based models as we have for
> > markets. That opacity could enable modelers to cherry-pick success
> > stories and ignore their failures if no-one is watching closely.
> I think some of what you're seeing is due to two factors:
> 1- fivethirtyeight uses nearest-neighbor analysis, so a new poll in
> demographically similar state to WV will affect the WV forecast.
> From his FAQ:
> "In plain English, we look at movement in the polling in recently-
> polled states and in national polls to predict movement in other
> states. For example, if there are new polls conducted in
> Massachusetts and Connecticut showing the Democratic candidate
> gaining 5 points, we can probably also infer that the candidate’s
> numbers have improved by about 5 points in Rhode Island."
> 2 - It might not look like a new poll, but it very well might be.
> Again, from the FAQ:
> "What precisely is indicated by the 'date' reported in association
> with the poll? It will indicate the median date of interviewing for
> that poll -- not when that poll was reported or posted to the site.
> For example, a poll which conducted interviews on July 1, July 2 and
> July 3, and was reported to the media on July 5, would be listed with
> a date of July 2."
> I agree, it would be very interesting to compare the two methods of
> forecasting by using detailed data from both fivethirtyeight and
> InTrade.
I've been reading Nate's blog for a while now, and he appears to be
pretty explicit when he makes even the most minor "tweaks" in the
model. Since things settled down after the conventions I don't think
he's made one since. (I think he wrote that having conventions and
their associated "bounces" so close together made the model go a
little wacky.)
Again, I completely agree that more data would be ideal.
> Yes, I saw the parts about nearest neighbors and regression
> estimates.. that's part of what I meant by trend adjustment, both the
> time-series and the cross-section. If we had access to historicals
> the meaning of the date would have been more clear. Actually on
> google cache you can see that it was the Zogby poll (10/25) released
> today that did it.
> On Oct 27, 8:41 pm, Jed Christiansen <jed.christian...@mercury-
> rac.com> wrote:
>>> Since sites like fivethirtyeight.com don't necessarily make their
>>> historical estimates available in any detailed form, I would like to
>>> record any interesting divergences over the next week here with the
>>> help of others.
>>> One divergence I noticed was in West Virginia, where 538 gave
>>> roughly
>>> 50% odds for Obama while Intrade had 40% from 10/16-10/17. Over
>>> that
>>> weekend the incorporation of some new polls swung 538 all the way
>>> down
>>> to 0-10% for Obama, agreeing with Intrade's direction. This morning
>>> Intrade had Indiana as a toss-up while 538 had Obama near 60%. The
>>> prices have now converged to 50% even though no new Indiana poll is
>>> listed on 538. Maybe it was the result of an updated trend
>>> estimate,
>>> or dare I say even a model tweak at this late stage.. I can't tell.
>>> As I said, this is unscientific and wouldn't be necessary if we had
>>> the historical data for all the poll-based models as we have for
>>> markets. That opacity could enable modelers to cherry-pick success
>>> stories and ignore their failures if no-one is watching closely.
>> I think some of what you're seeing is due to two factors:
>> 1- fivethirtyeight uses nearest-neighbor analysis, so a new poll in
>> demographically similar state to WV will affect the WV forecast.
>> From his FAQ:
>> "In plain English, we look at movement in the polling in recently-
>> polled states and in national polls to predict movement in other
>> states. For example, if there are new polls conducted in
>> Massachusetts and Connecticut showing the Democratic candidate
>> gaining 5 points, we can probably also infer that the candidate’s
>> numbers have improved by about 5 points in Rhode Island."
>> 2 - It might not look like a new poll, but it very well might be.
>> Again, from the FAQ:
>> "What precisely is indicated by the 'date' reported in association
>> with the poll? It will indicate the median date of interviewing for
>> that poll -- not when that poll was reported or posted to the site.
>> For example, a poll which conducted interviews on July 1, July 2 and
>> July 3, and was reported to the media on July 5, would be listed with
>> a date of July 2."
>> I agree, it would be very interesting to compare the two methods of
>> forecasting by using detailed data from both fivethirtyeight and
>> InTrade.
As of now there are large divergences in Arizona (Intrade: 19%, 538:
3%) and Georgia (Intrade: 26%, 538: 7%). Is this the result of 538
not being updated? If this is a fight over a couple points of error,
always being behind seems to matter. Tempting shorts though...
538 vs PMs - perhaps the two predictive models decouple as confidence increases around extreme odds because the PMs require extreme liquidity? for example, 538's algorithm could return 97% for McCain in AZ while the Betfair shows 80% because it takes 2 real opposing bettors to match at 90-10 odds, for example, and how many players do you need in the market at this point in order to find a 1-9 bettor against McCain in AZ?
On Mon, Nov 3, 2008 at 7:44 PM, Jason Ruspini <rusp...@gmail.com> wrote:
> As of now there are large divergences in Arizona (Intrade: 19%, 538: > 3%) and Georgia (Intrade: 26%, 538: 7%). Is this the result of 538 > not being updated? If this is a fight over a couple points of error, > always being behind seems to matter. Tempting shorts though...
Certainly the main difference was that the Intrade odds were less
confident near the extremes.
Using 9pm EST probabilities from Monday, for the presidential races by
state I get a mean absolute error of 5.6% for 538 and 10.4% for
Intrade. Even accounting for commissions, Nate Silver had a good
showing.
When considering the five states within a 30/70% band, the error is
44% for 538 and 42% for Intrade.
Using root mean square error instead does close the error gap, as that
method effectively removes the markets' penalty for the favorite/
longshot bias. I'm not sure that is justified though.
Going forward, capturing the public imagination with a range of
contracts, which won't necessarily have rich underlying datasets,
should be more important than a few percentage points of error.
