Posted by: structureofnews | November 28, 2016

Mea Culpa – And To Making Better Models

sotn-1This is more than a little belated, so apologies for that.

So the polls got it wrong.

Hillary Clinton did not, as it turns out, have a 75% or 80% or 90% or 99% chance of winning the election, as many of us – including our massive States of the Nation project – predicted up until Election Day.

(You’ll see that we’ve now updated our analysis to correct for an accurate understanding of the actual turnout, which shows that Donald Trump’s odds of success would have been 75%.  Kinda late, it’s true, but simply validating the underlying assumptions behind the project.)

So what happened?  And given Brexit, and Colombia, what does this mean for polls more broadly?  And how do we do better in the future?

(A shout out/caveat: I’m cribbing heavily from Mo Tamman‘s smart analysis of what went wrong in the polling this year.  Here’s also another nice, nuanced piece about where polls went wrong.)

To be fair, the polls weren’t actually that far off.  Seriously.  At least the national polls. They showed Clinton leading by a couple of percentage points, and as the final results trickle in, indeed the former Secretary of State is leading in the popular vote. It’s true that the polls may have overstated her appeal, but broadly the numbers are within the margin of error, especially if you assume the likely voter models are somewhat off.

Which they were.  And that’s one critical place all the polls fell down on.  And again, to be fair, predicting who will or won’t go to the polls – a once every two- or four-year event – is a tricky exercise at best.  But it doesn’t help that we tend to present the numbers we come up with as absolutes, rather than give a range of possible outcomes based on a range of possible turnout models.  As we noted when we launched the States of the Nation:

But most polling failures have come about less because the pollsters misunderstood voter preferences and more because the pollsters failed to accurately estimate turnout.

(Which, we hope, vindicates our decision to let users experiment with various turnout models in the States of the Nation project to see how the election would play out in the all-important Electoral College.)

But there’s a bigger, more mathematical reason why all the percentage-chance-of-winning models fell down.  Mo again:

In part, this is because polling analysts got the central metaphor wrong.

U.S. presidents are chosen not by the national popular vote, but in the individual Electoral College contests in the 50 states and Washington D.C. In calculating probable outcomes, election predictors generally treated those 51 contests as completely separate events – as unrelated to one another as a series of 51 coin tosses.

But that’s not how elections work in the United States. Voting trends that appear in one state – such as a larger-than-expected Republican shift among rural voters – tend to show up in other states with similar demographic make-ups.

And that’s what happened Tuesday: The election models calculated the probabilities of a Clinton win that turned out to be high, because they viewed each state too much in isolation.

In other words, the model was based on a flawed assumption – that state-level results were independent of each other, and hence that multiple simulations of state-level elections would provide statistically valid results.  As it turns out, they weren’t and they wouldn’t – and we should probably have figured that out as a common-sense reality check in the first place.

And those are probably the two best lessons to draw from the polling: One, to be less definitive about probabilities and give readers/users more information about other possible outcomes (or the ability to game them out themselves.)  And two, to really stress-test our models to understand all their strengths, weaknesses and shortcomings.

That’s all the more important as we use increasingly sophisticated tools in data journalism, many with inherent assumptions built into them. It isn’t simply that we should be transparent about our methods – and let’s face it, how many people actually read the nerd box that accompanies these stories? – but that we should actively challenge our assumptions and models well before we start building or gathering data.

That wouldn’t have changed the results of the vote, of course; but it may have given a much better heads up that this wasn’t a runaway election by any stretch.


Responses

  1. I actually read this.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: