Nate Cohn Explains What the Polls Got Wrong

Last week, Joe Biden was elected the forty-sixth President of the United States, defeating Donald Trump by what is likely to be several percentage points nationwide. The margins in crucial swing states, however, were closer than polling averages suggested that they would be. Though Biden won the three Midwest battlegrounds—Michigan, Wisconsin, and Pennsylvania—where Trump eked out victories in 2016, and the former Vice-President is likely to emerge victorious in other previously red states such as Georgia and Arizona, major networks took several days to call the election results, and what looked like a landslide for much of the campaign ended in another scramble for electors.

I recently spoke by phone with Nate Cohn, a domestic correspondent at the New York Times who spearheaded the newspaper’s polling this cycle. (Full disclosure: Cohn and I worked together at The New Republic, and are close friends.) During our conversation, which has been edited for length and clarity, we discussed how the pandemic affected polling, the role of data in election coverage, and the Times’ contentious “election needle.”

What was the biggest surprise to you when the results came in, or as they have come in?

The Hispanic vote. The swing towards Trump in Hispanic areas across the country is extraordinary. It was hinted at in the preëlection polls. The polls always showed the President faring better among nonwhite, and particularly Hispanic, voters than he did four years ago, but the magnitude of the shift was way beyond expectations. We learned that early in the night in Miami-Dade County, where no one had the President doing as well as he did. And it has proven true, as far as I can tell, basically everywhere in the country among Latino voters, to varying degrees. It’s true down-ballot. It’s not like this was just about the President. And I think it’s a huge and important political story.

And the second thing that really surprised me is the white, rural, Midwestern vote. The preëlection polls said that Joe Biden was doing much better than Hillary Clinton was four years ago among white voters without a degree. And those gains simply did not materialize. The results looked quite a bit like 2016 across most of rural America, and there were many areas where Donald Trump did better in white working-class areas than he did in 2016.

Because the definition of “Hispanic” seems to change from poll to poll, and because exit polls are unreliable, I had trouble finding the exact numbers for what percentage of Hispanic voters Trump won in 2016 and what percentage he won this time. What is your sense of the size of that change across the country?

I think it could easily be a double-digit swing in the President’s direction. I have not crunched these numbers conclusively, and it’s still too early to do that. But that would be my initial gut sense, yes.

The most obvious reason for this, I would assume, is the education divide in our politics manifesting itself across racial lines. Do you have a different theory?

I think that’s right. And I think this was not an election on immigration. Immigration was a major theme of the 2016 election, from Donald Trump’s announcement that he was seeking the Presidency all the way to the policy proposal to build a wall. The President’s position on immigration and his attitude toward Mexican-Americans were a central theme of that campaign. And I would say that it has not been a central theme of this Presidency, and it was certainly not a central theme of the 2020 campaign. So it makes sense to me that if we stop talking immigration and Hispanic voters start assessing the President without that in mind, that they might begin to shift in ways that are fairly similar to demographically similar white voters, but four years later. That strikes me as a possibility.

And it would also perhaps suggest that it was more Hispanic men than Hispanic women, correct?

There is a lot of preëlection survey data that suggests that the Hispanic gender gap got very, very large in this election.

It’s interesting you said that it wasn’t a big theme of the campaign or his Presidency. It feels to me like it was a big theme of this Presidency, but not so much the reëlection campaign.

If you were to choose a year that was dominated by immigration during his Presidency, what year would that be?

I think 2018. Family separation and then the caravan.

Yeah. I think that’s right. I think there was a period in late 2018 where immigration was at the forefront of our politics. I would absolutely say that was untrue of 2020, and I think it was also untrue of a lot of the Trump Administration. And I would say that immigration was always near the top in 2016.

In a lot of preëlection polling, Trump was running ahead of Republican Senate candidates. But Republicans seemed to do better than Trump in the generic House vote, and in some of these Senate races. Why do you think that was?

I don’t know why that turned out to be the case. I think one of the most interesting parts of the polling error this year is that it was greater down-ballot than it was as at the top of the ticket, and in 2016 it was the opposite. And so while a lot of our explanation for what went wrong in 2016 was searching for things that were mainly about the President, I’m not sure that the pollsters would be right to suppose that this year’s polling error is unique or specific to the President.

In 2016, the national polls were off a couple points, and then Midwestern polls, especially polls that didn’t weight by education, were off more. This year with your polling for the New York Times, with other polling, there were a lot of misses. And these were the polls that really took weighting by education seriously and took a lot of steps after 2016 to fix errors, and in your case nailed the 2018 midterms. Do you know what happened this time?

I don’t. I can offer you some theories.

Yeah, please.

But, before I say that, I do want to agree that this was a much bigger polling miss, in important ways, than in 2016. It was a bigger polling miss in the national surveys. It was a bigger polling miss for the industry’s most prominent and pricey survey houses. The state polling error will be just as bad, even though, as you mentioned, many state pollsters took steps to increase the number of white voters without a degree in their surveys. And state polls look a lot like they did in 2016.

But, if the state polls are just as bad as they were in 2016, despite steps that we know improved the President’s standing in the surveys, we can say with total confidence—and I know this was true in our data—that the underlying survey data has to be worse than it was in 2016. Or, if you prefer, if all the pollsters were using the 2016 methodology, the polls would have been far worse this year than they were in 2016. And that is really interesting. As I said, I can list a bunch of theories for you.

Yeah. What are your theories?

Well, the key framing is what’s changed since 2016. What would make the polls worse now than they were then? So one possibility is that it’s four more years of Trump, and that as American politics grew more and more defined along the lines of your attitudes about the President, and as old political allegiances sort of fell away, that non-response bias in polling became more and more correlated with Presidential vote choice.

Another possibility is that “the resistance” is what broke the polls. Think of all of the political engagement on the left, the millions of dollars that were spent to help Jon Ossoff in 2017 or to help Jamie Harrison in 2020. This portrays a tremendous increase in the level of political participation on the part of progressives. We know that politically engaged voters are more likely to respond to surveys. And so it may be that as the Trump Presidency has totally energized the Democratic base, it has also led those same kinds of voters to increase their propensity to respond to political surveys.

Another possibility is the high turnout this year. We in the polling world have tended to assume that higher turnout makes polling easier, because we think of turnout as something that’s an additional variable that the polls have to get right beyond just taking a nice sample of the population. This year, though, we have this huge increase in turnout, and most people have supposed that it was good for Joe Biden. Maybe it was good for Joe Biden. But I think we also have to be open to the idea that it was not good for Joe Biden. In Florida, where we were collecting turnout data live on Election Day, I can tell you with certainty that the electorate was more Republican than it was in 2016, more than our polls projected, no matter your likely voter methodology. That may be true elsewhere in the country. I don’t know. It may not be. We just don’t have that data yet.