Why are the Polls So Weird?

Written by Michael Leon on 4 November 2024

Updated on November 19, 2024

If you’ve been keeping up with this election cycle at all, you may have noticed something strange. In the battleground states, pollsters have been reporting the same thing for months: the candidates are within just about 3 points of each other in each and every one of them. Now, this isn’t incredibly shocking, many predicted that this might be a tight race. The bizarreness comes when you look at the variance of these polls. As an example, let’s take a look at Pennsylvania:

Using data from fivethirtyeight, in the past month Harris has seen a high of 48.1% and a low of 47.4% in the polls on average. That’s a difference of just 0.7%. Trump on the other hand has similarly seen a low of 47.2% and a high of 48.0%. The polls are not budging. Compare this with 2020’s race where Biden had a swing of 1.1% and Trump had a swing of 1.9% over the same time frame.

We can get into the nitty-gritty with this data as well. In the month of October, which also happens to be one of the most pivotal months for any campaign, Harris had a variance of just 0.04 percentage points day-to-day, while Trump had a variance of 0.08 percentage points. This is a difference of less than a tenth of a percentage point per day. Comparing this with 2020, where Biden had a variance of 0.12 percentage points day to day, while Trump had a variance of 0.26. The difference is striking. It is even more pronounced when we look at all the polls being produced. Let’s have a look at how the polling in swing states this election cycle compares to how it looked just 4 years ago.

Variance and Range are calculated by looking at the Net Results of all polls produced in the last month of the respective election cycles.

	AZ	GA	MI	NC	NV	PA	WI	Avg
Var 2020	9.1	11.6	16.6	6.3	10.0	9.5	14.2	11.0
Var 2024	5.3	4.9	6.5	4.1	8.3	4.3	3.2	5.2
Range 2020	14	13	23	12	10	17	17	15.1
Range 2024	10	13	12	8	15	10	18	10.9
Avg Poll Size 2020	1010	1258	1108	1126	1284	1204	943	1133
Avg Poll Size 2024	917	1098	927	1058	783	1185	871	977

On average across all seven swing states, variance decreased by over 52% between these two cycles, while the range decreased by just over 28%.

Now, this is a lot of numbers, so let’s unpack what all of this means. First, looking at variance. This is a measure of, on average, how far values are from the mean. The fact that variance has dropped so much tells us that pollsters are obtaining very similar results across the board. The next is range, which tells us our maximums on the spread of the poll results. The fact that this has dropped, too, further indicates that our data points are increasingly clustered together. In fact, as with any presidential race, we would expect the range to be quite high, to account for different polling methodologies, different survey groups, etc. Standard deviation is likewise a measure of the spread of our data. All of these signs point to polling data getting more and more clustered together.

To couple with this, the average number of voters, likely or registered, polled in these two cycles has actually dropped across the board in battleground states, on average by just shy of 14%, from 1133 to 977. With smaller sample sizes, one would expect to see more outliers in the overall range of results, not less. The fact that up until the buzzer all seven battleground states aren’t budging has some statisticians and journalists skeptical, to say the least.

Take Nate Silver, a statistician and journalist for the New York Times. In a recent opinion piece, Silver says:

“What’s most questionable is how an extremely high percentage of the polls in the swing states — something like 75 or 80 percent of them — have shown about a 2.5-point race or narrower. Statistically, that’s nearly impossible. You’re supposed to get more outliers when you’re only sampling 800 people at a time. So other than the New York Times/Siena polls, which have consistently published results that deviate from the consensus, I’m not sure we’re getting pollsters’ real opinions anyway.”

So what else is going on? Is it that pollsters are playing it conservative? Have their models changed? I would like to do my best to avoid spinning conspiracy theories, as many on the internet have already taken to that, claiming that pollsters are scared of career repercussions or seeming like they're endorsing a certain candidate (look at the backlash after the recent Iowa poll which had Harris over Trump by 3 points), but there may be a couple plausible explanations.

Poll Herding

“Poll Herding” refers to pollsters using existing data and polls to adjust their models and predictions to better fit with their notions of what the poll data should be. This could mean many things, from removing outliers from the survey results to weighting votes according to what other polls are showing or what other data is showing. A chief example of this is weighting votes according to census data. If, for example, you know that 45% of likely voters in the area are women according to the 2020 census, yet they make up 60% of your sample size, then you might unweight their votes, counting each as 0.9 or 0.8 votes, to compensate for this supposed over representation.

This becomes more dangerous when basing models on pre-existing poll data. This, in practice, lends itself very easily to a feedback loop, where nobody wants to be too wrong so every pollster produces the same results.

Ann Selzer, whose firm is behind the aforementioned Iowa poll, says to CNN’s Anderson Cooper: “The best news I can deliver is my best shot at what’s true, because then you know what you’re working with.” In essence, the data is what the data is.

There are some issues with weighting, even based on census data. The most recent U.S. census is over four years old at this point, and in that time frame the country has gone through and come out of the Covid-19 pandemic, has seen major fluctuations in the economy and job market, has seen major developments in military affairs overseas, among much more. The simple fact of the matter is that the census data may not be all that representative of actual demographics anymore. So when pollsters account for the traditionally large republican support in Iowa, backed in large part by older white males, they may be doing so unnecessarily.

Generative AI

A June article published by the Ash Center for Democratic Governance and Innovation at Harvard talks about the uses of generative AI for election polling. With conventional polling, there are a plethora of biases to be aware of that not only inform the results of the polls, but also inform the models that pollsters use to determine their final numbers. Insincerity, nonresponse, party bias, and even things such as misrecalling who you voted for in the previous election taint the results of a survey, and as such pollsters have to be aware and account for these in order to make their predictions.

It is entirely possible to simulate the results of a real survey with generative AI - in fact, the Ash Center article does just that. However, they note as well that this allows pollsters to query any number of times to obtain results from the LLM that align with their preconceived notions of what the survey results should be. In essence, it lends itself to a feedback loop. There is no doubt that real surveys are taking place, however how pollsters are deriving their final results is hidden, so several people on the internet have taken to explaining why, as Silver says, all of these polls are telling the same story.

The idea is as follows: pollsters are using small-scale polls to generate real poll data, which they are then cross-referencing with and using to get AI-generated poll data, which then they use in their models to make predictions based off the original poll data. The issue therein lies with the cyclic nature of how their models are being informed and informing.

To me, this sounds plausible at the least. However, to say that every pollster is similarly using generative AI to inform their numbers just to all derive the same results sounds unlikely. In all likelihood, it is probably a combination of many things. Pollsters were overzealous about Hillary Clinton in 2016, predicting a landslide victory that did not happen, the polls underestimated Barack Obama in 2012, and overstated Joe Biden's victory in 2020. It could be that, learning from their mistakes, they've opted to take the conservative route this time around. This, coupled with a greater number of polls being released, as well as perhaps the use of generative AI in some cases, could easily give us the results we're seeing now. It all comes down to tomorrow. If you haven't already, I urge you to get out and vote.

[1] Project FiveThirtyEight's polling data. Data is available on a state-to-state basis as well as viewing national averages.

Links to all sources and code used:

[2] New York Times opinion essay quoted.

[3] Iowa poll from 11/2/2024 which places Kamala Harris at +3 over Donald Trump in the state.

[4] American Association for Public Opinion Research writeup on Poll Herding.

[5] Anderson Cooper interviewing Ann Selzer on CNN.

[6] Ash Center for Democratic Governance and Innovation at Harvard Kennedy School article about the use of AI in political polling.

[7] Pew Research article discussing the inaccuracy of 2016 and 2020 polling.

Check out the GitHub for this project to get the code and cleaned up dataset