Almost every survey headline comes with a small line underneath. Margin of error, plus or minus 3 percentage points. Readers take it as a measure of how accurate the survey is. It isn’t, or not quite. It measures one specific kind of error, and often not the largest one.
The margin of error describes sampling error. If you draw a random sample of people from a population, you won’t get exactly the same answer as if you’d asked everyone. Draw another random sample and you’d get a slightly different result again. The margin of error says how far a result is likely to wander from the true population value because of that luck of the draw alone, usually at a 95% confidence level. Roughly, if you ran the same survey many times, about 95 out of 100 results would fall within that range of the true figure.
Where the 3 points come from
For a simple random sample, the margin depends mainly on sample size, not on the size of the population. That surprises people. A well-drawn sample of about 1,000 gives a margin of roughly plus or minus 3 points whether the population is a city or a whole country. About 400 respondents gives roughly plus or minus 5. To halve the margin, you need about four times the sample, which is why most national polls stop around 1,000 to 1,500 people.
The margin is largest when the answer is near 50%, and smaller near the extremes. A result of 10% from 1,000 people carries a margin closer to 2 points than 3. Published figures usually quote the worst case.
Two traps follow. First, the margin applies to each number, not to the gap between them. If a survey shows brand A at 46% and brand B at 42%, with plus or minus 3 points each, the lead of 4 points isn’t clearly outside the noise. The margin on the difference between two figures from the same sample is larger, close to double in a tight two-way split. Second, subgroups have their own, bigger margins. The 18 to 24 year olds in a sample of 1,000 might be 120 people, and their figure could carry a margin of 9 points or more.
What the margin doesn’t cover
The formula assumes a true random sample in which everyone in the population had a known chance of being picked, and everyone picked answered. Real surveys don’t work that way, and the gaps cause errors the margin knows nothing about.
Coverage error. If the sampling frame misses part of the population, people without landlines, people who never join online panels, people who don’t speak the survey’s language, the result describes the people who could be reached, not everyone.
Nonresponse error. Response rates for phone polls have fallen into single digits in many markets. If the people who answer differ from those who don’t in ways that matter to the question, weighting can only partly fix it.
Measurement error. Question wording, order and the answer options shape the result. Ask whether people “support” a policy and you’ll get a different figure than if you ask whether they “oppose” it, or offer a “don’t know” option.
Opt-in panels. Many market research and online surveys use volunteer panels, not random samples. For those, a classical margin of error doesn’t strictly apply. Good researchers publish a “credibility interval” or a modelled error estimate instead and explain how they weighted the sample.
How to read a survey number
Check the sample size and how people were recruited before the margin. Look for the field dates, the exact question wording and the weighting variables. Treat small subgroup figures with suspicion. And when two numbers sit within a few points of each other, read the result as close, not as a winner and a loser. The plus or minus line is a floor on the uncertainty, not a ceiling.
