Margin of error

that show the relative likelihood that the "true" percentage is in a particular area given a reported percentage of 50 percent. The bottom portion of this graphic shows the margin of error, the corresponding zone of 99 percent confidence. In other words, one is 99 percent sure that the "true" percentage is in this region given a poll with the sample size shown to the right. The larger the sample is, the smaller the margin of error is. If lower standards of confidence (95 or 90 percent) are used, the margins of error will be smaller (by 24 or 36 percent, respectively) for the same sample sizes.
The margin of error is an estimation of the extent to which a poll's reported percentages would vary if the same poll were taken multiple times. The larger the margin of error, the less confidence one has that the poll's reported percentages are close to the "true" percentages, i.e. the percentages in the whole population.

The margin of error can be calculated directly from the sample size (the number of poll respondents) and is commonly reported at one of three different levels of confidence. The 99 percent level is the most conservative, the 95 percent level is the most widespread, and the 90 percent level is rarely used. Formally, if the level of confidence is 99 percent, one is 99 percent certain that the "true" percentage in a population is within a margin of error of a poll's reported percentage for a reported percentage of 50 percent. Equivalently, the margin of error is the radius of the 99 percent confidence interval for a reported percentage of 50 percent.

Note that the margin of error only takes into account sampling error. It does not take into account other potential sources of error such as bias in the questions, bias due to excluding groups who could not be contacted, people refusing to respond or lying (selection bias), or miscounts and miscalculations.

Calculations and caveats

Margin of error at 99 percent confidence

Margin of error at 95 percent confidence

Margin of error at 90 percent confidence

The margin of error is a poll-level statistic that should not be used to evaluate or compare reported percentages. However, due to its unfortunate name (it neither establishes a "margin" nor is the whole of "error"), it has become one of the most widely overinterpreted statistics in general use by the media. It is frequently misused to judge whether one percentage is "significantly" higher than another or to specify the error associated with reported percentages outside of 50 percent.

Understanding the margin of error

A running example

The basic concept

Given the size of the sample (1,013), probability theory allows the calculation of the probability that the poll reports 47 percent for Kerry but is in fact 50 percent, or is in fact 42 percent, or is in fact zero percent. This theory and some Bayesian assumptions suggest that the "true" percentage will probably be very close to 47 percent. The more people that are sampled, the more confident pollsters can be that the "true" percentage is closer and closer to the observed percentage. The margin of error is a rough, poll-wide expression of that confidence.

Statistical terms and calculations

The standard error can be estimated simply given a proportion or percentage, p, and the number of polled respondents, N. In the case of the Newsweek poll, Kerry's percentage, p = 0.47 and N = 1,013. Given some statistical theory outlined below, the following holds:

Standard error =

Plus or minus 1 standard error is a 68 percent confidence interval, plus or minus 2 standard errors is approximately a 95 percent confidence interval, and a 99 percent confidence interval is 2.58 standard errors on either side of the estimate.

The margin of error is the radius (half) of the 99 percent confidence interval, or 2.58 standard errors, when p = 50 percent. As such, it can be calculated directly from the number of poll respondents.

Margin of error (99%) = 2.58 ×

The use and abuse of the margin of error

Incorrect interpretations of the margin of error

• Kerry and Bush are "statistically tied" or are in a "statistical dead heat".
• *It only "matters" if Kerry leads Bush (or vice versa) by more than 4 percent.
• Any change in the percentages for future polls does not "matter" unless it is more than 4 percent.
• Because Nader got 2 percent and the margin of error is 4 percent, he could potentially have 0 percent.
• The margin of error is the same for every percentage, i.e. 47% ± 4%, 45% ± 4%, 2% ± 4%.
  

  Arguments for the use of the margin of error

• For casual comparisons of different polls, it is helpful to define a benchmark (99 percent confidence interval radius for an estimated percentage of 50 percent) that reflects the sampling variance of the poll.
• *Sampling variance does not decrease linearly with increasing numbers of respondents (it decreases by the square root of N), so using the number of respondents as an inverse measure of standard error might be confusing.
  

  Arguments against the use of the margin of error

• The margin of error is a simple transformation of the number of respondents into an ambiguous term that is neither a "margin" nor the whole of "error".
• The margin of error is being confused with the confidence interval of reported percentages.
• *The 99 percent confidence interval radius is smaller than the margin of error for any percentage besides 50 percent.
• * It is much smaller and more asymmetric for very high and very low percentages.
• It is not a "margin" at all; the probability of the true percentage being outside the margin of error is low but nonzero.
• There is no agreed-upon confidence level. Most pollsters use 99 percent, but many use 95 percent or 90 percent; this makes their polls look more accurate.
• When the purpose of polls is to compare percentages, the use of the margin of error is tempting but inappropriate.
• Perhaps most importantly, there are many different sources of error in polling, and variance due to sample size is not likely to be the only contribution. Other possible contributions to error include:
• *Sampling bias, when the sample is not a representative sample from the population of interest.
• *The phrasing of the question may not be appropriate for the conclusions of the poll.
• *Response error (Sudman & Bradburn, 1982)
• **Deliberate distortion (fear of consequences, social desirability, response acquiescence).
• **Misconstrual (not understanding the question).
• **Lack of knowledge (guessing to try to be helpful).
  

  Margin of error and population size

This may seem counter-intuitive at first; after all, each person in the population has a unique personality and opinion, and in a very large population, only a very small fraction of such people would actually be polled, and it would thus seem that the poll is not capturing enough information. However, because a poll involves only a very specific question, there is only one relevant attribute in the population that needs to be considered, and this means that an individual's opinion is effectively equivalent to those of many other members of the population, some fraction of which will be polled. For instance, in the running example, the only relevant attribute of a population member is whether he or she is a Bush voter, a Kerry voter, or a Nader voter - all other characteristics of a population member are irrelevant. Thus for instance if there are 100,000,000 registered voters, and 48,000,000 of them were Kerry voters, then for the purposes of this statistical
analysis all of the 48,000,000 individuals in this group would be completely interchangeable and equivalent. An individual Kerry voter may fret that because he or she was not polled directly, his or her opinion was not reflected in the results of the poll; but this voter has 47,999,999 other voters with identical opinions (as far as the poll question is concerned), and it is exceedingly likely that a poll of 1,013 voters will contain a properly representative fraction of this group, provided of course that the voters being polled were selected randomly.

To give an analogy, suppose that one is trying to estimate the percentage of salt in an ocean. This can be easily accomplished by taking a glass of seawater and then chemically analyzing the proportion of salt in that sample. The amount of salt and water in this glass is far smaller than the amount of salt and water in the ocean under study. Nevertheless, the sample is likely to give a very accurate measurement of the ocean's salinity, provided of course that the salt is evenly distributed across the ocean (this hypothesis is the analogue of the hypothesis that the poll sample is being randomly chosen). In fact, one could already obtain a crude but reasonable estimate of salinity by testing just a single drop of seawater, though of course the larger sample in the glass would provide a more accurate measurement. This analogy may help explain why it is the sample size, rather than the population size, that determines the margin of error in a poll.

Comparing percentages: the probability of leading

Tables

The margin of error is frequently misused to determine whether one percentage is higher than another. The statistic that should be used is simply the probability that one percentage is higher than another. This can tentatively be called the Probability of Leading. Here is a table that gives the percentage probability of leading for two candidates, in the absence of any other candidates, assuming 95% confidence levels are used:

For example, the probability that Kerry is leading Bush given the data from the Newsweek poll (a 2% difference and a 4% margin of error) is about 68.8%, provided they used a 95% confidence level. Note that the 100% entries in the table are actually slightly less. Here is the same table for the 99% confidence level:

If the Newsweek poll used a 99% confidence level, the probability that Kerry is leading Bush would be only about 74.1%. It is evident that the confidence level has a significant impact on the probability of leading.

Derivation

The rest of this section shows how the Newsweek percentage might be calculated. This probability can be derived with 1) the standard error calculation introduced earlier, 2) the formula for the variance of the difference of two random variables, and 3) an assumption that if anyone does not choose Kerry they will choose Bush, and vice versa, i.e. they are perfectly negatively correlatedd. This assumption may not be tenable given that a voter could be undecided or vote for Nader, but the results will still be illustrative.

The standard error of the difference of percentages p for Kerry and q for Bush, assuming that they are perfectly negatively correlated, follows:

Standard error of difference =

These calculations suggest that the probability that Kerry is "truly" leading is 74 percent.

More advanced calculations behind the margin of error

is approximately normally distributed with expected value 0 and
variance 1. Consulting tabulated percentage points of the normal
distribution reveals that P(−2.576 < Z < 2.576) = 0.99, or, in other
words, there is a 99 percent chance of this event. So,
:

This is equivalent to
:

Replacing p in the first and third members of this inequality by
the estimated value X/N seldom results in large errors if
N is big enough. This operation yields
:

The first and third members of this inequality depend on the
observable X/N and not on the unobservable p, and are the
endpoints of the confidence interval. In other words, the margin of
error is
:

References