20 Questions A Journalist Should Ask About Poll Results - continued
3. How many people were interviewed for the survey?
Because polls give approximate answers, the more people interviewed in a scientific poll, the smaller the error due to the size of the sample, all other things being equal. A common trap to avoid is that "more is automatically better." While it is absolutely true that the more people interviewed in a scientific survey, the smaller the sampling error, other factors may be more important in judging the quality of a survey.
4. How were those people chosen?
The key reason that some polls reflect public opinion accurately and other polls are unscientific junk is how people were chosen to be interviewed. In scientific polls, the pollster uses a specific statistical method for picking respondents. In unscientific polls, the person picks himself to participate.
The method pollsters use to pick interviewees relies on the bedrock of mathematical reality: when the chance of selecting each person in the target population is known, then and only then do the results of the sample survey reflect the entire population. This is called a random sample or a probability sample. This is the reason that interviews with 1,000 American adults can accurately reflect the opinions of more than 210 million American adults.
Most scientific samples use special techniques to be economically feasible. For example, some sampling methods for telephone interviewing do not just pick randomly generated telephone numbers. Only telephone exchanges that are known to contain working residential numbers are selected, reducing the number of wasted calls. This still produces a random sample. But samples of only listed telephone numbers do not produce a random sample of all working telephone numbers.
But even a random sample cannot be purely random in practice as some people don't have phones, refuse to answer, or aren't home.
Surveys conducted in countries other than the United States may use different but still valid scientific sampling techniques, for example, because relatively few residents have telephones. In surveys in other countries, the same questions about sampling should be asked before reporting a survey.