20 Questions A Journalist Should Ask About Poll Results - continued
5. What area (nation, state, or region) or what group (teachers, lawyers, Democratic voters, etc.) were these people chosen from?
It is absolutely critical to know from which group the interviewees were chosen.
You must know if a sample was drawn from among all adults in the United States, or just from those in one state or in one city, or from another group. For example, a survey of business people can reflect the opinions of business people—but not of all adults. Only if the interviewees were chosen from among all American adults can the poll reflect the opinions of all American adults.
In the case of telephone samples, the population represented is that of people living in households with telephones. For most purposes, telephone households are similar to the general population. But if you were reporting a poll on what it was like to be homeless, a telephone sample would not be appropriate. The increasingly widespread use of cell phones, particularly as the only phone in some households, may have an impact in the future on the ability of a telephone poll to accurately reflect a specific population. Remember, the use of a scientific sampling technique does not mean that the correct population was interviewed.
Political polls are especially sensitive to this issue.
In pre-primary and pre-election polls, which people are chosen as the base for poll results is critical. A poll of all adults, for example, is not very useful for a primary race where only 25 percent of the registered voters actually turn out. So look for polls based on registered voters, "likely voters," previous primary voters and such. These distinctions are important and should be included in the story, for one of the most difficult challenges in polling is trying to figure out who actually is going to vote.
The ease of conducting surveys in the United States is not duplicated around the world. It may not be possible in practice in some countries to conduct surveys of a random sample throughout the country. Surveys based on a smaller group than the entire population—such as a few larger cities—can still be reliable if reported correctly and may be the only available data.