20 Questions A Journalist Should Ask About Poll Results - continued
6. Are the results based on the answers of all the people interviewed?
One of the easiest ways to misrepresent the results of a poll is to report the answers of only a subgroup. For example, there is usually a substantial difference between the opinions of Democrats and Republicans on campaign-related matters. Reporting the opinions of only Democrats in a poll purported to be of all adults would substantially misrepresent the results.
Poll results based on Democrats must be identified as such and should be reported as representing only Democratic opinions.
Of course, reporting on just one subgroup can be exactly the right course. In polling on a primary contest, it is the opinions of those who can vote in the primary that count—not those who cannot vote in that contest. Primary polls should include only eligible primary voters.
7. Who should have been interviewed and was not? Or do response rates matter?
No survey ever reaches everyone who should have been interviewed. You ought to know what steps were undertaken to minimize non-response, such as the number of attempts to reach the appropriate respondent and over how many days.
There are many reasons why people who should have been interviewed were not. They may have refused attempts to interview them. Or interviews may not have been attempted if people were not home when the interviewer called. Or there may have been a language problem or a hearing problem.
In recent years, the percentage of people who respond to polls has diminished. There has been an increase in those who refuse to participate. Some of this is due to the increase in telemarketing and part is due to Caller ID and other technology that allows screening of incoming calls. While this is a subject that concerns pollsters, so far careful study has found that these reduced response rates have not had a major impact on the accuracy of most public polls.
Where possible, you should obtain the overall response rate from the pollster, calculated on a recognized basis such as the standards of the American Association for Public Opinion Research. One poll is not "better" than another simply because of the one statistic called response rate.