20 Questions A Journalist Should Ask About Poll Results - continued
13. What other kinds of factors can skew poll results?
The margin of sampling error is just one possible source of inaccuracy in a poll. It is not necessarily the source of the greatest possible error; we use it because it's the only one that can be quantified. And, other things being equal, it is useful for evaluating whether differences between poll results are meaningful in a statistical sense.
Question phrasing and question order are also likely sources of flaws. Inadequate interviewer training and supervision, data processing errors and other operational problems can also introduce errors. Professional polling operations are less subject to these problems than volunteer-conducted polls, which are usually less trustworthy. Be particularly careful of polls conducted by untrained and unsupervised college students. There have been several cases where the results were at least in part reported by the students without conducting any survey at all.
You should always ask if the poll results have been "weighted." This process is usually used to account for unequal probabilities of selection and to adjust slightly the demographics in the sample. You should be aware that a poll could be manipulated unduly by weighting the numbers to produce a desired result. While some weighting may be appropriate, other weighting is not. Weighting a scientific poll is only appropriate to reflect unequal probabilities or to adjust to independent values that are mostly constant.
14. What questions were asked?
You must find out the exact wording of the poll questions. Why? Because the very wording of questions can make major differences in the results.
Perhaps the best test of any poll question is your reaction to it. On the face of it, does the question seem fair and unbiased? Does it present a balanced set of choices? Would most people be able to answer the question?
On sensitive questions—such as abortion—the complete wording of the question should probably be included in your story. It may well be worthwhile to compare the results of several different polls from different organizations on sensitive questions. You should examine carefully both the results and the exact wording of the questions.