The National Council on Public Polls (www.ncpp.org), an association of polling organizations, has prepared these answers to frequently asked questions about survey research. Answers To Questions We

Often Hear From the Public NATIONAL COUNCIL ON PUBLIC POLLS

1. Why am I or my friends never included in political polls conducted for the media? The reason is fairly simple. There are about 200 million adult or voting-age Americans. But the average poll has a sample size of 1,000 adults. This means that only one person in 200,000 will be included in any one national or state poll. To put it another way, it would take 200,000 polls with samples of 1,000 for pollsters to get around to all Americans -- and this assumes no one is called twice. Of course, national and local media organizations conduct several polls in one year. The number of national or local media political polls you see in a single year is about 250, but can vary depending on where you live. But even 250 polls in a single year means your chance of being interviewed at least once is still small. 2. Is a larger sample always better than a smaller sample? Larger sample are generally more precise, but sometimes not. The important rule in sampling is not how many poll respondents are selected but, instead, how they are selected. A reliable sample selects poll respondents randomly or in a manner that insures that everyone in the area being surveyed has a known chance of being selected. One step taken to insure that all members of a population have a known chance of being selected includes dialing randomly generated phone numbers to insure that people with unlisted phone numbers are included along with people with listed numbers. Another is to dial the number again and again when no one answers to insure that people often away from home are included in the sample. Another measure is to randomly select a member of the household for interview instead of interviewing the person who happens to answer the phone. Non-random samples occur when there is no chance of including all members of a population. If enough people have no chance of being included, survey results will be invalid. Here are some examples: Conducting a poll only between 9 AM and 5 PM, which excludes the opinions of most working people. Another is a poll based on a sample that selects itself, such as a radio station call-in poll, which rules out people not caring to respond or not even listening to the station. Other examples of self-selected samples are poll questions that appear in newspapers, on web pages, and in magazines, or any poll likely to have a high percentage of members of the general public not able or not motivated to make the effort to respond. In these cases, people who choose to respond are likely to differ from others who do not respond. 3. How can a sample of only 800 or 1,200 truly reflect the opinions of 200 million Americans within a few percentage points? Sampling methods and measures of sample reliability or precision are derived from a mathematical science called statistics. Statistics is a subject taught in colleges and some high schools. Text books on the subject are available in most libraries. At the root of statistical reliability is probability; i.e., the odds of obtaining a particular outcome by chance alone. As an example, the chances of having a coin come up heads in a single toss is 50%. Heads is one of only two possible outcomes. The chance of getting two heads in two coin tosses is less because two heads are now only one of four possible outcomes; i.e., a head/head, head/tail, tail/head and tail/tail. As the number of coin tosses increases, it becomes increasingly more likely to get outcomes that are either very close to half heads or exactly half because, as with two coin tosses, there are more ways to get such outcomes. Sample survey reliability works the same way -- but on a much larger scale. As in coin tosses, the most likely sample outcome is the true percentage of whatever it is we are measuring across the total population. Next most likely are outcomes very close to this true percentage. A statement of potential margin of error or sample precision reflects this and often appears in poll stories. Using a sample of 1,000 as an example, the statement could read: the chances are 95% of coming within +/- 3% of a hypothetical survey conducted among all members of the population. This means that 95% of all samples that could possibly be drawn will yield an outcome within 3 percentage points of the true percentage among the population. Keep in mind that estimates of potential sample error always assume random samples. But even in true random samples, precision can be compromised by other factors, such as the wording of questions or the order in which questions were asked. There is no single ideal sample size. Samples of any size have some degree of precision. The question is always whether there is sufficient precision to draw conclusions as determined by statistical formulae. 4. Why do polls often disagree with how I and my friends feel about things? It is unlikely that you have a circle of friends as diverse as a randomly selected sample, described before, a sample of the entire nation or of the state where you live. This would mean you have friends from all neighborhoods, of all ages, very wealthy friends and friends with no wealth, who are from all walks of life and with educational levels ranging from grade school to post-graduate. No poll has ever shown all people feeling exactly the same way on one issue. So the next time you see a poll showing only 30% in agreement with your point of view, remember, although you may not be in the majority, it still means that 60,000,000 Americans feel the way you do. 5. Why is job approval for elected officials asked so often in polls? Job approval is a question asked in polls for over 50 years, asked about presidents from Harry Truman to Bill Clinton. It is a measure of potential electoral success for an incumbent president (or state and local elected officials) and is useful for trend purposes. Incumbents who fail to win approval from a majority of the public for the job they are doing are very often in trouble on election day. Experience shows presidential job approval has a lot do with how well people think things are going in the country today. High job approval does not mean the officeholder is necessarily held in high, personal esteem. Job approval is just one question which, when analyzed in combination with responses to other questions, gives a good overall picture of how an incumbent is perceived. 6. How accurate are polls? The NCPP analyzed final presidential election polls conducted by the national media dating back over 50 years. When compared with actual election outcomes, average poll error for presidential elections between 1956 and 1996 has been declining. Average poll error on each candidate during this period was 1.9 percentage points. Important to this analysis of accuracy is that most of these polls were conducted within days or even hours before election day. Polls conducted 1-2 weeks before election day or even longer by local newspapers and TV stations cannot usually be expected to closely match election outcomes. Earlier polls are intended to monitor the success of campaigns and to identify the issues or events that will influence voter preferences on election day. 7. Can wording of questions bias poll results? How questions in a poll are worded is as important as sampling procedure in obtaining valid results. Most professional polling organizations and their media clients review the wording of questions as carefully as editors would examine a manuscript before publication. This process usually calls for a review of several drafts prior to fielding a poll. Questions are checked for balance: Are they worded in a neutral fashion, without taking sides on an issue? Does the question represent both sides of an issue fairly? Answer choices read to poll respondents must also be balanced; e.g., approve or disapprove, favor or oppose. The order of questions must be logical. That is, general questions are asked before specific questions. For example, overall job approval of an incumbent must be asked before specific questions are asked that may remind respondents about the incumbents successes or failures. The same goes for questions asking respondents what side they take on an issue which may influence a later question about opinion of a candidate who takes the opposite side. Questions are written using clear, unambiguous, concise language to insure that all respondents, regardless of educational level, understand them. And since most polls are conducted by phone, a writing style suitable for the ear is often adopted as opposed to a style more suitable for reading. 8. Why should I respond to a poll? Each year, tens of thousands of Americans respond to poll interviews on subjects of national interest. They do so because they want their opinions heard. A poll or survey is an unbiased way for people to make their views known to each other, to their government, to businesses, to educators, and many other institutions. This is one way for average Americans to add their voices to the debate over important issues of our day. It is our sense that people want to be included in polls. Polling is only a part of a $3 billion U.S. survey research industry that affects your life in many ways. Medicines in your cabinet became available only after they had been tested for effectiveness and safety, tested among samples of patients. Manufacturers use sampling methods to guard against producing defective products. Businesses conduct market research by sampling consumer opinions to help develop products people want and avoid costly mistakes. Even the Bureau of the Census only interviews everyone in the nation once every 10 years. The Census Bureau along with other government agencies also conduct sample surveys to obtain social and economic data we see in the news. One thing legitimate survey researchers will never do is try to sell you something. That is called telemarketing. According to direct marketing industry estimates, there are now 2 million telemarketers who call the general public to sell products and services. The number of survey research interviewers who conduct telephone interviews is probably only a small fraction of 2 million.