Polling: A Greater Margin for Error

The concerns about reliability extend beyond New Hampshire. In California, most pollsters correctly predicted a Clinton victory in the Feb. 5 primary but produced widely varying estimates of the margin. Polls in South Carolina in the days before its Jan. 26 contest showed large numbers of undecided voters, yet Obama wound up winning in a romp, with 55 percent of the vote. This result raised questions about whether there was a reverse Bradley effect in play, in which voters surveyed were for some reason reluctant to express support for a black candidate. Meanwhile, another type of headache surfaced in states such as Wisconsin and Minnesota that allow election-day registration, which complicated pollsters’ projections of turnout.

The collective concern is partly driven by the fact that more firms are polling than a decade ago. Besides media-sponsored efforts such as the widely praised Washington Post-ABC poll and the long-running Gallup and Roper surveys, major players include public opinion research firms such as Rasmussen Reports and American Research Group and schools such as Quinnipiac University in Hamden, Conn., and Marist College in Poughkeepsie, N.Y. Consumer surveying firms also dip into polling during election years in an effort to boost their own name recognition.

All of the activity creates the possibility that polls will disagree with one another. That in itself shouldn’t come as a shocking development: Variability is a fact of life in a profession that revolves around people randomly dialing telephone numbers until they find someone willing to open up for 10 minutes or so and answer a set of questions.

But experts say that reliability has become crucial at a time when many Americans are acutely stressed about job security, stagnant wages, falling home prices and rising energy costs, and are skeptical about the government’s ability to address the concerns. In such a sour, anxious environment, pollsters need to distinguish among cautious hope, pessimism and indifference — and measure which candidate elicits which emotions. Careful phrasing of poll questions can also offer important indications about how many voters could change their minds, and which candidate’s support is softest.

The challenge for pollsters is how to capture these kind of subtleties — which might ultimately explain how an election turns — and still satisfy the competitive pressure to generate a headline-grabbing “horse race” snapshot of who’s leading and by how much. Experts say the industry needs to be up to the task, particularly in battleground states with shifting proclivities and hard-to-read voters.

“There are a lot of indications from a variety of sources this will be a close election, and the margins will be very close in many places. That makes pollsters uneasy,” said Michael Traugott, a University of Michigan professor of communications and an authority on surveys and polls who is chairing the AAPOR panel.

Like Counting Beans

A big part of polling is about mathematically smoothing out the random nature of public opinion. At its heart, the business depends on finding people who know what their preferences are and are willing to state them. But to obtain a statistically valid cross-section of the public, pollsters compare their survey populations with other studies that achieve high response rates, such as the census, by gender, age, race and where people live.

Because most surveys wind up skewed — often including too many women and too few minorities and not enough people with low education levels — pollsters mathematically weigh their results to compensate for raw data that does not mirror the population. Final poll results often have slightly extrapolated figures for African-Americans and Hispanics, men, younger people and those who never finished high school.

Missing the Mark: Click Here to View Chart

This mathematical rigor is supposed to impose a measure of quality control. But experts question whether those efforts have been undermined by a lack of candor and public focus in a year that’s experienced a grinding, longer than expected Democratic primary between a black man and a white woman, the emergence of two major-party candidates who are each working hard to win crossover voters, and mobilization efforts among customarily unengaged groups.

“The general election still is four months away; no vice presidential candidates have been selected, and the issues haven’t really been articulated for a large percentage of voters,” said David W. Moore, a former managing editor of the Gallup Poll and author of the forthcoming book “The Opinion Makers: An Insider Exposes the Truth Behind the Polls.”

“If you consider 30 to 40 percent of the public only occasionally follows the news, they will be heavily influenced by the way the poll questions are phrased,” Moore said. “There are a lot of little things that might not affect their preferences, but will influence what they say.”

Polling: A Greater Margin for Error

The fascination with what people will say has existed a lot longer than the computer data sets, telephones or other trappings of modern sampling. In July 1824, the Harrisburg (Pa.) Pennsylvanian surveyed groups of citizens in Wilmington, Del., about that year’s four-way presidential race, concluding that Tennessee senator and military hero Andrew Jackson had a commanding 70 percent support, and his nearest rival, Secretary of State John Quincy Adams, 23 percent. The poll proved a somewhat reliable indicator: Jackson won the popular vote, but after none of the candidates received a majority of electoral votes, the election was decided by the House of Representatives in Adams’ favor.

Such “straw polls” prevailed for the next century. In the early 20th century, the venerable Literary Digest managed to cull huge samples of prospective voters — and increase its subscription sales in the process — by mailing questionnaires to readers and to Americans listed in phone books or with automobile registrations. The magazine’s 1936 mailing to 10 million people yielded a staggering 2 million responses but also revealed the perils of bad survey design. The results predicted that Republican Gov. Alf M. Landon of Kansas would defeat President Franklin D. Roosevelt. But Roosevelt, riding the support of millions of Depression-era voters who were too poor to have cars, phones or magazine subscriptions, won one of the most lopsided elections in history, capturing 61 percent of the popular vote and 46 of 48 states. Disgraced, the magazine’s owners folded it soon afterward.

Around the same time, George Gallup and other public opinion pioneers such as Elmo Roper and Archibald Crossley began studying much smaller samples of voters using mathematical models to reflect a better cross-section of the population. The homespun Gallup likened the exercise to scooping the correct proportion of black and white beans out of a barrel, but he used sophisticated statistical models to rigorously break down the voting population by region, profession and party affiliation. While the methods resulted in some noteworthy failures — Gallup and his colleagues stopped interviewing a few weeks before the 1948 election and predicted that New York Republican Thomas E. Dewey would unseat President Harry S Truman, thus missing a crucial third-party swing in favor of the incumbent — they provided much of the foundation of the modern profession.

The Race Question

Even in the early days, the more prescient pollsters tried to gauge attitudes about race in politics. In 1958, as the civil rights movement was gaining momentum, the Gallup Organization began asking respondents if they would be willing to vote for a qualified black presidential candidate; 53 percent polled in the first survey said they would not. Though the number has plummeted to only a few percent in recent years, there remains considerable skepticism about whether the results can be taken at face value. Experts acknowledge that racial attitudes are improved today, but they still question whether some respondents are providing “socially desirable” answers, believing it’s taboo to express anything but support for a black candidate.

“There’s a norm or etiquette of politeness that prevents some people from saying what they really feel, and there’s a more subtle stereotyping that goes on at the same time in ways people don’t fully appreciate,” said Bruce E. Cain, a political scientist at the University of California Berkeley and an authority on racial and ethnic politics. “Many people believe black candidates are radical progressives because their experience with many black mayors and U.S. representatives has shown them to be very liberal. And previous presidential candidates like [the Rev.] Jesse Jackson were very liberal.”

Shifting Racial Attitudes: Click Here to View Chart

Cain said Obama has gone out of his way to act against the stereotype. Though that’s helped him develop significant crossover appeal, it has, at times, upset progressive members of his base. That, though, won’t stop Obama’s opponents from trying to link him to more radical politicians in the hope of tapping latent biases.

Cain and others note that Republicans and Clinton’s campaign already tried to tap into these sentiments by linking Obama to controversial statements made by his former pastor, the Rev. Jeremiah A. Wright Jr., who has preached a message of black empowerment.

Though it is impossible to quantify how many voters participate in the tolerance gap toward black candidates — supporting them with pollsters but opposing them in the voting booth — it’s been implicated in a number of elections pitting black and white contenders in the past few decades. The year after Bradley narrowly lost his gubernatorial bid to Republican George Deukmejian in California, Harold Washington, an African-American Democrat, narrowly edged Republican Bernard Epton in the Chicago mayoral race despite pre-election polls showing him with a 14 point lead. In Virginia, Douglas Wilder became the nation’s first elected black governor in 1989, defeating Republican Marshall Coleman by less than 1 point after leading in some pre-election polls by double digits. The same year, David N. Dinkins became the first black mayor of New York, narrowly defeating Rudolph Giuliani despite leading by 18 points in one poll a week before the contest. And in a bitterly contested North Carolina Senate race in 1990, African-American Democrat Harvey Gantt lost to incumbent Republican Jesse Helms despite leading in two of three independent polls just before the election.

Though the possibility of hidden votes for white candidates lingers, pollsters such as Scott Keeter, director of survey research for the Pew Research Center, believe racial prejudice is factoring less in public judgments than 15 or 20 years ago. He noted that independent polls conducted in five gubernatorial and Senate races in 2006 that included black candidates — in Ohio, Pennsylvania, Massachusetts, Maryland and Tennessee — more or less tracked with the final results. Keeter said this winter’s New Hampshire polling miscue appears to have been an isolated instance, rather than some indicator of pervasive bias.

“New Hampshire certainly was scary to pollsters, but I just haven’t seen a consistent enough pattern of understating support for Hillary Clinton or magnitude of error to make me think race is the reason,” Keeter said.

Which is not to say that race isn’t weighing heavily on the minds of even the more optimistic pollsters. A pressing concern of those predicting voter turnout in battleground states is finding new ways to reach a statistically sufficient number of African-American voters, whose support for Obama hovers around 95 percent, according to most surveys. Heavy black voter turnout could tip the balance Obama’s way in such swing states as Ohio, Michigan, Nevada and Virginia. Last month, the Obama campaign deployed 3,000 volunteers in a nationwide voter registration drive to expand turnout, particularly among African-Americans and young people.

Polling: A Greater Margin for Error

Black voters, particularly those who are poor and less educated, have been historically underrepresented in surveys and remain less likely to answer the phone or agree to participate in a poll, experts say. That has made it difficult for pollsters to assess whether they are likely to vote, or how heavily to weight the results of the African-Americans they do reach. Selzer, the Iowa pollster, said the problems were seen on a smaller scale during the 2005 Detroit mayoral race. Democratic incumbent Kwame M. Kilpatrick, plagued by scandals and accusations of mismanagement, was nonetheless able to close a 20 percentage point gap and defeat challenger Freman Hendrix, also a Democrat, through a last-minute campaign blitz aimed at young voters, who typically didn’t turn out.

“It was a case where pollsters underestimated black voter turnout even in a place with a predominantly African-American population,” Selzer said. To avoid repeats, she predicted, pollsters will take steps such as conducting separate surveys of cell phone users in an attempt to reach more inner-city residents who may not have land-line phones.

About 14 percent of the adult population now relies solely on cell phones — a disproportionately young cohort (typically 18-29 years old) that contains a greater proportion of African-Americans and Hispanics, males and the religiously unaffiliated. Because Obama derives so much support from these groups, pollsters are having to decide whether to conduct random cell phone-only surveys — which can cost three times more because of the extra screening involved and federal regulations that require the numbers to be manually dialed by interviewers, not auto-dialed.

Late last year, the Pew Research Center conducted two surveys of people who use land lines, cell phones only, or both, and found that all three groups responded similarly on key political measures such as presidential approval, the war in Iraq, party affiliation and which candidate they were supporting in primaries. However, the demographic mix of the cell-only population made respondents more sensitive to such issues as immigration and less concerned with Social Security or pension reform. They also were more likely to get their news from the Internet, late-night comedy shows, or social networking sites than newspapers or network evening news programs.

Pew polls now include periodic cell phone samples, along with land-line polling, to bolster the representation of groups that are disproportionately cell-only, such as young blacks and Hispanics. “If cell phone use continues to grow and skew young, we and others will be forced to do more of that,” Keeter said.

Answering the Call

The ability to connect with a younger, more diverse electorate could be a welcome development in a field that’s been handicapped by declining response rates in the past decade. The prevalence of caller ID, answering machines and call blocking services, combined with the fact that Americans are spending more time at work and in their cars, has brought participation levels to less than 30 percent of sampled households, compared with nearly 40 percent in the late 1990s, according to figures compiled by Pew. While that hasn’t undercut the credibility of most surveys, it’s put considerably more pressure on pollsters to find statistically valid population samples, because individuals who are less interested or engaged in politics are less likely to consent to a phone interview, especially at home in the evening.

“It gets to the point where it’s like telemarketing, particularly in small states, because there are a lot of media organizations and private organizations out there calling some of the same people,” said Adam Berinsky, a political scientist at the Massachusetts Institute of Technology and author of “Silent Voices: Public Opinion and Political Participation in America.”

Phone Habits Influence the Sampling Mix: Click Here to View Chart

“The phone has made it much cheaper than flying people to a location and making them go door-to-door, but you still need good interviewers and a good sample,” he said.

These difficulties have contributed to the growing popularity of automated phone polls that solicit political preferences and personal information by asking respondents to enter numbers on their keypad. Many experts in the polling business look down on these computer-driven surveys, because of their inability to verify the identities of the people being surveyed. But because they cost a fraction of live-interview surveys, survey firms are gravitating to the technique, particularly for polls that track opinion on the same set of questions over time.

One concern about non-participation in live interview polls is that it can distort the way polls portray the undecided. Many public opinion experts believe that individuals who hang up or are generally unresponsive to questions may have concrete concerns and political views, but their opinions are not sufficiently developed to succinctly answer a survey question. Others might not have the time or patience to think through a political question and simply say they don’t know. That is why polls such as the University of New Hampshire survey press to force a choice. The dilemma for the polling profession is whether to continue trying to tease out aspects of individuals’ personal situations, by pushing them to state a preference, or allow the poll to reflect their uncertainty or ambivalence.

This has a particular bearing in questions about party identification and political attitudes. As many as half the respondents to some surveys identify themselves as independents when asked their political affiliation. Pollsters tend to find this an inadequate characterization and push the individuals by asking whether they “lean” Democratic or Republican. The tactic has proven a useful barometer of public sentiment in a year such as this, when growing numbers of people are identifying with the Democratic Party, partly out of dissatisfaction with President Bush. However, experts caution that the identification question may be unreliable for predicting election results because other, more rigorous polls indicate that the Democrats’ public standing has not improved significantly; it’s more a function of the Republican Party losing support among independents.

Polling: A Greater Margin for Error

Gallup, working with USA Today, took another measure of voter independence in June, concluding that 23 percent of 1,310 likely voters it surveyed are “swing voters” who could change their minds before Election Day. That suggests that Obama and McCain would be best served by focusing their pitches on uncommitted voters instead of concentrating on mobilizing their respective bases. But Moore, the former Gallup managing editor, suspects that the public mood isn’t quite as volatile as the survey suggests, and that many of these individuals haven’t even decided whom to support. He suggests that pollsters could obtain a more accurate characterization of undecideds by rewording their questions and asking people up-front if they know whom they will support in November.

“Most independents don’t vote all the time, but when they do, they tend to really be undecided going into the final days of an election,” Moore said. “Many are susceptible to some kind of influence from media coverage, and past surveys showed one in six didn’t vote the way they said they would a few days earlier. So a lot of the answers are still out.”

Predicting the Future

If pollsters aren’t yet asking all the right questions, many at least seem determined to use lessons from this year to fine-tune their craft. One focus is on how to use the Internet to reliably survey large populations. Though consumer product manufacturers, sports teams and other businesses already use online polls to sample preferences, experts say these cybersurveys haven’t yet proven an accurate indicator of voter behavior. That’s because the online public tends to be a younger and more affluent subset of the general population. Moreover, there’s no way to independently verify the identities of survey-takers, or establish whether they’re even of voting age.

Experts say a British online service called YouGov could provide a model for pollsters in the United States. The company polls individuals who visit its Web site and sign up for surveys, then filters the responses based on demographic information provided by respondents. Though this methodology has been criticized for excluding a large segment of the population without Internet access, it accurately predicted a number of British political races, including the tight London mayoral outcome this spring.

Another approach is for polling firms to recruit random households to join Internet survey panels, and equip respondents with laptops and high-speed data connections if they agree to respond to regular surveys. Some experts believe this option, however expensive, will become the norm if ordinary telephone service moves to the Internet backbone.

“There are only two ways to study an election reliably and accurately: work hard with phone polls or use the Internet with a representative sampling,” said Jon A. Krosnick, a Stanford University professor of humanities and social sciences and an authority on survey methodology. “Given the economic circumstances, with big media companies squeezing their polling units, I’m expecting business as usual. It’s potentially a missed opportunity, because this is a remarkable election season and someone should be collecting quality data.”

Krosnick is a principal investigator in a National Science Foundation-sponsored effort called the American National Election Studies, which since 1948 has interviewed sets of voters before and after elections with the goal of developing new kinds of data collection and methods for choosing questionnaire items. Not surprisingly, an important area of study this year is using psychological research to better measure racial attitudes and elicit honest responses.

Krosnick envisions next-generation polling techniques such as showing survey participants Chinese characters and asking them to identify those that are most visually appealing. Pollsters would then show the same ideographs with photos of candidates Obama and McCain briefly interspersed between the selections that were identified as most appealing to establish if the subliminal response would change the second set of choices.

Another method of measuring attitudes without overt questions involves asking respondents how many of four named things they dislike, using objects that elicit particularly strong responses: anchovies, watching a sunset, making abortion legal or working as a garbage collector. Some randomly selected participants would have a fifth item to consider: a black man as president. Comparing the numbers the two groups identify could yield important clues about racial tolerance.

“We’re good at measuring a lot of things, but the trickiest is measuring racial attitudes,” Krosnick said. “A lot of pollsters haven’t specialized in it or taken it seriously, and now have to believe or live with what respondents way. You need to identify all the factors that might push a person’s vote and devise questions to tap them.”

The University of Michigan’s Traugott predicts that another byproduct of this year’s race might be better measures of young people’s attitudes, specifically what moves them to get politically involved in communities where they have not put down roots. It’s part of a generational shift in public opinion research that’s veering away from statistics and taking a nuanced measurement of attitudes.

“Generally, there’s a relationship between age and voting. People who are less likely to be settled in a place, who are harder to find in telephone surveys, have not been a big focus for pollsters. This year, with Obama having traction with young people, we’ll have to phrase questions new ways to learn whether they’ll behave differently.”

Polling: A Greater Margin for Error

FOR FURTHER READING: Independent voters, CQ Weekly, p. 1604; Ohio as a bellwether, p. 1532; “robocalls,” p. 1064; 2008 campaign outlook, p. 1086; candidates’ economic promises, p. 422.