Franklin & Marshall College Franklin & Marshall College

    • u-h-20ccfc70467a-jpg
    • u-h-9af179e6f4f3-jpg
    • u-h-c8018e0f0ae5-jpg

Gresham's Law of Junk

January 8, 2002

by Dr. G. Terry Madonna and Dr. Michael Young

The New Year is here. And with it, the campaign begins for the next Pennsylvania governor. More accurately, the public campaign begins. Heck, the four candidates for governor, Republicans Mike Fisher and Barbara Hafer and Democrats Ed Rendell and Bob Casey have been campaigning for the better part of a year in what aptly has been termed the “invisible primary”.

But soon, the below the radar screen phase of the race will morph into public campaign mode. The candidates and their surrogates will be ubiquitous. Press coverage will intensify. Television and radio commercials will fill the airwaves. And pervading all of it will be an endless series of opinion polls.

The polls in particular deserve some attention. Yes, we know, get out the Ouija boards. Many think political polls are worthless, and indeed some are little more than thinly veiled efforts at influencing the course of a campaign. Behind these polls one can often find the not so subtle work of the pollsters who work for the candidates themselves, trying to improve their fundraising or create a buzz with press and the political community.

The “selective leak” of only favorable findings and only releasing partial numbers are tools widely used by political strategists. But the favorite trick employed to inflate their candidate’s standing is the so-called “Mr. Smith Question”. This little legerdemain matches the candidates against each other, not once but twice, the second time providing the respondent some questions in between that either informs the respondent of the great accomplishments of their candidate or the foibles of the opponents. Bet your sweet life the second match up will be the one you learn about.

In recent years, a new entry has appeared in the polling market place--computerized response audience polls, or CRAP, as they are known in the trade. No we did not invent the acronym, but we do approve of it.

These polls are amazing because they use computer technology to call thousands of households in just several hours. An automated dialing system transmits a digitized voice that reads the questions. Respondents can answer by punching the keypad on the telephone. The promoters of these polls boast of speed and brevity. The interview often takes less than five minutes.

Social scientists have been sharply critical of CRAP polls for several justifiable reasons. One problem is the identity of the respondent. Your five year old can punch the keys on the telephone and voice her opinion, or your favorite uncle from Tulsa who happens to be around when the phone rings can have his say as well. Who really knows who’s on the other end of the phone? Real survey researchers call the practice the uncontrolled selection of respondents, and it can be a real source of error. It’s tantamount to standing on the street corner and haphazardly asking questions of anyone who happens by.

Then, there is the problem of response rate. CRAP poll machines call thousands of people to complete six or seven hundred interviews, producing a very low response rate. Phone hang-ups especially plague the computer polls. The more people who don’t answer the poll, the more likely the results will be misleading--reflecting the views of those who were called and answered the poll, but not those who were called but failed to answer. Even high quality polls have response rate problems, but they are magnified many times over by the computer polls.

In the 2000 presidential election, the CRAP entry was the Rasmussen poll, conducted by Scott Rasmussen. His Portrait Of America poll was off by 4.5% compared to an average of 1.1% for the final polls taken by the ten leading national surveys just prior to Election Day. The traditional polls used random samples and real interviewers. Rasmussen himself is not a trained survey researcher. He was one of the cofounders of ESPN, the sports cable network. He admits that his purpose is to create a bunch of number addicts that will “feed the sports junkie’s appetite”, and not to conduct scientific polls for serious survey research.

One computerized response audience poll now working in Pennsylvania is SurveyUSA. It boasts that it was the first research company to eliminate the “single largest cost “ leading to questionnaire bias: the human interviewer. The California based computer company boasts of scores of local television, radio and newspaper clients. KDKA, the Pittsburgh television station, is a client. KDKA is also Pennsylvania’s entry into the world of CRAP-based political polling.

SurveyUSA claims its company specializes in “fast-turn opinion”, which can be “interpreted instantly by Journalists” Translation, they produce simple polls done cheaply, using methods social scientists find less than acceptable for important survey research.

The bottom line here is that these polls are not scientific and suffer from numerous methodological shortcomings. Maybe these computers polls are useful for some types of marketing and useful to convince advertisers of their value, but not a single national network, university research operation, or major national pollster has adopted this practice. Nor are they likely to do so any time soon.

The KDKA poll already has caused a dust up with the Hafer campaign, which challenged the results of a recent match up between Fisher and Hafer. The computer poll showed Fisher with a sizable lead, a finding not consistent with the results of other Fisher-Hafer non-computer polls conducted during the fall.

Here’s the good news. The media in Pennsylvania are suspicious of computer polls and do not widely report them. The working press corps in the state has over the past decade developed a much higher level of survey research knowledge, and have learned first hand the danger in utilizing the data generated by such polls. KDKA runs a highly respected news operation, and KDKA’s use of CRAP for its political coverage is quite surprising. Predictably, the KDKA poll will remain controversial.

Let’s be clear here. Polling in the current environment is not easy under any circumstance. Aggressive marketers have alienated previously agreeable interviewees. Answering machines cut off access. People are less inclined to agree to be interviewed. High response rates are not easy to achieve. In this environment, no single method of conducting polls is without its problems. Even good polls can give bad results. Every pollster has had a polling goof.

But professional pollsters across the country condemn these computer polls. Respected pollster Daniel Yankelovich has been among the most blunt. “The notion that a poll is a poll, and it doesn’t matter how cheap and simple minded it is such a deterioration. It’s Gresham’s Law of junk driving out quality. They couldn’t be more misleading.”

Michael Traugott, a former president of the American Association For Public Opinion Research, has said it best about CRAP polls: “ Until there is more information about their methods and a longer track to evaluate their results, we should not confuse the work they do with scientific surveys, and it should not be called polling.”

If it can’t be called polling, maybe these pseudo surveys will need a new name to describe them. In the meantime, we can just call them "Crap."

Politically Uncorrected™ is published twice monthly. Dr. G. Terry Madonna is a Professor of Public Affairs at Franklin & Marshall College, and Dr. Michael Young is a former Professor of Politics and Public Affairs at Penn State University and Managing Partner at Michael Young Strategic Research. The opinions expressed in this article are solely those of the authors and do not necessarily reflect the opinions of any institution or organization with which they are affiliated. This article may be used in whole or part only with appropriate attribution. Copyright © 2002 Terry Madonna and Michael Young.