Introduction
On November 14, 1991 the Wall Street Journal printed an article by staff reporter Cynthia Crossen about the growing trend towards using research studies as a powerful tool of persuasion. She points out that while there are many examples of good research studies, some of the reports and the resulting claims may be biased and should be viewed with caution if not skepticism. She writes:
Today, studies have become vehicles for polishing corporate images, influencing juries, shaping debate on public policy, selling shoe polish and satisfying the media’s – and the public’s – voracious appetite for information. Yet while studies promise a quest for truth, many today are little more than vehicles for pitching a product or opinion.
To support her claim, Crossen provides several examples of studies in which bias may affect the reported results. We present some of her examples along with additional examples collected from other sources.
Synopsis
Abstract
Today’s news and advertising media are filled with reports of the statistical results from research studies. This story presents a number of examples where the persuasive arguments may be the result of biased studies.
Data Set
Anecdotal examples
Extensions
Project ideas
1 Question
Sources of bias, experimental design
Basic: Q1
Results
Levi’s Jeans
A survey conducted by Levi Strauss & Co. provided students with a list of clothing and asked which clothes would be most popular this year. They reported that 90% of the students said Levi’s 501® jeans.
They were the only jeans on the survey’s list.
Black Flag®
A survey for Black Flag® said: “A roach disk…poisons a roach slowly. The roach returns to the nest and after it dies is eaten by other roaches. In turn these roaches become poisoned and die. How effective do you think this type of product would be in killing roaches?”
79% said effective.
Disposable-Diapers
A Gallup poll sponsored by the disposable-diaper industry asked: “It is estimated that disposable diapers account for less than 2% of the trash in today’s landfills. In contrast, beverage containers, third-class mail and yard waste are estimated to account for about 21% of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers?
84% said no.
USA Today
A call-in poll conducted by USA Today concluded that Americans love Donald Trump.
USA Today later reported that 5,640 of the 7,800 calls for the poll came from the offices owned by one man, Cincinnati financier Carl Lindner.
Chrysler
A study sponsored by Chrysler contends that Americans preferred a Chrysler to a Toyota after test driving both.
The study was conducted using 100 people in each of two separate tests involving both auto makers. None of the subjects owned a foreign car.
Diet Products
A news release for a diet products company reports: “There’s good news for the 65 million Americans currently on a diet.” Its study showed that people who lose weight can keep it off.
The sample: 20 graduates of the company’s program who endorse it in commercials.
American Express® and France
A study sponsored by American Express Co. and the French government tourist office found that old stereotypes about French unfriendliness weren’t true.
The respondents: “More than 1,000 Americans who have visited France more than once for pleasure over the past two years.”
Holocaust Survey
A 1992 Roper poll asked “Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” Twenty-two percent of the American respondents said “seems possible.”
A reworded poll in 1994 asked “Does it seem possible to you that the Nazi extermination of Jews never happened, or do you feel certain that it happened?” This time only 1% of the respondents said it was “possible it never happened.”
Midway Airlines
Ads in the New York Times and the Wall Street Journal stated that “84 percent of frequent business travelers to Chicago prefer Midway Metrolink to American, United, and TWA.”
The survey was “conducted among Midway Metrolink passengers between New York and Chicago.”
Roll Over or Get Tough
During late 2009, a contract deadline loomed between Time Warner Cable and the broadcasting company Fox. It was reported that Fox wanted a $1 per subscriber per month broadcasting fee, but Time Warner wanted to pay substantially less. If a fee agreement was not reached, there was a possibility that any stations owned by Fox would no longer be available to Time Warner subscribers. To drum up subscriber support, Time Warner set up the website rolloverorgettough.com. Anyone logging in to the website could choose to “roll over” (accept Fox’s terms for the new contract) or “get tough” (fight Fox’s broadcasting fee increase at the risk of losing Fox-owned networks). Anyone who chose the “roll over” option on the website was directed to a page with links allowing the respondent to change their vote to “get tough”. No such options occurred if one chose the “get tough” option.
Time Warner claimed overwhelming support from subscribers. Their regional vice president Patricia Fregoso-Cox pointed out that over 90% of the almost 600,000 respondents chose to “get tough”.
Questions
For each of the anecdotes, suggest an honest research question, describe the source(s) of bias in the study, and provide a possible approach to eliminate the bias from the study.
a) Levi’s Jeans
b) Black Flag®
c) Disposable Diapers
d) USA Today
e) Chrysler
f) Diet Products
g) American Express® and France
h) Holocaust Survey
i) Midway Airlines
j) Roll Over or Get Tough
Below is a sample solution for part j) Roll Over or Get Tough. Here’s the research question:
Do Time Warner Cable customers prefer to accept Fox’s contract terms and face higher subscription rates or do they prefer Time Warner to fight Fox’s contract terms at the risk of losing channels owned by Fox?
There are several sources of bias. One, since this is an internet poll, the 600,000 individuals are a voluntary response sample. This makes the sample biased towards individuals who have a strong opinion regarding the issue at hand. It should also be noted that this voluntary response sample is not limited to only Time Warner customers, as anyone could access the webpage and vote. So, the sample includes individuals who are not even in the population of interest. Two, Time Warner’s website has biased question wording, as it refers to one alternative as “rolling over” and another as “getting tough”, which makes the latter option seem far more appealing. Three, the fact that the poll only allows those who chose “roll over” an option to change their vote creates a bias toward the “get tough” option. The lack of a simple random sample from the desired population, biased question wording, and different treatment of respondents depending on their initial vote makes any statistical inference from the information gathered in this study meaningless, even though there is a very large sample size.
To eliminate the bias, several things need to be done. The study should be performed by an independent company. They should draw a random sample of Time Warner subscribers, and ask the respondents the following question, or something similar, without wording that favors one response over another:
Would you prefer that Time Warner accept Fox’s contract terms and risk a rate increase, or do you prefer that Time Warner attempt to fight Fox’s contract terms and risk losing channels owned by Fox?
Finally, an option to change the vote should not be available only to those who choose to accept Fox’s proposed contract terms.
References
Adler, J. (July 25, 1994)
Barnett, A. (1994)
Crossen, C. (1991)
Strauss, B. (29 December 2009). "Time Warner-Fox showdown". Los Angeles Daily News. 30 December 2009.
http://www.dailynews.com/search/ci_14090371?IADID=Search-www.dailynews.com-www.dailynews.com.
Projects
Collect a series of advertisements from magazines and newspapers in which statistics are used as a means of persuasion. If possible, locate the studies from which the statistics were taken and examine the studies for potential sources of bias similar to those in this story. If the studies are unavailable, design unbiased studies that would investigate the advertised claim.
Credits
This story was prepared by Greg Elfring. The anecdote about the Holocaust Survey was added by Kathleen Fritsch. This story was updated by Josh Svenson on December 30, 2009.