In case you missed it, read Part 1 on Sampling and Response Bias and watch the video clip there first.
All the sample surveys from the clip use convenient sampling. There is no mention of any attempt to sample the US population properly to get an idea of what the general public opinion is on any issue. It is likely that there is undercoverage of segments of the population. The language used is usually “x percent of you” feel this way, which is technically correct in the sense that “you” represent the viewers who vote. One can easily get the impression of “you” representing the US population. To be fair, FOXNews (in the first clip) did qualify the statement on the first clip about the poll results with:
93% of you, who know how to text, say no! That we should not be talking to moderate factions of the Taliban
He would’ve been more accurate if he had said 93% of the votes (who knows if callers can vote more than once or whether the 93% who know how to text actually texted) on this poll which is most likely not representative of Americans or even the viewership of FOXNews (convenience samples are rarely representative) have voted NO (negative or strong emotions are typical of voluntary responders).
After the students identified voluntary response bias due to convenience sampling as a problem with all the clips, I asked students to change the wording of the questions so that they’re more neutral, or at the least, use words that are less emotionally charged. This was quite challenging since the textbook does not give examples or guidelines about how to reword the questions. Many students who don’t follow the news or read newspapers or magazine are unfamiliar with political correctness and did not see problems in the wording of questions. Some questions seemed neutral to the students. For most of the questions, they simplified the wording of the question but essentially the tone is similar.
Some of the more successful rewording and observations from students are:
Are you outraged that you are expected to tighten your belt and make tough choices, while Washington goes on a spending binge of historic proportions?
This question essentially is worded as “Are you outraged that you [have to suffer], while [somebody else doesn't]?” Even if we remove the more emotionally charged “tighten your belt” and “make tough choices” along with “spending binge of historic proportions” the sentence is still biased. Students offered “Do you agree with the amount of government spending?” Another student came up with “Do you think that the government should spend more or less to help the economy?”
Are you outraged that big business and socio-ethnocentric special interest groups are trying to destroy the most effective government program to fight illegal immigration?
Once again the question is worded as “Are you outraged that [bad people] and trying to destroy [something good that fights something bad]?” Even if “big business” and “socio-ethnocentric special interest groups”, “destroy the most effective government program”, and “fight illegal immigration” are replaced with more neutral terms the question serve no purpose other than to get people upset at certain groups. One student attempted “How do you feel about businesses and interest groups’ lack of support of immigration laws?” Another tried “Many businesses and special interest groups do not support the enforcement of immigration laws. Do you agree with that position?”
Do you believe it’s time illegal aliens said, “thank you” for all the help and support they get in this country, help and support they don’t get from the countries they’ve left?
Is there any point in asking this question? He might as well have asked “Do you believe [people should say "thank you"] for [all the nice things others do for them], [nice things that no one else would do for them]?” Students just gave up on trying to improve on this one.
Do we need a tax hike?
The way it’s worded is so general as to be meaningless. Most people will oppose tax hikes. In the context of the clip, it’s about providing health care to people who don’t have them. The least they could do is to mention what the tax hike revenue will be used for.
Is it fair to tax the [top] 1.2% of the richest Americans to pay for health care?
It’s basically asking whether the 98.8% of Americans should get a benefit paid for by people who have the most money and derive the most benefit from our economy. How would you vote?
All in all, students had a lot of fun watching the clips but struggled to come up with better and more neutral rewording of questions. They also got a pretty good idea about why most of these polls are bogus. First, the samples are not representative of Americans. Second, the wording of most questions suggest that the pollers are not interested in accurately measuring public opinion.
I was happy the next day when a student told me that she watched some of these news shows with her sister and they had a pretty good conversation about sampling and response bias. Students are learning. If you use this in your class, please let me know how you used it or what activities you did with it. I’d like to know about your experiences.
Lastly, a moment of Zen.
This last clip gives a good reason of why we should ignore internet or phone polls that were not sampled properly.
Sampling and Response Bias (The Daily Show Edition) Part 1
Sampling and Response Bias (The Daily Show Edition) Part 2
