Mathing…

In case you missed it, read Part 1 on Sampling and Response Bias and watch the video clip there first.

All the sample surveys from the clip use convenient sampling. There is no mention of any attempt to sample the US population properly to get an idea of what the general public opinion is on any issue. It is likely that there is undercoverage of segments of the population. The language used is usually “x percent of you” feel this way, which is technically correct in the sense that “you” represent the viewers who vote. One can easily get the impression of “you” representing the US population. To be fair, FOXNews (in the first clip) did qualify the statement on the first clip about the poll results with:

93% of you, who know how to text, say no! That we should not be talking to moderate factions of the Taliban

He would’ve been more accurate if he had said 93% of the votes (who knows if callers can vote more than once or whether the 93% who know how to text actually texted) on this poll which is most likely not representative of Americans or even the viewership of FOXNews (convenience samples are rarely representative) have voted NO (negative or strong emotions are typical of voluntary responders).

After the students identified voluntary response bias due to convenience sampling as a problem with all the clips, I asked students to change the wording of the questions so that they’re more neutral, or at the least, use words that are less emotionally charged. This was quite challenging since the textbook does not give examples or guidelines about how to reword the questions. Many students who don’t follow the news or read newspapers or magazine are unfamiliar with political correctness and did not see problems in the wording of questions. Some questions seemed neutral to the students. For most of the questions, they simplified the wording of the question but essentially the tone is similar.

Some of the more successful rewording and observations from students are:

Are you outraged that you are expected to tighten your belt and make tough choices, while Washington goes on a spending binge of historic proportions?

This question essentially is worded as “Are you outraged that you [have to suffer], while [somebody else doesn't]?” Even if we remove the more emotionally charged “tighten your belt” and “make tough choices” along with “spending binge of historic proportions” the sentence is still biased. Students offered “Do you agree with the amount of government spending?” Another student came up with “Do you think that the government should spend more or less to help the economy?”

Are you outraged that big business and socio-ethnocentric special interest groups are trying to destroy the most effective government program to fight illegal immigration?

Once again the question is worded as “Are you outraged that [bad people] and trying to destroy [something good that fights something bad]?” Even if “big business” and “socio-ethnocentric special interest groups”, “destroy the most effective government program”, and “fight illegal immigration” are replaced with more neutral terms the question serve no purpose other than to get people upset at certain groups. One student attempted “How do you feel about businesses and interest groups’ lack of support of immigration laws?” Another tried “Many businesses and special interest groups do not support the enforcement of immigration laws. Do you agree with that position?”

Do you believe it’s time illegal aliens said, “thank you” for all the help and support they get in this country, help and support they don’t get from the countries they’ve left?

Is there any point in asking this question? He might as well have asked “Do you believe [people should say "thank you"] for [all the nice things others do for them], [nice things that no one else would do for them]?” Students just gave up on trying to improve on this one.

Do we need a tax hike?

The way it’s worded is so general as to be meaningless. Most people will oppose tax hikes. In the context of the clip, it’s about providing health care to people who don’t have them. The least they could do is to mention what the tax hike revenue will be used for.

Is it fair to tax the [top] 1.2% of the richest Americans to pay for health care?

It’s basically asking whether the 98.8% of Americans should get a benefit paid for by people who have the most money and derive the most benefit from our economy. How would you vote?

All in all, students had a lot of fun watching the clips but struggled to come up with better and more neutral rewording of questions. They also got a pretty good idea about why most of these polls are bogus. First, the samples are not representative of Americans. Second, the wording of most questions suggest that the pollers are not interested in accurately measuring public opinion.

I was happy the next day when a student told me that she watched some of these news shows with her sister and they had a pretty good conversation about sampling and response bias. Students are learning. If you use this in your class, please let me know how you used it or what activities you did with it. I’d like to know about your experiences.

Lastly, a moment of Zen.

This last clip gives a good reason of why we should ignore internet or phone polls that were not sampled properly.

Sampling and Response Bias (The Daily Show Edition) Part 1
Sampling and Response Bias (The Daily Show Edition) Part 2

Your assignment is to identify as many examples of bias as you can. Here’s a list to work from:

Sampling Bias is caused by samples that are not representative of the population. These typically result from faulty sampling design.

• Voluntary Response Sampling (Self-Selection Bias) – A sample that is often biased toward those with strong opinions or those who are strongly motivated. People with very negative opinions tend to respond more often than those with equally strong positive opinions. These samples are rarely representative of the population even if every individual in the population may have been offered the chance to respond.
• Convenience Sampling – A sample that is created by including the individuals who are at hand. These samples are rarely representative of the population.
• Undercoverage Bias – A sample where a significant portion of the population is not sampled at all or has a smaller representation in the sample than it has in the population.

The following bias is not caused by design but can be a factor in the results.

• Nonresponse Bias – Bias caused when a significant portion of the sample does not respond to the sample survey. The people who do not respond may differ from those who do. The results may not be representative even if the initial sample created by design may be representative of the population.

Response bias is caused by anything in the survey design that influences the responses.

• Interviewer (Response Bias due to Interviewer) – Bias caused by who the interviewer is (e.g. a woman interviewer asking if women are ready to be president), the appearance of the interviewer (e.g. a girl with heavy makeup asking if girls should wear makeup), or the action/reaction of the interviewer (e.g. an interviewer showing surprise or judgment at a respondent’s answer or showing positive or negative emotions about the survey question).
• Wording (Response Bias due to Wording of question) – Bias caused by the wording or phrasing of the question in the sample survey.

I like these clips more than the standard educational or instructional resources. The examples are not as contrived and students get to apply critical thinking from what we learn in class to something anyone can see on TV (given the shows’ own claims regarding the size of their audience, students are bound to know people who watch the shows in this clip). The Daily Show writers did the bulk of the work for us Statistics teachers, if they keep this up they may be inadvertently growing their audience.

And yes, I preface this to students that it’s not about the politics when I show this clip. It’s about understanding how Statistics is used by people to serve their agendas. It’s about not falling for what these TV shows pass off as evidence of the views of Americans.

Note: Definitions from Stats: Modeling the World by Bock Velleman and De Veaux.

Sampling and Response Bias (The Daily Show Edition) Part 1
Sampling and Response Bias (The Daily Show Edition) Part 2