22 Arguments VII: Analyzing Generalizations
In this chapter we’ll look in more detail at generalizations. In doing this, we’ll see what is required for a good, or strong, generalization, and what makes for a bad, or weak, one.
Let’s start with a more precise definition than we gave in the previous chapter:
- A generalization is an argument that draws a conclusion about all the members of a group (= the target group) based on knowledge of a subset of members of that group (= the sample group). (Each has one premise, listed above the conclusion coming.)
- This means that the target group is the larger group that a generalization draws a conclusion about.
- And the sample group is a subset of the target group, information about which is used as the basis for inferring the conclusion about the target group.
Examples of Generalizations, with Target and Sample Groups Identified
- Almost all of the students in the classes I teach at IUSB have part-time or full-time jobs.
Conclusion: almost all of the students at IUSB have part-time or full-time jobs.
Target Group: All IUSB students
Sample Group: IUSB students in my classes
2. More than 80% of two thousand randomly selected Americans polled about their views on Congress disapproved of the job it is doing.
Conclusion: more than 80% of all Americans disapprove of the job Congress is doing.
Target Group: All Americans
Sample Group: Two thousand randomly selected Americans
3. A new cholesterol medication lowered cholesterol in 30% of the people who were given it in a clinical trial.
Conclusion: the medication will lower cholesterol in 30% of all people who take it.
Target Group: All people who will take the medication
Sample Group: The people given the medication in the trial
4. I met this guy from Kalamazoo who was a real jerk.
Conclusion: everyone from Kalamazoo are jerks.
Target Group: People from Kalamazoo
Sample Group: The one guy from Kalamazoo whom I met
5. 74% of people who responded to an NPR online poll (which got several thousand responses) said they agreed with the statement that Obamacare should remain the law of the land.
Conclusion: most Americans support Obamacare.
Target Group: All Americans
Sample Group: Several thousand respondents to NPR’s poll
6. Several hundred randomly taken samples of water from the river showed high levels of pollutants when tested.
Conclusion: the river is highly polluted.
Target Group: The river (i.e., all the water in the river)
Sample Group: Several hundred samples of water from the river
Important note on how to identify the groups: In many of these, the argument makes reference to a percentage or portion of a group, but when the target and sample groups are identified, these percentages or portions are not mentioned. These percentages reflect claims about the groups. But the groups themselves are the whole sets of whatever people or things about which the percentage claim is being made.
So, when is a generalization strong? The basic idea is that, if the sample group mirrors in relevant ways the larger group of which it is a part, then you can legitimately infer claims about the target group from the sample group. But, if the sample is too small, or is in some way chosen so as not to mirror the larger group, this inference will be problematic.
More precisely:
Criteria of Strength for a Generalization
For a generalization to be strong, the sample group must be…
- … sufficiently representative of the target group.
- Specifically, this means that the members of the sample group must be similar to (share features or properties with) the members of the target group in whatever ways are relevant to whatever is at issue in the argument.
- Unrepresentative sample groups that do not capture the relevant features of the target group are said to be biased in some way
- The best way to get a representative sample group is to have its members be randomly chosen from members of the target group.
- … large enough to capture all the relevant features of the target group.
- There is no general rule for what counts as large enough – it depends on what the members of the group are and what about them is at issue. In some cases it’s pretty obvious when the sample is too small, but not always.
- Surprisingly, however, if the sample group is well chosen, a fairly small sample can be adequate. For instance, polls that draw conclusions about what tens of millions of voters are likely to do often rely on surveys of only a few hundred or a few thousand, and yet when done well they can be fairly accurate. (See the Important Note on Polling below.)
Based on this we can define our first fallacy, or error in reasoning. An inductive generalization that is weak because the sample is too small is said to be a hasty generalization.
One of the most common causes of weakness is self-selection of the target group. Self-selection occurs when you have people volunteering to be part of the sample in a study or survey, rather than being randomly chosen.
For example, when a news source asks readers to say whether they agree or disagree with something, the results only represent the views of those readers who have chosen to participate. But they may not be representative of all readers, since those willing to put the effort in to participate may hold more extreme or stronger views than those who aren’t willing. And even if they represent all the readers or viewers of that source, since news sources are themselves chosen, and tend to reflect the political and cultural views of those who choose them, what’s true of the readers or viewers of one source may not generalize to the population as a whole. So, even if the sample is very large, in this case you can’t be sure it’s representative.
Given what it takes for a generalization to be strong, are the above examples of generalizations strong? You may not easily be able to answer one way or another. What you should be able to do, however, is specify what you would need to know in order to answer that question for each argument.
Here’s how the analysis would go for each of those examples.
The Above Examples Analyzed for Strength
- You need to know (a) whether the students I teach are representative of IUSB students generally and (b) whether I teach enough students to make a good generalization. As it happens, since I teach a lot of classes that meet the general education requirement in critical thinking, I get both a lot of students and a pretty wide selection of students. So, there’s nothing that would obviously bias the sample. If I’m right about that, it’s a pretty strong argument. But until I told you, you wouldn’t know enough about what I teach to judge.
- You need to know (a) whether the two thousand Americans are representative of Americans generally. Since it tells you that they are randomly selected, you can reasonably conclude that it is a representative sample. And you need to know (b) whether two thousand is a sufficiently large sample. It might not seem like it, given that there are over 300 million Americans, but if the poll was conducted properly, it is, in fact, enough. (See the IMPORTANT NOTE ON POLLING IN THE BOX BELOW.)
- You need to know (a) whether the members of the clinical trial are representative of those who will be taking the medication. What’s stated doesn’t give you enough information to determine that, but if the trial is of the sort routinely conducted by drug companies, they will likely have taken care to have chosen a representative sample. You also need to know (b) whether a sufficiently large enough group participated in the trial. Again, you are not given enough information to decide, but if the trial were real, then you can be reasonably sure the sample size would have been large enough. (As with polling, sample groups in clinical trials are quite small relative to the projected target groups.)
- You need to know (a) whether this guy is representative of all Kalamazooans. It’s almost certain that he’s not – even a small city will have lots of different kinds of people. And you need to know (b) whether a sample of one is large enough. It almost never is, so this is a clear example of hasty generalization.
- You need to know (a) whether the thousands of respondents to NPR’s online poll are representative of all Americans. But there are two sorts of bias to be concerned about here: NPR tends to have a listener base that is somewhat more liberal than the country as a whole; additionally, as with any such poll, you have self-selection of the respondents, which means they are not randomly chosen. This is enough to conclude that this is a weak. You can also ask (b) whether the sample group is large enough. And in this case, given what decent national polls require, it is. So the problem is not hasty generalization.
- You need to know (a) whether the several hundred samples of water are representative of the river water as a whole. Since the example doesn’t specify anything about the method of sampling, you have no way of knowing if there is any bias involved. If they were randomly chosen, or deliberately chosen from a wide variety of locations, then they probably would be representative. If they were all taken from as close as possible to the drainage pipes of industrial plants, they wouldn’t fairly reflect the water of the river as a whole. The other question is (b) whether several hundred is enough samples. Unless you are a water quality testing expert, you probably don’t know. So this is an argument where the right thing to say is that it might be strong, and there’s not enough information to conclude that it is weak, but without knowing more you can’t say for sure. By arriving at that analysis, you at least know what questions to ask to find out what you need to know.
A couple of these examples rely on polls. Our intuitions about what sample size is needed for a good poll tend to be way off. So please pay attention to this (and follow the link):
- IMPORTANT NOTE ON POLLING: There is a fairly well-developed science of polling, so pollsters know how many people they need to get accurate results. A well-crafted poll might survey only a few hundred people and yet be able to make a strong claim about a very large group, like the population of a state or the country as a whole. National polls on political matters, for example, routinely involve around a thousand people, give or take. Thus, one of the most common errors in exercises is not remembering that good polling does not necessarily require large samples.
- Here’s a helpful webpage (with additional links at the bottom) to show you how accurate polls are with different sized sample groups: https://www.sciencebuddies.org/science-fair-projects/references/sample-size-surveys
