Questioning the Science

3.1 Comparison Groups and Dichotomous Variables

Prior to collecting data, the researcher develops a research design that includes a plan on how the data will be presented. In quantitative research, this typically includes a strategy to compare across groups and share the results side by side, usually in a table format. While this may make sense initially, it is important to consider the ramifications of that decision. For BIPOC communities, this usually means being compared to White peers. For women, this means being compared to men, and so on and so forth. Scholars have also interrogated the binary nature of collecting gender and sex information in surveys with particular attention given to implications for queer and trans* students (Garvey et al., 2019). The normative group that “others” are compared to is the one with the most privilege—plainly stated, in higher education, this group is normally White males. Normalizing privilege is an act of oppression. Comparing groups is also problematic. Using White students as a comparison group perpetuates White supremacy—irrespective of the results of the analysis.

Similar issues arise when artificially creating dichotomous groups for the sake of analysis. In other words, the groups created perhaps were not purposefully targeted at the time of data collection and the researcher later wants to compare data across two groups. Another reason dichotomous variables are created ad hoc is because other categories were lacking in responses. Using race/ethnicity for example, if a survey is administered to all students but there are not enough responses by BIPOC students, the researcher may create a White and non-White group aggregating Black, Latinx, Asian American, Indigenous, and other groups. While statistically this may “balance” the composition of the groups numerically, this decision infers that all non-White individuals share the same lived experience, which in some contexts could be the furthest from the truth.

Finally, when results are compared across groups or dichotomous variables, any deviation from the norm is ascribed to the difference in group membership. In the example above, if the non-White group scores below the White group on an outcome, then the difference in scores is attributed to some deficit due to the student’s race/ethnicity. These conclusions are often made when there is a limited number of independent variables in the dataset and the researcher is searching for disparate outcomes.

3.2 Eliminating Outliers

One of the first steps in “cleaning” quantitative data after completing data collection is for researchers to confirm there is sufficient variability and that the variable distributions are relatively normal. This means that the variables, when illustrated in a histogram, show a bell curve shape. If the data does not, the next step in standard practice is for the researcher to look for outliers and remove them. Eliminating outliers means deleting any data points that lie outside of the bell curve at the extremes. Statistically speaking, outliers skew the data in the positive or negative direction. Removing them allows for better fitting data, since most quantitative methods assume normality. In the real world though, this practice means that researchers are silencing someone’s voice from the study. This action discredits an individual lived experience as invalid, simply because they did not answer like the rest of the respondents.

A study examining response tendencies among university students found that “Hispanic respondents exhibited not only extreme response bias (the tendency to use “1”s and “7”s more often on a seven-point scale) but also used significantly fewer midpoints on the scale” (Culpepper and Zimmerman, 2006, p. 75). The comparison group of White students tended to answer in the midpoint. They conclude their study with a warning to researchers who survey large proportions of Latinx students saying, “In cases, where extreme response is shown, corrective measures—such as normalizing the data through Z scores—should be taken” (Culpepper and Zimmerman, 2006, p. 75). While this recommendation is statistically sound, the underlying notion is that the way Latinx students are responding to surveys is wrong, and thus has to be quantitatively corrected to provide for more accurate comparisons against White students. There is absolutely no questioning on whether the instrument they are using is applicable to Latinx students. This idea of “one size fits all” embodying a color-evasive ideology (Bonilla-Silva, 2010) is pervasive among quantitative assessments. Any group’s divergence from the (White) norm is seen as a deficit and is typically attributed to a categorical difference (race/ethnicity, gender, SES, age, etc.). Harper (2012) in his systematic review of 255 journal articles reminds us that racial justice “will not be achieved if we continue to study race without critically examining racism” (p. 25).

3.3 Non-Response Bias

Another aspect to examine is how the very instruments used to collect data influence the results. In the practice of survey administrations, certain segments of the population are more likely to respond than others. Women are more likely to respond than men; Students of Color, first-generation college students, low-income, and students with lower GPAs are all less likely to respond (Eagan, et al., 2016). To account for non-response bias within a sample, researchers typically apply weights to these groups with lower responses to ensure the sample is representative of the national population (such as the case with the Cooperative Institutional Research Program (CIRP) surveys collected at the University of California Los Angeles). The Freshmen Survey is one example, but the practice of weighting samples is common practice for researchers utilizing large data sets.

Those few voices that do respond to the survey from underrepresented groups then become amplified representatives of their entire group. In other circles, this practice is called essentialism (Delgado and Stefancic, 2012), which is counter to the understanding of diverse groups. What is more, embedded within these extrapolations of data are personal biases. For example, women are more critical of themselves than men. Fifty-year trend data shows that women consistently report higher high school GPAs (Eagan, et al., 2016), yet rate themselves lower in academic ability. From 2017 college freshman data (Stolzenberg, et al., 2019), 51.5% of men report that their average grade in high school was an “A−, A, or A ” while 62.1% of women do. When asked to compare themselves to their peers, 74.8% of men rate themselves “above average” or in the “highest 10%” when considering academic ability as compared to 69.2% of women. Even more dramatic is the difference when asked about intellectual self-confidence: 69.9% of men and 50.2% of women rate themselves “above average” or in the “highest 10%.” Although women earn higher grades (by more than 10 percentage points), they rank themselves lower (almost 20 percentage points) than men. These augmented biases are considered the lesser of two evils. The more important goal, statistically speaking, is to maximize the data to have a more representative national sample to get as close as possible to the actual population.

3.4 Small “n” (Sample)

One of the more egregious practices is to completely leave out student populations in the analyses. The most common reason for doing so is because there are not enough cases in that sample to conduct any meaningful statistical analyses. This is virtually always the case with Native Americans. As a matter of fact, it has become such a common practice that it is considered the norm. Any study that actually has enough representation to present quantitative findings on Native Americans is revered. Typically, the inability to report findings due to small sample size is illustrated with an asterisk on a table with its accompanying note. In some cases, the phrase “not statistically significant” (Shotton et al., 2013) is used to explain the inability to report. Take a minute to pause and reflect on the ramifications on an entire group of people being labelled as statistically insignificant. Beyond inequity and oppression—this action induces trauma. Shotton et. al. (2013) state:

“The absence of data on Native American students reinforces our invisibility, where our presence is hidden by the ever-present “asterisk,” and further marginalizes Native people. Furthermore, the asterisk mentality concerning Natives in academia has resulted in a serious lack of understanding of and dialogue on appropriate solutions” (p. 2).

They further cite Fryberg and Townsend (2008), “invisibility is an intentional act involving an active ‘writing out’ of the story of a particular group, often serving to maintain a status quo that benefits the dominant group” (p. 175). Even if this action is unintentional (as some would argue), it has very real impact perpetuating systemic oppression.

A new approach grounded in QuantCrit that aims to address the issue of small sample sizes is put forth by Van Dusen and Nissen (2020):

“To help interpret the uncertainty around the predictions of our models, we included standard error values for each coefficient. We do not include p values because of their consistent misuse in the sciences [76,78] and in research on equity [99]. Using p values to examine equity is, in part, problematic because scientists often interpret them as go no-go tests. Since p values are sample size dependent, they can show that large and meaningful effects for small groups of students are not statistically significant. Scientists and science consumers often think a result that is not statistically significant is not meaningful, but this is incorrect [78]” (p.8).

This is an alternate way to consider the presentation of results with groups that have smaller sample sizes. The researchers find a new way to present their data while questioning (and curtailing) one of the most established modalities to convey significance: the p-value. Typically, significance is presented in three categories followed by one, two, or three stars (*** p < 0.001, ** p < 0.01, * p < 0.05). The presentation of standard errors instead of p-values is not conventional as the reader has to work through the results more intently to understand them, but it certainly addresses the concern of sample inequality.

The way non-response bias and small sample issues are handled (as described here) exemplify essentialism and invisibility. At the core, they both attempt to address the same challenge: What do you do when there is not enough representation among groups of interest? The inherent tension is palpable. Researchers should keep in mind that the voices of the few should not be essentialized, yet those few voices should not be silenced either. Perhaps one way to address this tension is for researchers to exercise their agency and be more transparent in manuscripts and present the data they truly collect without it being evaluated as sub-par. It is not lost on the author that this level of transparency would also require the peer-review publishing process to undergo a transformation to address systemic inequity, oppression, and supremacy.

3.5 Theory Development

One final example of how White supremacy persists through quantitative methodologies is in the development of a theory. Initially, when a new idea enters the field, it is considered merely a model or a conceptual framework. Only through multiple successful replications of results, conducted by various scholars, will the idea rise to being considered a theory.

If the concept is tested outside of the population that it was developed within, it will likely not yield the same results. It is also important to note that the concepts of validity, reliability, rigor, and meritocracy are cornerstones in the development of a successful survey and they also carry over to the development of new theories. Some of the foundational higher education theories have been validated by scholars for over seven decades. If we consider the homogeneity of the student population in those early decades, it is no wonder that they have successful replication of the results. Today’s student population is increasingly diverse and does not resemble the student bodies of the 1950s, 1960s, or 1970s, yet we still use those theories developed decades ago. As such, new models and frameworks must be developed that emerge from the lived experiences of the emerging majority student groups.

4.1 (Re)Education and (Re)Calibration

Some of this work is already underway. A first step is to educate oneself with different perspectives than those we were socialized to see the world through in our own educational journeys. Foundational literature by Zuberi and Bonilla-Silva (2001, 2008, 2010) are great starting points to initiate this transformational process. A great primer on QuantCrit is the 2018 special issue of Race Ethnicity and Education (Garcia, López, and Vélez, 2018), with these scholars continuing to advance the work in this area. More contemporary examples question the use of algorithms in day-to-day activities and how they perpetuate oppression (Noble, 2018). A closer look at mathematics as a field (Gutierrez, 2015, 2017) and the use of big data (O’Neil, 2016) also provide entry points into this critical conversation. The current and future use of artificial intelligence and the problem raised by its misuse has elevated to the level of the United Nations (2020) developing regulations around bias, racism, and other forms of oppression.

With the increasing reliance on technology, it is important to consider inequity in the online environment. Noble (2018) examines racist and sexist algorithmic bias and the way people are oppressed by digital media platforms. She highlights that most people think of algorithms as a mathematical formulation, where in actuality, they are more about automated decisions. Further complicated is the fact that many search engines (such as Google) are driven by advertisement dollars that dictate certain content appear at the top of the results pages. What do these search engine results mean in a broader social, historical, and economic context? She argues that these tools further marginalize people suffering from systemic oppression, yet the results show up in the platforms as if they are credible, fair, objective, and neutral. Noble provides an example where she Googled “beauty” several years ago, and the results that popped up were exclusively images of White, blonde, blue-eyed models. Another search for “black girls” yielded primarily photos from pornography sites. The virtual monopoly that Google possesses among internet search engines allows for biased results—that privilege Whiteness and perpetuate sexism—to go unquestioned in the way discoverability is created online.

Accepting knowledge as truth is commonplace in a number of academic disciplines. Mathematics is another example of where people believe blindly and take what is presented to them at face value. Along the same lines of argumentation that Noble (2018) presents, in Weapons of Math Destruction, O’Neil (2016) demonstrates how big data increases inequality and threatens democracy. She argues that the decisions that affect our everyday lives are increasingly not being made by humans, but rather by mathematical models. The algorithms that score teachers and students, sort résumés, grant (or deny) loans, evaluate workers, target voters, set parole, monitor our health, and make several other decisions that reinforce discrimination are being conducted by models developed by humans, and individual biases are duplicated in the algorithms.

Quantitative work is built on a foundation of mathematics. Gutiérrez (2015) explains why there is a preconceived notion where mathematics is accepted as truth: “Disconnected from emotions or morals, it is viewed as an ‘objective’ arbiter of ‘truth,’ as having no agenda of its own other than to reflect the natural order of the world. In general, we do not challenge the power mathematics exerts, because we see its properties operating outside of humans” (p. 267). Yet Gutiérrez argues that mathematics itself operates as Whiteness. Those who are credited for doing and developing mathematics, those perceived as capable, and those who are seen as part of the math community are generally viewed as White. In subsequent work, Gutiérrez (2017) proposes a concept she terms “living mathematx,” which describes a new vision for practicing mathematics that incorporates ideas from ethnomathematics, postcolonial theory, and Indigenous epistemologies, among others. She challenges us to interrogate the nature of mathematics and question who does it and how its practice affects us all. In the radical reimagination of math, she calls for “a version that embraces the body, emotions, and harmony” (Gutiérrez, 2017, p.11).

4.2 Training The Future of Higher Education

The second step in addressing the structural inequity, systemic oppression, and White supremacy that is perpetuated in higher education is to address the curriculum. The first step is to educate oneself and conduct research critically, the second is to educate future generations of scholars and researchers. Part of this process is recognizing the shortcomings in the way we teach quantitative methodologies in graduate education. Teaching students “the way we were taught” without introducing critical perspectives (such as QuantCrit) perpetuates the status quo and does nothing to transform the field. As higher education scholars, we also have to consider our own socialization processes. In graduate school we are trained under an established cannon of literature and methodological epistemologies. We are taught to pursue the research “objectively.” If we are part of a research team, those practices enforced by the primary investigators most likely reflect long-standing methods that are accepted as “rigorous,” perpetuating White logic and White methods (Zuberi and Bonilla-Silva, 2008).

The call is to recognize the embedded biases of the tools we were taught to use in examining the student experience and landscape of higher education. The White normative continues to fragment any research that is focused on the non-White experience and is even more problematic when it emerges from BIPOC scholars. There is a disconnect between addressing the research imperatives of historically marginalized groups, yet the very tools that were developed to fragment our communities continue to be used. Even when scholars today quantitatively examine the experiences of BIPOC students, they are utilizing theoretical lenses that originated from data centering White students, at White institutions, with White notions of success. We must be critically aware of these processes and begin to find better ways to carry on this vital re-education.

Luckily, we have models of scholars who have already begun the challenging work of exploring the higher education curriculum. In their article titled “Disrupting Whiteness in Introductory Statistics Course Design” Tabron et al. (2020) employ a duoethnography approach to share their personal experiences in a graduate level statistics course. As the only tenure-track professor of color teaching statistics in the College of Education at a historically White institution, the faculty member examines this power dynamic with two graduate students—who are also Black women—through ongoing discussions about their racialized gendered experiences. At the core of their collaboration is a “pedagogical dedication to disrupting Whiteness in higher education through teaching and praxis, while actively finding innovative ways to promote racial equity” (Tabron et al., 2020, p. 9). They close with a call out to faculty and students to critically interrogate how they have been socialized to teach and consume statistics in ways that perpetuate Whiteness. Other work also highlights the importance of instructors facilitating discussions about their positionality and providing a space for reflexivity in order to interrogate their own biases and how they show up in the classroom (Strunk and Locke, 2019). This process is also important for students to bear witness and take part in the decolonization and re-education process.