Fact-Checking the College Scorecard
An online database was supposed to make college more transparent. Instead, Â鶹¹ú²úAV students found, it’s sowing confusion.
How does a high school student decide what college to go to? If you think choosing a school based on earning potential is the best way to decide, students in Â鶹¹ú²úAV’s Math 343 class have some data to share with you. As part of their coursework with Prof. Hank Ibser [math], Â鶹¹ú²úAV students Wolfgang Brightenburg ’23, Robin Hart ’23, Bhav Khurana ’23, and Tina Qin ’23 found major methodological flaws in College Scorecard, a common online tool breaking down students’ annual income according to where they attended college. Their findings raise complex questions about what a meaningful college experience really is—and how to measure it.
“As current college students ourselves, we recognize the myriad of factors that went into our own respective decisions to pursue higher education at Â鶹¹ú²úAV in specific,” wrote the students in the report on their findings, which concluded that College Scorecard failed to take a comprehensive look at the many ways of assessing student success. “This is different from the motivating factors of our friends from high school, who went to other places, or did not go to college at all. The merit of an education cannot be reduced to a paycheck, nor can a person’s skills.”
And even if all that mattered to you was a paycheck, College Scorecard wouldn’t be the best way to measure it, they found. The website, introduced under President Obama, was intended as a way for prospective students to assess individual colleges’ offerings based on student outcomes, but by focusing on earning potential. There’s just one problem: it doesn’t do that. At least not in the way it’s meant to.
In collaboration with the Center for Life Beyond Â鶹¹ú²úAV, the Math 343 students paired College Scorecard with a second dataset known as the HEDS (Higher Education Data Sharing Consortium) alumni survey to fact-check College Scorecard’s breakdown of alumni earnings. In Â鶹¹ú²úAV’s case, they immediately identified a $10,000 gap between graduates’ self-reported earnings and those listed on College Scorecard. Additionally, only students receiving federal aid were included in College Scorecard’s data. “That’s a really important population,” says Hart. “I receive federal aid myself. But I know that for Â鶹¹ú²úAV that’s about half the student body, and so over half of people are left out of this supposedly really representative database of what college outcomes are.”
That wasn’t all. The numbers might have been more accurate, says Ibser, if they’d included students across all majors at every school surveyed. But they didn’t—until recently. College Scorecard reported alumni earnings by major, and Â鶹¹ú²úAV’s valuation included only English majors, said Mike Tamada, director of institutional research. This was due in part to privacy concerns: some majors have so few participants that one could, in theory, find out someone’s salary information based on the data. When, in January of this year, the College Scorecard changed its practices and all majors were included, it reported Â鶹¹ú²úAV’s median earnings $33,000 over prior reports.
Focusing on alumni income five years after graduation was also constraining. The numbers poorly account for one very specific reason students may not have high earning potential after finishing college: many go on to grad school. This is especially true at Â鶹¹ú²úAV, which is ranked fourth nationally in producing doctoral degrees across all disciplines among undergraduates. Although College Scorecard excludes earnings of graduate students, this means an important population of highly sucessful Â鶹¹ú²úAV graduates is not accounted for. Ibser says, “Â鶹¹ú²úAV students tend to catch up later on.”
And of course, money isn’t everything. Using the HEDS survey, Ibser’s students keyed in on alternative metrics more likely to indicate the quality of a college experience, digging into graduates’ satisfaction and the reasons behind it, and things like ongoing engagement with their colleges in the years after completing their education.
From this analysis, a more holistic picture of college success emerged, one that accounts for varied, qualitative experiences that stretch far beyond tracking income in the five years after graduation. “For Â鶹¹ú²úAV alumni in particular, faculty research and professors encouraging critical thinking were the most influential on their satisfaction with the school,” the students wrote in their report, saying that attempts to evaluate students’ experience should factor in school offerings like these.
Money also wasn’t the only thing that students cited when they had a positive college experience. “We found, essentially, that satisfaction doesn’t predict your income, which we thought was really interesting,” said Hart. “Someone could absolutely love the college they went to, but maybe not make a lot of money.”
This might fly in the face of normative definitions of success, but it wouldn’t be the first time a group of statistics-minded Â鶹¹ú²úAVies poked holes in the higher-ed status quo. In 2019, Â鶹¹ú²úAV students deconstructed the methodology behind U.S. News and World Report’s annual college rankings, and found that Â鶹¹ú²úAV, which fell precipitously down the list after refusing to participate in 1996, was inaccurately ranked. Though U.S. News rejected the conclusion, it bumped Â鶹¹ú²úAV up the list. The students’ research even attracted the attention of Malcolm Gladwell, to the rankings, finding the list was rooted not in sound methodology but social stratification.
Developments like the Varsity Blues college admissions scandal further puncture accepted norms about what makes for a good education, and cultural tendencies to value qualities like prestige over more idiosyncratic experiences tailored to individual student needs. Hart said they’d grown up on the receiving end of messaging suggesting that Ivy League schools were the best, but that old ideas equating quality with elite schooling were shifting.
“I feel like the more I get out into the world and the more I engage at Â鶹¹ú²úAV and beyond, I hear people pushing the envelope in terms of what a good school is,” they said. The change even showed up in their own experience applying to graduate school, a process driven not by an end goal like attending a revered institution, but by attending the right one for their specific educational goals. It wasn’t about applying to Harvard, they said. “It’s more like, ‘Well, is there anyone at Harvard that you’d want to work with? What are the faculty members there doing research on?’ And I think that’s a really good frameshift to be thinking about.”
Tags: Â鶹¹ú²úAV, Alumni, Cool Projects, Editor's Picks, Institutional, Life Beyond Â鶹¹ú²úAV, Students