The straw man in the standardized testing debate

Frank Bruni wrote a column in yesterday’s The New York Times, in which he expounded on the virtues of college admission committees’ decisions to look past marginal test scores in a handful of underprivileged applicants in order to diversify their classes.

Depending on your perspective, what Bruni describes can either be construed as a noble undertaking or the symptom of a corrupt system that unfairly disadvantages hardworking, middle-class applicants, but I’m actually not concerned with that particular debate here.

Rather, my issue with Bruni’s column is that it perpetuates a common straw man argument in the debate over college admissions — namely, that test scores have traditionally been the be-all end-all of the admissions game, and that only now are a handful of intrepid admissions officers are willing to look past less-than-stellar scores and consider other aspects of a student’s application. 

Bruni is the author of Where You Go is not Who You’ll Be, a book that very validly emphasizes the questionable relationship between name-brand colleges and overall success in life, but in terms of actual admissions, his authority appears to stem primarily from the fact that he turned down Yale to attend the University of North Carolina at Chapel Hill and still managed to land a job as a columnist at the Times. Although he’s generally familiar with the field, he is not actually an expert in admissions the way, say, Paul Krugman is an expert in economics. As a result, it’s hardly a surprise that he misrepresents some of the issues at play.

In short, what is news to Bruni is a long-established practice known as “holistic admissions” — a practice that was, incidentally, introduced in the 1920s, when the Ivies first decided to consider “character” in order to limit the number of Jews. Since then, the purpose of evaluating applicants according to factors beyond grades and test scores has changed again and again, but colleges continue to select students according to their particular set of institutional needs — be it diversity, donors, athletics, or physics research — and test scores play a role in that process only insofar as they garner universities freshman classes with the desired characteristics. For example, a major reason for inflating scores on the new SAT was presumably to allow colleges that aren’t quite ready to go test-optional to admit more applicants from under-represented demographics without compromising their USNWR rankings. The College Board has danced around this fact with various euphemisms about “opportunity,” but it is difficult not to conclude that this type of demographic manipulation was not a driving force. 

For at least four decades, though, admissions committees’ willingness to give disadvantaged applicants a boost has had a very real effect on thousands of students — life-altering effects, in many cases. When people attack colleges for relying too heavily on test scores, they’re obviously thinking of all the other thousands of applicants who didn’t get that boost.

What’s interesting (but not at all surprising), though, is that the other side of the argument is almost never considered — that is, the students who are given every advantage but who never achieve scores anywhere remotely what they would need to be competitive applicants at top colleges are rarely mentioned. Yes, the majority of students achieving high scores are well-off, but it does not follow that every well-off student achieves high scores. As I’ve pointed out before, the lowest-scoring students I worked with tended to be from the wealthiest families. 

As a result, I left Mr. Bruni the following comment, which can be viewed here (and which, might I add, was selected as a Times “top pick!”):


With all due respect, what you describe in this column is holistic admissions, which has long been the policy at the vast majority of selective private colleges in the United States. There’s a reason that schools do not publish — and, to the best of my knowledge, have not ever published — official cut-off scores for applicants. Admissions committees are well aware that applicants come from a wide range of socio-economic backgrounds, and that some applicants face far more obstacles than others. That said, what about the opposite end of the spectrum, e.g. a student who has spent 10+ years in a $40K/year Manhattan private school and whose parents have doled out an additional $10K or more for tutoring, but who can barely break 600 on any section of the (old) SAT? These numbers are not exaggerations, by the way: I tutored students in that demographic for a considerable period, and some of them could not in fact achieve scores that would make them even remotely competitive at most top-25 or so schools. (In case anyone is wondering, money and connections only get you so far). Some of those students were reasonably bright and hardworking, but their scores were also very accurate reflection [sic] of their academic limitations. The fact that there is a correlation between scores and family income does not in itself mean that scores cannot provide an important piece of information when considered in their full context.

The real problem is that test scores mean such different things for different applicants. Sometimes they reveal an awful lot, and sometimes very little. 

In my experience, scores for the most privileged applicants do tend to be a roughly accurate reflection of what those students know.  A slew of 750+ scores from a student at a top private school is by no means indicative of brilliance, but 500/600-range scores from a student at the same school are usually a sign that there are some real gaps. That’s a significant piece of information for an admissions committee to have when evaluating those students against their classmates, as well 30,000 other applicants. 

On the other hand, how is a committee supposed to judge 500-range test scores in an applicant from an academically marginal school and a single-parent household with an income of less than $20,000/year? It would be obtuse to believe that that applicant’s scores did not also reveal some gaps (even though 500-range scores are actually quite an achievement in that context); but the question is what sort of potential other aspects of the application reveal, whether and to what extent they outweigh the test scores, and whether the college has the resources in place to help that student catch up academically to his or her peers. 

The fragmented nature of the American school system and the relationship between real-estate prices and school quality ensures that these are not apples-to-apples comparisons. They’re not even apples-to-bananas comparisons. They’re more like apples-to-skyscrapers comparisons.

Scores are not everything. Admissions officers know this. They struggle with these kinds of calculations every for day, for months, and in the end they just can’t take everyone. Exactly what role test scores should play in the process is up for debate. But to suggest that everyone has just been playing a straightforward numbers game all long… well that’s just not true.