05 December 2011

Tricky is in the eye of the beholder

Sometimes I feel like the SAT is a kind of Rorschach test. It’s so laden down with cultural baggage and anxieties (about race, class, social mobility, you name it) that people’s opinions — and they tend to be very, very strong opinions — seem to reveal more about their own concerns than they do about the actual test itself. I also sometimes feel as if people who complain about the SAT’s purported “trickiness” are missing the point of the it: both the questions and the incorrect answer choices are deliberately written to exploit the kinds of mistakes that people are most likely to make when working through the various kinds of questions. The real issue is whether that whole setup is a valid means of testing, well… whatever it is that the SAT is supposed to be testing (which is of course something that no one can agree on anyway).

I do feel obligated to point out that for the small percentage of test-takers whose skills are such that they can disregard the multiple-choice aspect and simply answer the questions, the whole concept of trickiness is essentially a moot point.

I say this because so much of the criticism that gets leveled at the SAT seems to be based on the notion that no one could possibly do well on the SAT without extensive coaching on how to avoid the traps, and that’s simply not true. If it were that impossible, the College Board would never keep administering the test in its current form. Some people can in fact do extraordinarily well with virtually no studying at all. It may be horribly unfair, but they do exist. (I know, I went to high school with them, and no, they weren’t lying about not getting tutored and are now getting Ph.Ds at Stanford in things like particle physics). The fact that other families are willing to spend thousands of dollars so that their children can obtain scores that are even remotely competitive in no way lessens that reality. And no, that isn’t fair either.

I also say this because in my experience, focusing on the trickiness of the answer choices is often a convenient way for people to deflect from the fact that they really didn’t know how to answer a particular question. I’d say that a good 85% of the time, people miss Critical Reading questions because they didn’t totally get what was going in the passage or what the question was actually asking. They might think they understood, but when I ask them to put things in their own words, they usually don’t quite hit the mark. And if they don’t nail the big picture, other things have a tendency to go off as well.

But back to the question at hand, namely what do people actually mean when they say that the SAT is tricky? To start with, I’m going to give an example of a couple of Writing questions that my students routinely have difficulty with:

1. For people in many ancient societies, work was only a means of survival rather than a way to improve your standard of living.

2. At the reception were the chattering guests, the three-tiered cake, and the lively music that have become characteristic of many wedding celebrations.

For the first question, my students consistently pick B (a means of) because “a” always goes before a singular noun, and “means” looks like it’s plural because of -s on the end of it.

For the second question, they pick D (characteristic of) because they interpret it as a noun agreement question and think that it should be “characteristics of,” not realizing that “characteristic of” = “typical of” and is a fixed phrase that doesn’t need to agree.

The first time I had a student get these questions wrong for those reasons, I was a bit taken aback; it had never actually occurred to me that those particular answer choices could be misinterpreted in those particular ways. I just assumed that my student was an anomaly in reading them the way she did. But then the same thing happened with someone else. And then someone else. And before I knew it, I was explaining their reasoning to them before they could even explain it to me. Only at that point did it begin to occur to me that ETS might have deliberately included those answer because kids would be likely to use that reasoning. But they might have also not worked things out that precisely — they might have underlined it randomly, and it just so happened that lots of kids picked that answer in the focus groups.

It’s easy to fall into a kind of circular logic/conspiracy theorist mentality and start thinking that everything that you perceive as tricky was put on the test specifically to trick you, when in reality ETS occasionally does things by chance that lots of people happen to misinterpret the same way. Anyway, I digress.

The point I’m trying (perhaps not very lucidly) to make is that those kids undoubtedly perceived those questions to be evidence of how “tricky” the SAT was. They had learned certain rules about SAT grammar, and now those rules were being broken. Why should they be penalized? And besides, what sort of test would judge them on something so blatantly unfair?

But the problem is that missing those questions tends to correlate with other weaknesses: almost inevitably, the kids who miss those questions are the ones who do not read SAT-level texts independently and who consequently haven’t really developed an ear for what is and is not standard usage — and that in turn shows up in their general writing and reading skills. The ones who do read extensively on their own generally have the same reaction as I did; it simply doesn’t occur to them that it could be a trick. For questions like the ones I mentioned, it doesn’t matter how much coaching someone has received; even if they’ve learned the rules flawlessly, there’s absolutely no way to prepare for all of the exceptions.

Worse, people who conceive of the SAT primarily in terms of tricks are more likely to miss those questions because they tend to only see it in terms of “the way the SAT tests things” and not in terms of how English actually works. They also tend to be the ones who are surprised to discover that concepts tested on the SAT actually have something to do with real life writing. More than one student of mine has been surprised to discover that the rules tested on SAT Writing are not just something made up by ETS.

One last example that I find particularly illustrative: I confess that I sometimes browse College Confidential for insight into the kinds of misconceptions people bring to the SAT, and recently I stumbled across a post in which someone cited the appearance of both “undermine” and “underscore” as Critical Reading answer choices to “prove” that the College Board is out to trick people. Because both words start with “under,” the reasoning went, they can be easily confused, and thus they’re put on the test to confuse people.

Now, from that kid’s perspective, the argument seemed entirely logical: if two words start with “under,” how could someone possibly be expected to remember that they have entirely different meanings? From an educated adult’s perspective, however, that kind of confusion is virtually¬†unthinkable. The fact that two words can have the same prefix but opposite meanings is simply part of the English language. They just mean different things, end of story. There’s no trick whatsoever. And a kid who doesn’t know that is still incredibly naive about how language works. The problem goes far beyond the test.

I know that everyone is intent on denying the relationship between SAT scores and actual skills, but from what I’ve seen, the correlation between those two little questions and overall dexterity with the English language is remarkable. To be clear, I am not saying that someone’s responses to these questions form any sort of final judgment about their capabilities and/or potential, simply that they serve as a remarkably accurate signpost for other kinds of knowledge. It is of course theoretically possible to be a phenomenal writer and brilliant English student and still miss those two questions… but from what I’ve seen, the reality is that most of the time it just doesn’t work that way.

Leave a Reply