A while back, in the course of my discussion about how to choose between the current SAT, the new SAT, and the ACT, I mentioned in passing that the SAT would no longer be written by ETS. Larry Kreiger (of Direct Hits and APUSH Crash Course fame) posted a comment expressing his surprise and asking what my source was for that information.
I responded somewhat sheepishly to Larry that I didn’t actually remember — I had been given the information so long ago that I actually no longer recalled who had told it to me. Until Larry asked me, I had just assumed it was common knowledge, or at least somewhat common knowledge.
As I pointed out in my previous post, the tests released thus far seem sloppier and less consistent than what I’ve come to expect from ETS (more about that another time). Even when correct answers were justified, they seemed to be lacking the precision I’ve come to associate with the SAT. Granted, that could be because the details of the test are still being worked out, but these questions simply didn’t have an ETS feel. Having spent countless hours analyzing SAT questions in order to mimic them as effectively as possible, I think my instincts are pretty reliable.
Larry’s response to me, however, was as follows:
I can tell you that the CB has a long standing contract with the ETS. In fact they have a person who earns a six figure salary whose job is to monitor the contract. Given the lack of an authoritative source I believe it is likely that the ETS is writing the new test. I do agree that the questions are below the usual ETS standards.
I was willing concede that I – or, rather, my source – had been mistaken, but now my curiosity was piqued. I didn’t want to be responsible for disseminating misinformation, but something about those questions seemed “off.”
I did some googling, which turned up absolutely nothing.
I also tried calling the College Board where, after a several surprised silences, various representatives quickly passed me off to other representatives, who eventually left me right back at the original menu. I considered making some further attempts; however, after considering that any response I did manage to elicit would inevitably consist of a non-answer involving edu-babble about “best practices,” “evidence-based standards,” “21st century skills,” and “preparation for college and career readiness,” I decided I was better off pursuing other avenues.
I got back in touch with the blog-reader who had worked for ETS several decades ago and had written to me to express her surprise at the clearly lowered standards. She mentioned that she’d heard David Coleman had “cleaned house” and had fired many experienced ETS writers, and that some of the sloppiness could be attributed to that.
When I thought about it, though, that didn’t make sense. My understanding has always been that the College Board and ETS are separate entities; ETS has traditionally contracted to write the SAT. Why would the head of the College Board have the discretion to fire ETS employees?
I posted on the LinkedIn SAT prep teachers forum, where one tutor (and former ETS writer) with ETS contacts reported that she couldn’t get a straight answer out of anyone affiliated with that organization.
While trying to find more information online, I stumbled across “The Revenge of K-12,” by Richard Phelps and R. James Milgram, which confirmed that “house cleaning” did in fact occur. For those of who haven’t been following my blog, Jim Milgram is an emeritus professor of math at Stanford and one of two members of the Common Core validation committee who refused to sign off on the Standards; he has since co-authored a number of papers presenting in-depth critiques of the process by which they were created and implemented.
As Milgram and Phelps write:
Prior to Coleman’s arrival, competent and experienced testing experts suffused the College Board’s staff. But, rather than rely on them, Coleman appointed Cyndie Schmeiser, previously president of rival ACT’s education division, as College Board’s Director of Assessments. Schmeiser brought along her own non- psychometric advisors to supervise the College Board’s psychometric staff. While an executive at ACT, Schmeiser aided Coleman’s early standards- production effort from 2008–2010 by loaning him full-time ACT standards writers. (It should be no surprise, then, that many of the “college readiness” measures and conventions for CCS-aligned tests sound exactly like ACT’s.)
I was aware that Coleman had brought in a number of people from the ACT, but prior to reading the article, I had not realized that Coleman had brought in new, less qualified people to advise the psychometricians themselves. (This is hardly a surprise, though – as Coleman has indicated in the past, he’s not particularly interested in whether his hires are qualified.)
I emailed Jim Milgram, who told me that unfortunately he had no more information regarding who was writing the actual test than what I had turned up.
Then, a couple of days later, I happened to find myself at the house of a friend whose son is a junior. My friend wasn’t sure whether I’d seen this year’s practice PSAT booklet, so she made a point of showing it to me. I’d already seen the test, but as I looked over all the fine print, I realized I should check it for references to ETS. I couldn’t find any, which meant nothing in itself, but then something else occurred to me – perhaps I could compare this year’s booklet to previous years’ booklets and see whether those booklets contained references to ETS.
Sure, enough, at the bottom of the back page, previous years’ PSAT booklets contained a standard disclaimer stating that the views presented in the passages were not intended to reflect those of the College Board, the National Merit Corporation, or ETS.
This year’s booklet did not mention ETS.
Furthermore, references to the SAT on ETS’ website link back only to the current version of the test; aside from one mention of an ETS testing code, I could find no reference to ETS on any of the material intended for post-January 2016 use.
I realize that this is no way represents conclusive proof, but it does suggest that ETS is playing a less prominent role in the writing of the new test than it did in the old.
If it is in fact true that ETS is no longer writing the SAT, it would mark the end of a nearly 70-year relationship and create an even more radical break with the current exam than what has already been publicized.
SAT questions have always been written by a motley group – professional test-writers, teachers, even students – but they have also undergone an extensive, rigorous field-testing process closely supervised by people qualified to supervise it.
So if ETS is no longer responsible for writing the test (or assembling groups to write the test) and overseeing the test-writing process, then who is?
A group of test-writers handpicked/led by David Coleman? (We know how well his attempt at national standards-writing has been received.)
Former ACT writers and remaining College Board employees deemed sufficiently loyal to the Coleman regime?
Khan Academy employees?
Or, dare I say, it…might Pearson be somehow involved? (I really did think that was a stretch until I saw Dipti Desai’s graphic; I emailed her to ask what information she had about the connection, but I still haven’t heard back.)
Based on Milgram and Phelps’s report, it would certainly seem that regardless of who is writing the actual questions, the people supervising the process have far less expertise than those who did so in the past.
Furthermore, with the experimental section gone, there will no longer be a way for new questions to be tested out nationally on an actual group of test-takers. That absence of an experimental section is, I imagine, a significant part of the reason ACT scales can be so unpredictable; the questions just aren’t vetted as rigorously.
Say what you want about the SAT, but it is nothing if not consistent. It’s actually quite remarkable to watch a student take a test administered in 2007 and one administered in 2014 and get exactly the same score. In the absence of an experimental section, it’s hard to see how that kind of consistency will be retained.
In reality, though, the new test is not really the SAT at all. Call it a modified (ripped-off) ACT, a Common Core capstone exam, or just a grab for lost market share. The name is simply being retained because altering it would risk calling attention to the extremity of the changes and create too much potential for backlash.
Likewise, the return to the 1600 scoring scale is a carefully calculated distraction, designed to make adults think that the test will be closer to what it was when they were in high school and thus not to bother to investigate further.
Somehow, though, I wouldn’t be surprised if the same problems that have plagued other Common Core-aligned tests (PARCC, SBAC) start cropping up with the new SAT. Witness, for example, this discussion on College Confidential about determining cutoffs for National Merit. Somehow, I don’t recall so many pages of a thread ever being devoted to something so basic in the past. Elegance and transparency…right. And I’m guessing that this is only a warm-up for what’s to come.
Trackbacks/Pingbacks