What Do the SATs Measure?

Twenty years ago today, I became something of a marked man. It was a Friday and I received a phone call anxiously awaited telling me that ETS was offering me its newly created position of Chief Learning Officer. I had gone over to the dark side of the people who made the SAT. Of course, ETS doesn’t own the SAT; The College Board always has. (And it drove the people at the Board bonkers that the public didn’t understand who was really in charge of the test even if at the time I was hired almost all of the work involved with making the tests and scoring the tests was done by ETS employees.)

Yet most people don’t know or care about the College Board proprietary feelings and when at a party or family gathering, my new job came up in conversation there was always someone who attached him or herself to my side in order to unleash an attack on the SAT. I never had anybody go on and on about how much their AP course sucked or that the English Literature achievement test in 1969 focused too much on interpreting Robert Frost poems. My Chinese sisters-in-law — yes, I have two of them — did tell me how they thought I personally should improve the marketing of the TOEFL and TOEIC exams, which they already admired. But most of the incoming vocal missiles were about the SAT and its evil aspects including how early you have to get up in the morning to take it. (Best advice for kids taking it? Get a really solid night’s sleep and have a healthy breakfast. Also, read, read, read during their entire childhood.)

And then there were the people who would send me the latest article criticizing, damning, or just blasting the SAT. They always seemed to miss the point that everybody at ETS knows about and reads those articles. That’s because everybody I ever met of ETS cares about making the best possible test for its given purposes. My former organization was filled with passionate nerds who would subject a single test item to seven reviews and still be arguing about the placement of a comma in the prompt. Not being an education measurement professional perhaps made me much more sensitive to these disparagements of our work. My inherited combative nature also made me likely to argue with those foolish enough to send me the latest pronouncement that the SAT deserved to be banished from our kingdom of education.

The anguish of these critics is not incomprehensible to me whether from the amateurs spilling their Chardonnay on me while making one more point or the professional ones like Valerie Strauss in the Washington Post and Nick Lemann in the Atlantic, the New Yorker, and his very well researched book, The Big Test. (BTW ETS especially my friend Eleanor Horne helped Lemann with that research.) I get to a point the anger at a test that in the minds of these people proved extraordinarily consequential. To a point. Getting into college is hard. Or not. Or just for some specific colleges. All of those opinions are debatable, but an undebatable fact is that people endlessly like to write… stuff about the SAT as if that exam was the mother of all woes on a par with the melting Antarctic ice sheet, the atrophy of American democracy, asteroid collisions, and policing the restaurant deliverymen on Ebikes in Manhattan. Trust me that last one is very serious.

Only 2/3 of Americans do not have a four-year college degree and thus they have much bigger problems than the SAT as pointed out in this recent New York Times article. The other 33% did not all take the SAT or the ACT or any other admission test. So how is the SAT so important?

After all, Canada avoids this issue altogether. Malcolm Gladwell seventeen years ago – WAAAAYYY before all the ‘Varsity Blues’ scandals – wrote in the New Yorker about the differences between US and Canadian college admission. In the latter country, it’s much simpler. So why all the fuss down here? Gladwell noted that not just the media but our overall culture overplays the general advantages of a top tier college admission:  “For most students, though, the general rule seems to be that if you are a hardworking and intelligent person you’ll end up doing well regardless of where you went to school.” There is a difference but as Alan Krueger said in the Gladwell article: ““The average graduate from a top school is making nearly a hundred and twenty thousand dollars a year, the average graduate from a moderately selective school is making ninety thousand dollars. That’s an enormous difference, and I can see why parents would fight to get their kids into the better school. But I think they are just assigning to the school a lot of what the student is bringing with him to the school.’”

What exactly is the SAT measuring? According to this paper cowritten wth David Lubinski by my friend and former colleague Harrison Kell (who happened to be one of the most engaging and open people I met in my long service at ETS), the SAT is not testing the very important attribute of spatial reasoning. (Thomas Edsall had a longer piece that cited Harrison’s work here.) Of course, Harrison would be the first one to tell you that the SAT was not developed to measure “a capacity for mentally generating, rotating, and transforming visual images, (that) is one of the three specific cognitive abilities most important for developing expertise in learning and work settings.”

The College Board will tell you that it’s not measuring IQ. Even though the correlation is .8 between IQ scores and SAT scores, which if you recall our conversations on validity is a very high correlation coefficient.

And the College Board will also protest if you say that it’s a good measure of your capability to handle the job such as a Goldman Sachs or Amazon exec trainee, which is interesting because my friend Larry Prusak once told me that was exactly how Goldman Sachs measured the ability of somebody to be one of their masters of the universe trainees. And Jeff Bezos in the early days of his company asked everyone what their SAT scores were.

As previously noted, it’s not measuring conscientiousness or hope.

The SAT  does identify candidates who will succeed in their college studies as this University of California study demonstrated. The study was later ignored for political reasons, but that’s a whole other issue. As Zach Hambrick, Professor in the Department of Psychology at the University of Michigan, pointed out in a letter replying to Nick Lemann again denigrating the test in the New Yorker, “Decades of research … have confirmed that the SAT measures academic skills that are important for success in college classrooms, including text comprehension and mathematical reasoning; SAT scores predict future academic performance to a statistically and practically significant degree, even after taking into account differences in socioeconomic status.”

So, what about these writers? Is the SAT measuring something within them? In their case, I argue that the SAT measures very well is retrospective external attribution. Yes, I made that term up. External attributions “are explanations that stress environmental or situational factors.” The retrospective part of it is that they are still years and years later attributing the way their life went to a belief that the SAT screwed them over by not providing an accurate view of their merit.

Or as Freddie DeBoer puts it:

“You hate the SAT. I get it. I will repeat myself in saying that, frankly, I think this is mostly because you didn’t do as well at it as you thought, it didn’t confirm your place as one of the lonely geniuses of our times, and this is your revenge. Cool, cool. You’ve firmly established your own place on the ladder, so now you want to pull it up. Cool, cool. “

And this from him on getting rid of the SAT requirement…

“I guess for many people the SAT is such a visceral nexus of anxiety and insecurity, even decades after they took it, that they are simply incapable of critical thinking about this issue. And so they never pause to ask, “could these existentially elite institutions, which deepen inequality and further the interests of the moneyed and powerful at every turn, perhaps be getting rid of the ’SAT’ for reasons other than a pure and sincere commitment to racial and socioeconomic diversity?”

The SAT in my experience tests for a rather limited population of mostly white, mostly well-off, mostly well-educated individuals their level in a particular reservoir of resentment at what did or didn’t happen in the latter years of high school. I would assert that its validity is astonishing in determining exactly how pissed off someone is that their sense of themselves (as DeBoer notes above) was not confirmed and/or their choice of higher education institution was not granted.

What’s my evidence for such a bold statement?

1. The protests when they appear in articles and think pieces deliberately ignore enormous amounts of research that point to the validity and reliability — however limited that may be as it would be for any test — regarding its purposes
2. The tone is usually quite emotional, which suggests a wound inflicted by the SAT
3. They never tell you what their SAT score was.

I know, I know: that’s not very good evidence. My conclusion is just a hunch that started in 1968 when the Jesuits distributed to everyone in my class our scores from what was probably the PSAT. The report was on a very thin strip of colored paper, which created the scene in the cafeteria of all these suit jacketed or letter-sweatered young men hunched over under the neon lights trying to make sense of these primitive computer readouts. One classmate who regularly finished at the top of our class of about two hundred teenage males looked over my shoulder and exploded in anger when he recognized that my scores were higher than his totals. The unfairness of someone whose grades were at best middle of the pack, whose lack of industry had him twice taking a summer school course, who was so definitely not cool having higher scores provoked apoplexy in him. I understood only then that this was a competition. That had not been apparent to me until that moment in part because I cared about other things. I would rather have won the 220 yard dash in the Jersey City championships than get great scores on the SATs. My take was always that I was competing against myself on a test, but clearly for others there was a need to finish ahead of the other contestants. For them, the SAT was supposed to measure correctly and highly their worth, to confirm their superiority. And the SAT can’t do that. That’s something you do to yourself.

And to keep this a very personal history of testing, my SAT scores gaudy though they were failed spectacularly to predict my woeful performance once I got to college. Graduating in four years required me to take 44 credits in my senior year because I had with so many classes in the early days when I appeared to be majoring in drugs and rock ‘n’ roll. Sorry, no sex. It was a Catholic college in the 60s.

I have zero interest in defending the SAT. My concern here is the way in which the inordinate and distorted emphasis that these writers impose upon that test damage the image of ALL educational measurement. Personally, I wish the College Board would turn its extraordinary resources to formative assessment even though that’s not why they were founded and that’s not why colleges to a large extent still use them. But times change and individuals as well as organizations need to look at ourselves and change as well. Not because of these sometimes hysterical criticisms, but because you are a not-for-profit organization that is supposed to be contributing to the common good. The measure of the SAT in that context ultimately is whether it is doing something good for society NOW.

Bonus SAT section: read this for extra credit.

And yet the anti-SAT fusillade continues

Here’s one of the latest screeds from Nick Morrison: ‘It’s GPAs Not Standardized Tests That Predict College Success.”

Not only is the sample extremely limited here and the interpretations are way too broad from the study cited, but there’s no mention of other larger studies like this one by Rebecca Zwick that is considered extraordinarily revealing about GPA and SAT in determining college success.. In other words, he came in with a bias and only selected info that supported his existing position.

deBoer writes a lot about the SAT

Freddie has some wisdom to impart https://freddiedeboer.substack.com/p/you-arent-actually-mad-at-the-sats

Here is his litany of ‘my SAT stuff’:

• Your anger at the SATs is actually anger at broader social conditions which the SATs reveal
• None of the arguments against the SAT are correct, or even internally coherent
• Essays, too, are income stratified – because affluent kids actually are better prepared
• Grades (and all other educational data) are race and income stratified too, because those gaps in preparedness are real, so what the fuck are we even talking about here
• Educational testing is in fact remarkably valid, reliable, and predictive
• Why eventually ability reigns