Cautionary tales abound about the danger of persnickety obsessions with seeing the one thing wrong with an other wise splendid entity. Nathaniel Hawthorne’s tale The Birth-Mark presented to the Sophomore English Honors class at St. Peter’s Prep in 1966 by Mr. John Casey told of the peril of obsession with trying to remove the one flaw of a beloved beauty. You just might kill it.
And then there’s the stern advice from Jesus about fault-finding:
Why do you take note of the splinter in your brother’s eye but do not notice the wooden plank in your own eye? 4 How can you say to your brother, ‘Let me remove that splinter from your eye,’ while all the time the wooden plank remains in your own? 5 You hypocrite! First remove the wooden plank from your own eye, and then you will be able to see clearly enough to remove the splinter from your brother’s eye.
Matthew 7:3-5, New Catholic Bible
Jesus makes a good point — as usual. BUT on reading this insightful and very interesting column by Monica Hesse that tells a very uncomfortable truth about SOME of the USA one defect did catch my eye, the one with the plank in it. And I feel a little bad now raising it because fittingly, the author who grew up in Normal Illinois, has a keen vision for spotting the deviations that now define part of our citizenry.
What was the flaw?
The phrase “teaching to standardized tests with shifting goalposts” repeats a common and unfortunate misunderstanding.. The belief allows for even more harmful judgments and should be confronted every time it appears. First, ‘teaching to the test doesn’t work so why would people still be talking about doing it? More perniciously, such myths, which are quintessentially 21st Century American in their vague anti-science cast, aid the overall rejection of educational measurement. If you want to say No Child Left Behind didn’t work or the Obama era follow-up initiatives were ineffective there’s an argument to be made for those sentiments although the reality of what happened is highly complicated. But the teaching to the test (Tx4) bugaboo was wrong from the start.
Did Tx4 happen in many places? Yes. Was it a stressor for teachers? Undoubtedly. Should administrators at each level of governance — federal, state, district, and building — have stoipeed the parctice because of its demonstarted ineffectiveness? YES. Did they? Mostly no.
How do we know that Tx4 didn’t work? That depends upon the circumstance.. In states that employed the cheapest version of tests, the scores might have risen but subsequent National Assessment of Educational Progress showed those results to be fraudulent. In some cases, the ‘teaching’ involved giving the questions ahead of time to students for practice and feedback, which is not so much teaching to the test as cheating to the test. In other cases, the technique simply didn’t work that should, able been np surprise because its already proven and well published inefficacy.
Teaching to the Test
In 1997, long before NCLB was instituted and Tx4 became a watchword for the perceived indignities of educational measurement, Lorrie Shepard gave the 3rd Angoff Lecture at ETS on ‘Measuring Achievement: What Does It Mean to Test for Robust Understanding?’ (Looking up the paper and seeing my friend and former colleague Shilpi Niyogi’s name in the acknowledgements as its editor was a bonus in this little excursion.) Lorrie demonstrated that there were similarities between what the textbooks contained and what the tests then included as questions. The significance of those findings was that the curriculum focused on a narrow set of information that didn’t necessarily advance more critical learning. Here is one of her examples:
“…pages from textbooks from the 1980s make the case that both first-time presentation of content as well as end-of-chapter tests closely resembled skills and formats from standardized tests. For example, because it appeared on standardized tests, second and third graders spent significant amounts of time learning to write out, from numerals to words, the narrative translation of 3,467 rather than doing problems to check on their understanding of what the numeral means (not what it “says”). Not surprisingly, some children who can write out three thousand, four hundred, sixty seven do not have the place understanding to be able to subtract 26 from 3,007.”
That kind of teaching to the test was NOT the teacher’s choice but developed at a remove in conversations between state boards of education and educational publishers. In fact, Lorrie’s paper documents the ‘scandalous’ finding that state reports overestimated progress by students.
This changed, and Lorrie’s call for more ‘performance tasks’ as opposed to multiple choice in these tests was answered by many educational researchers. Why performance tasks?
“(P)erformance tasks are intended to prevent the format effects of multiple-choice tests that have distorted instruction and in turn enabled students to
pretend to know. Open-ended tasks are more challenging and directly reflect desired understandings and applications. Therefore, it is less likely that students
could do these items correctly and still not understand the underlying concepts.”
My late friend and colleague, Bob Mislevy, stimulated significant changes in the shape of assessments with his conception (alongside Russell Almond, Linda Sternberg, Jan Lukas, Malcolm Bauer, David Williamson and others) of Evidence Centered Design of assessments. As Lorrie Shepard noted, tests and the preceding instruction need to be clear about what the intention of the learning is, the construct. That happened with ECD WHERE the unfortunately rare powers that be would approve and pay for it; happily . Bob was a pro-learning guy, not necessarily a pro-testing guy.
Do State Education Officials Read the Research of Their Field?
When NCLB starts in January 2002, the evidence already exists that teaching to the test only works if you are rigging the tests with the textbook ploy above or out and out cheating by distributing the tests beforehand as happened in Atlanta, Georgia. That same year, 24 years before the mention in the WaPo column that got me exercised over its Tx4 canard, Judith Langer on an NPR program — listen at around 22 minutes for the exact testimony — told of her experience in schools that teaching to the test only supplied very short term results.. 24 years ago! But you still hear about teaching to the test.
More damningly, this paper by Craig D. Jerald from twenty years ago pointed to research by Newmann, Bryk, and Nagaoka who “analyzed student test-score gains on the commercially developed, nationally norm-referenced Iowa Test of Basic Skills (ITBS) assessment and the state-developed Illinois Goal Assessment Program (IGAP) exams. The results were startling. In classrooms where teachers employed more authentic intellectual instruction, students logged test-score gains on the ITBS that exceeded the national average by 20 percent. However, students who were given few authentic assignments [i.e., taught to the test] gained much less than the national average. A similar pattern emerged when researchers examined results on the IGAP assessments.”
In other words, if you taught to the test in the way critics interpret that phrase, your kids did WORSE.
Given my role as Chief Learning Officer, accusations of bias regarding my views about educational measurement merit attention. I worked almost two decades for what was AT THE TIME the largest and most prestigious testing organization in the world. That must make me an apologist for the tests produced there, right? Where you stand depends upon where you sit is a worthwhile aphorism. But it does not describe me in this case because my own views of ‘No tests but for learning‘ were NOT shared by many of my colleagues especially those who were so-called ‘business facing’. After all, the current contracts for testing then produced important revenue and with a quarterly results focused leadership messing with revenue was anathema . Why? Another quote, this one from Upton Sinclair, explains their views — and many things that go on in corporations:
It is difficult to get a man to understand something, when his salary depends on his not understanding it.
It is true that even Sinclair’s wife found him to be “a man with every gift except humor and silence” but his quote above well describes in part the reality in which ETS never developed the systems that would have consigned the poor tests and the useless teaching to the test methods to the dumpster of educational practices. Maybe that’s why the teaching to the test misclaim irks me so much. While people were putting about such nonsense ETS and the nation were missing an opportunity to change the entire educational landscape through Cognitively Based Assessment of, by, and for Learning or CBAL Instead of futzing about the current tests, every stakeholder should have demanded the scrapping of the summative system, the one where certain grades devote a week to the old style of tests, to an adaptive formative system woven into actual instruction. I didn’t have anything to do with dreaming up this system, my friend and brilliant researcher Randy Elliot Bennett led that effort. But when Randy and others like Ida Lawrence first exposed the idea in a crowded hotel meeting room, I wanted to be part of getting this plan into the world. After all, that was part of my job, a ;part at which I mostly failed.
During my nineteen years at ETS, an effective strategy for taking the unique and consequential research on these matters effectively into the marketplace never materialized. In this 2017 paper here, I wrote about how the layers of conservative attitudes in an older ‘Operate and Maintain’ organization like ETS constituted a kind of gravity that made it difficult to put new products into orbit.
But my paper now reads as seriously flawed because a) it’s too optimistic about how ETS will escape gravity and b) my role in the failure to make that happen receives no notice. And that’s why a comment like the one Monica Hesse made about the supposed USA reality of ‘teaching to standardized tests with shifting goalposts‘ galled me so much that for the day I abandoned work on another project to compose this rant. I very much agree with Hesse on the other things she wrote of in her column about our country where “The American Dream comes down to OnlyFans, dodgeball and reality TV .” The mess she illustrates is connected to all of this:
“A teacher is paid $37,000, a scholarship is $125,000, a college tuition is a gazillion dollars, the cost of living is too much. Wherever you live, it’s too much, but this is America so you must find a way to monkey-bar your way to solvency and/or pull yourself up by your own boobstrap.“
The paragraph gets at the issues of inequality and opportunity that have gotten worse in the last fifty years. As Paul Krugman noted recently: “Between 1979 and 2000, the share of total income going to labor as opposed to capital was basically unchanged. But while the typical worker’s income, adjusted for inflation, barely increased over that period, the average real labor income of the highest earning 1 percent, such as top executives, nearly tripled..”.
At the end of her thought-provoking and soul-disturbing essay with stories of teachers turning to Only fans and dodgeball teams being decided by political affiliation, Hesse asks and answers a question that many of us have pondered this summer of the 250th anniversary of the Republic: “Where has life, liberty and the pursuit of happiness gotten us in 2026? Heaven only knows, but on television it’s gotten us to some very weird places.” It’s not just on television. The inequality Krugman documents above has gotten us to a place where it is more difficult than ever to get a man (and/or a woman but definitely more a man) to understand something like this impending societal disaster, when his/her salary plus bonus, stock options, network membership, fear of exclusion form the oligarchy, etc. depend on not understanding it. It’s gotten us to a place where the very health of our country is endangered.
What does educational testing have to do with all of this? Education could make a difference in reversing this decline. (Even staunch Republicans like former Senator Ben Sasse calls for a return to teaching coivics.) Not the kind of difference that will change everything. That’s a ridiculous and naive supposition, but testing for learning could make a difference in this situation. How? The comprehending of the way in which the world really works makes us all more sensitive to the way in which we are being worked over.
I understand those that doubt any statement justifying assessments even on like CBAL. Testing is vilified and in some cases that judgment is deserved especially in States that have perverted the standards and gone cheap on methods and assessment. If you take a look at the states that have done that, you’ll also see a connection to either inaction or actions against their own interests among citizens of all generations there. Is that a harsh correlation? Yes. If you think that making ends meet by putting your wife on Only Fans is a good idea (read the Hesse column to understand what this means) then what I’m suggesting here won’t make any sense to you.
There is another way. If testing is connected to learning as it should be, and the learning is focused on those constructs that most matter to having a productive life in a functioning Republic, then creating an overall methodology where the assessment is of, by, and for learning makes a difference. Learning will be more effective in that it will be more actionable, Able to be acted upon or put into practice.
Is that kind of learning going to bring down Only Fans or stop what Hesse describes as even the politicization of dodge-ball? (Have you no shame, sir?) Certainly not. But it could make a difference in tilting more of the citizenry to asking questions as to what are they getting out of all of this? How is the current circumstance serving them? What will the continued decline of the Republic happening in concert with the increase of inequality affect their children or anyone else they care about in the Next Generation? Those goal posts are not shifting. And the test is still ongoing from all of us although I have the sense that time is running out.
Oh, and here’s what CBAL looks like for writing