Just when I thought I was out…

Testing, I can’t quit you

As the Tom Hanks character in You’ve Got Mail suggests the Godfather movies provide a plethora of metaphors for the dynamics of our careers. In my case, I welcomed the absence of actual physical ‘hits’ even though corporate board room often did resemble a meeting of the Five Families.

The famous scene in Godfather III where Michael Corleone (Al Pacino) complains that “Just when I thought I was out…. they pull me back in” has resonated with me as the calendar pulled me further and further from my days as Chief Learning Officer at ETS. The sentiment proved resilient even as I busied myself through Knowledge Workings Theater with writing and producing plays.

And this magnetism isn’t a bad thing if my continued reflex to read everything within my attention span that has to do with testing, assessment, meritocracy, etc might benefit someone else. (The latter obsession of meritocracy stems from what I believe is an erroneous connection between testing and meritocracy or an even greater public misconception that there IS a meritocracy, but that hang-up will be the subject of a later series of rants.. er… posts.) But what is it to which I am being pulled back in? For those who were not my colleagues, a brief explanation seems apt. Unfortunately, my explanations tend not to be brief. Sorry.

The testing life accidentally on purpose 

From 2002 to 2017, I was the Chief Learning Officer (CLO) at the Educational Testing Service more commonly known as ETS — and often tagged with colorful obscenities such as “*$@#**%* ETS!!!” According to its own factsheet, “ETS develops, administers and scores more than 50 million assessment tests annually in more than 180 countries, at more than 9,000 locations worldwide.” That indicates that during my CLO tenure ETS produced 750 million tests. Add my three years subsequent to being CLO as a part time Knowledge Broker (a self-contrived title) in that nonprofit organization’s R&D division after I stepped down as an officer and the total of tests during my time pushes into the neighborhood of 1 billion. (I retired at the end of January 2020 to resume full time my earlier career as a playwright, an occupation my wife in the 1980s suggested suspending so that our children might enjoy food and shelter. She has always provided similarly thoughtful advice throughout our forty years together.) 

Not the little old testmaker

I didn’t make any of those billion tests. My work as CLO for which the people who hired me claimed I was qualified involved figuring out how to make the knowledge that people needed to do their jobs at ETS more effectively available; assembling tests is really about assembling different types of knowledge. CLOs also look out for such work as performance management, after action reviews, group facilitation, project management, and new product development. Learning a great deal about test construction and measurement along the way helped me when I had to handle from a human resources or disciplinary perspective some of the more harmful errors made, which were thankfully very rare. 

For several years, my portfolio even included responsibility for Facilities & Security for which I had no talent other than the good sense to listen to the people who did. A career at ETS that careened through so many different parts of our byzantine workings still never saw me making a test. I never wrote an item. I never scored an essay.  

Nonetheless, tests made me even before ETS. Prior to working there, all of the terms and issues related to testing even if then unknown to me guided my life. And I contend they guided YOUR life as well.

But after quitting that world those dimensions of testing — claims, validity, reliability, fairness, learning, summative or formative, criterion-referenced or norm-referenced, equating, error, evidence-centered design, and on and on and on –still hound me. They still surround my thinking. An adulthood of habits formed channels that continue to discharge information into my eyes, ears, and brain about SAT, ACT, GRE, NCLB, ELL, ELA, NAEP, PIAAC, PISA, and hundreds of other acronymized subjects.  Upon retirement, these subjects were supposed to hold no further interest for me; freedom from them was not only desired but expected.  

The fact that this has not transpired reminds me of a research finding on dreamers by  Michel Jouvet “At the beginning of their detention, prisoners’ dreams take place in locations associated with their previous life of freedom, whereas after leaving prison their dreams continue to take place in the cell.” In my last years as CLO at ETS, the organizational politics part did feel a little bit like a prison sentence . But retirement was to mean release. Yet like Jouvet’s prisoners, I am still dreaming, both the day and night kind, of tests because they form one map of my life thus far. Despite the freedom in writing and producing my new plays, I am still dreaming of tests. I am still distracted by every article that rails against tests especially the more gleeful ones and I am still lured to research posted by colleagues and acquaintances about how to improve educational tests. I’m stuck perhaps because like many people tests defined my life in many ways.

 

Writing this personal history might be a way of becoming unstuck. Psychoanalysis uses the technique of historical reconstruction, as Michael Roth wrote, “not as a representation of the past as it really was, but a reconfiguration of it so as to give meaning and direction to the present.” Perhaps a reconstruction of the tango danced by testing and me over my lifetime might prove useful here; a second look at the past might quell those dreams or at least make their persistence easier to understand. 

But this is not just about me. What if all of us could help to fill in the gaps that I perceive in the overall conversation about testing?  I’m not looking to incite further argument, but I welcome opposition opinions on testing that would inform. Maybe those who construct tests would gain inspiration to make them even better by visiting a place where people share their personal histories of testing

And tests could be better. Much better. Right now. I know enough about testing to make that assertion. Those responsible for educational measurement could innovate more to address the gaping demands of learners around the world.

A history, not a defense

My history is not a defense of testing. First, my qualifications and knowledge are unequal to that task. Secondly, not all testing merits defense; some of it is beyond awful and a terrible waste of resources. But more importantly, such apologia usually degrade to a kind of sophistry like that used by my revered Jesuits who writing at the time of the Spanish Civil War advised that “It is untrue that you hate Capital, friend worker. No, you want it yourself, as we all do… You do not hate Capital, but the Capitalist who does not fulfil his obligations towards you. To a certain extent you are right. But you cannot defend yourself with hatred. You will only answer injustice with injustice.”¹ Uh, those fellow workers did hate capital that capitalists used to oppress them. Trying to talk them out of their beliefs was not the answer. 

Despite being educated by Jesuits and called Jesuitical more than a few times, I don’t want to plead to others that “you do not hate testing, but you hate people who have not conducted testing in the way that they should have.” I don’t wish to tell anyone anymore how to feel about testing, but would instead consider it useful, an action of added value to my fellow citizens, if this history prodded the haters of testing and the lovers and the millions in the middle to explore their own feelings about something that so influences their life but is usually misunderstood, Testing remains a force that despite outbursts of outrage and conspiracies to cancel is unlikely to disappear. 

And that is why I am soliciting some friends to offer their own takes on testing to prime the pump for a greater flow of ideas and impressions. Jump in with a reply below; the water’s fine.