When a bad test was good for… me

In yesterday’s post, I asked readers (of whom I am now delighted to say number in the hundreds and not the dozens) whether any of you had ever been involved with a test either as the person giving it or the person taking it where you thought that its score derived from test might not be valid. Still looking for those stories, but I promised at the end of that post to share a story of such a test as I experienced it. After all, this is supposed to be a personal history of testing. 

But the twist to this story is that even though I am persuaded that the test failed in terms of validity, its impact upon my life was nothing short of marvelous. Someone in talking to me about this blog and my series complemented the purpose by saying that there are many people out there who are not good test takers and the sharing of stories might get at some of the problems they experienced. That will definitely happen before this month is out, however, I must confess that I probably fit on the other end of that spectrum: I’m a good test taker. (My wife hates me for this quality BTW so there are unintended and undesirable consequences.)  For the most part, tests have been very very good to me. And none was better than the civil service examination for alcoholism counselor for Saratoga County New York that I took in 1975. 

My interest in that test stemmed from a desire to move to Saratoga Springs. My experience with alcoholism treatment of any sort extended only to working with some kids at the reform school or as they later called it juvenile residential detention facility where I had been an English teacher and counselor since graduating from College in 1973. My then mother-in-law (yes, a first marriage is a test that I definitely failed) shared my enthusiasm for us living in Saratoga Springs. She worked for the county government and kept her eye out for any jobs that might suit me. The alcoholism counselor position was one of the first and after I applied a letter arrived telling me the date of the examination in Ballston Spa New York. 

I prepared for the examination by taking out the only three books on the subject of alcoholism from the Kingsbridge Ave Bronx branch of the New York City Public Library. My recollection is that the books were generalist in tone and anecdotal in content. I devoured them, but did no other preparation for the exam. The questions on that civil service test did cover the aspects of alcoholism as a disease, the basics of counseling technique, and some general notions of professional behavior. I felt I had done well, but I usually had that feeling walking away from an exam. Months passed and then another letter came to me informing me that I had finished amongst the top three of all examinees and, therefore, under New York State civil service law was entitled to an interview for the position. Elated, I travel to Saratoga Springs and met with the head of the county mental health service and a local college chaplain who was advising them on the hire. I am a firm believer in the life advice given by Sister Mary Lucille Socciarelli: “Always tell the truth then you only have to remember one story.” thus, I am formed my interviewers that the reason they should hire me was because I was admitting my lack of knowledge and experience about alcoholism and thus would be a piece of clay that they could mold in any way they wished to achieve their goals for this position. Eventually, they stopped laughing, asked a few more questions, and I was on my way. I would have one other interview with the chief psychiatrist, which was not as easy because he was a brilliant clinician and knew how to grill a 24 year old wise guy like me. 

Reader, I got the job. I found out their decision in a public telephone booth in the strip mall on Riverdale Ave in the Bronx. That job changed my life. And yet… Was that test valid? Remember yesterday when I cited Michael Zieky said of validity that, “The test is doing the job it was designed to do.  The test is working as it should. The scores are valid for the intended purpose.”  

Remember the seminal Sam Messick definition? Validity is an overall evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions on the basis of test scores or other modes of assessment.” This test had none of that; e.g., how is an answer to a ninety-question exam an okay method of gathering evidence as to whether someone can diagnose and treat alcoholism? The idea of such an exam is to allow a claim that people who test above a certain score can do that job. Getting one of the top three scores after reading a trio of library books suggests any such claim would be quite shaky. 

Yet I worked as an alcoholism counselor and then as the head of the county drug agency in Saratoga County until 1980. I then was family alcoholism counselor at Saint Vincent’s hospital’s federally funded alcoholism treatment unit in Manhattan, Director of Education at Gracie Square Psychiatric Hospital’s substance abuse treatment … you get the idea. I had a robust career in alcoholism and substance abuse treatment before leaving that field in the 1990s. Nothing on that civil service test was relevant to being an alcoholism counselor. Feature this: the test was multiple choice. It didn’t even have an essay section that might ask me how I would deal with a particular type of patient. 

A few years later and perhaps in reaction to someone noticing how inadequate that civil service test was New York State decided to credential alcoholism counselors. Since I was already in that cohort, I got involved in the design and administration of that credentialing. The written test produced certainly had much greater validity, but what mattered the most was an exacting oral portion of the test where 2 veteran alcoholism counselors would pose case problems to a candidate for credentialing. In other words, we asked them to show some evidence of the construct – alcoholism counselling – upon which they were being tested. 

While that original civil service test worked out very well for me because of my ability to do well on such an exam, the person who was the alcoholism counselor at that time when they were committing it to a civil service post in Saratoga County did not finish among the top three. He was already doing the job and would continue to do alcoholism counseling jobs for the rest of his career; we later got to know each other and became friendly despite the fact that I had taken his job. I once dated a woman who told me that she had also taken the test and had finished in the top three.  While she was highly intelligent and a lovely person, she knew nothing about alcoholism whatsoever. (Yes, a woman I ‘dated’. You can connect the dots: I stayed in Saratoga with my ex-mother-in-law while my first wife moved on happily.) 

How does this fit into our overall narrative about testing? Poorly designed tests that lack validity have consequences not just for people but for communities. In my case, the verdict was that it all worked out. But that probably happened because I recognized my deficiency as did my superiors and they put me through a rigorous training after hiring me. But I encountered many other tests that produced scores and claims that lacked validity. Myers Briggs lacks validity as do most popular personality tests. And my story there gets to the man who debunked that test. But enough personal history for one post. Time for some of you readers to ante up your stories of tests, please.