In yesterday’s post, the issue of increasing disregard of the authority that is necessary to create a meaningful test occupied my daily rant. But even then the need to consider the other side of authority — whether those with the decision-making rights about educational measurement seek sufficient counsel from the people who actually take the tests — crept into the discussion. One of the most frustrating experiences at ETS was the constant battle over expertise. There were people in marketing who thought they knew about testing and didn’t. And there were people in assessment development who thought they knew about marketing and didn’t actually, those people were more likely to tell you they didn’t care about marketing but they should’ve known that was a mistake as the environment for testing changed remarkably. The battles were usually fought to a draw which means that the largest casualty was innovation. It’s hard to make things different if you spend all your time trying to prove that you’re right, that others should heed your authority without question.
Yesterday, our post asked the question of whether problems with testing were really problems with authority. The premise was that in our society citizens regularly debase authority. In some instances, a defense of their degradation and diminution of authority is possible because the authority in question has screwed up royally or just failed to ever meet their promises that their claim to authority is eroded significantly. In one sense the fear of testing is the rejection of authority and having pointed out so many examples of invalid, unreliable, unfair tests in the course of the last 27 days, who am I to defend broadly that authority? Instead, I’d like to take a look at the consequences of the battle inside testing organization when uncertainty about authority led to inertia, skirmishes as to what to do next resulted in doing nothing that made a difference.
A recent description of this kind of battle is ‘epistemic trespassing’.
Epistemic trespassing, or epistemic squatting?
Who gets to define the boundaries of expertise?
Noah Smith wrote, “The other day I saw someone use the phrase “epistemic trespassers” in a Twitter rant. I traced it back to this tweet, which links to an essay (paper?) by the Fordham University philosopher Nathan Ballantyne”
Then Smith picks a bone with Ballantyne:
“Contrary to what Ballantyne says, it’s not always clear who’s an expert and who’s a trespasser. In practice, expertise is defined by consensus among communities of people who all or mostly agree that each other are experts. These communities can be formal, like the American Economic Association, or informal, like “DSGE macroeconomists”. But the basic idea here is that these communities are self-judged — they deny outsiders the right to adjudicate whether they possess actual expertise.
This strategy often produces good results, as demonstrated by the remarkable practical success of a bunch of scientific fields. But it’s not foolproof. Thanks to the quirks of human sociology, it’s possible for exclusive communities of self-described experts to arise who don’t actually have the expertise they claim — and even for these communities to be recognizes as experts by the broader public, at least for a time.’“
In order to make the kind of assessments that will be formative in nature and to figure out how to get organizations and individuals to accept them, the testing industry needs to invite more non-experts, more users of their end products, into the conversation, not just for show but as actual partners.
If there are going to be changes in the way the testing is done in this country in order to shift resources toward development rather than computation of individuals then we have to acknowledge that those who currently hold decision-making rights about what is tested and how that measurement occurs may not be the right people for the next era of educational measurement. And since most of those decision-makers are being either not proper corporations that are supposed to be serving a public good or governmental positions that likewise are supposed to be serving a public good, all of us bear some responsibility for not even asking the question as to whether these are the right people doing the right things.
Why do I have the suspicion that those in authority may be missing a bet about what to do next? The concentration of power over testing within a small group often leads to a silo effect where the experts only talk to other experts with the same qualifications in the same background. It’s what EriK Dane referred to as cognitive entrenchment. It’s also what the late biologist EO Wilson wanted to see go away in his promotion of consilience.
What is cognitive entrenchment? Erik Dane defined it this way: “Research suggests that as one acquires domain expertise, one loses flexibility with regard to problem solving, adaptation, and creative idea generation. Here, I reconsider this trade-off between expertise and flexibility by examining the concept of cognitive entrenchment—a high level of stability in one’s domain schemas. Proposing that cognitive entrenchment varies not only with expertise but also with one’s task environment and attentional focus, I contend that the inflexibility-related limitations of expertise can be circumvented.”
If you look back at the origins of standardized testing this country, there were committees that included teachers, professors, and policymakers involved in the making of every test. The ETS library (formerly somewhat infamously called the Brigham library) had many artifacts of the history of testing and one of them was a video of an early documentary about the making of an advanced placement test that showed a dozen white men of roughly the same age in roughly the same white shirts and ties chainsmoking roughly the same cigarettes. No test takers were to be seen in that gathering and for the most part the users of tests still don’t get invited to such confab’s.
In fact, one of the CEOs at ETS used to tell the story of how in his first tour of duty at the company as a junior executive he received a memo along with everyone else that forbade the use of the term marketing. For those of you who are anti-capitalist or anticonsumerist, I get that marketing might have a bad connotation for you. But at the heart of marketing is an understanding of what people need. It’s not about just getting them to buy endlessly especially not in the case of something like a test. When I came to ETS, the concept of marketing was no longer verboten, but there was still a big argument about who the company served. And in this sense, I don’t think the organization was different than any other testing organization: it served some big organization that was willing to pay for the design and administration of the test even if that money one way or another came from people who were willing to pay to take the test.
The argument from the R&D side was that there was no test without a warrant, the rationale or generalization for grounding the claim the individual available data. A warrant establishes “the credibility, relevance and strength of the evidence in relation to the target conclusions.” The issuing organization whether it is the Department of Motor Vehicles or the College Board or the Vietnamese Air Traffic Control Authority was the owner of the warrant and, therefore, ETS’s duty was to that entity. Individual test-takers were ignored. Lots of time was spent answering their questions and making it possible for them to take the test even when that required some sort of accommodation. But there needs were not essential. And those other entities were not interested so much in developing the individual as they were in being able to make a claim about what the individual knew now.
The specialists in all of those places were entrenched in what I referred to in a book chapter about trying to make innovation happen at ETS as ‘operate and maintain’. They were afflicted with ‘status quo bias’. The change expert Bas Verplanken introduced me to this term in a lecture (arranged through my good friends at LILA Marga Biller and Daniel Wilson) where he stated a deceptively simple truth, “people tend to do what they tend to do.” If you look at the history of testing since 1947, much of it has been cloning existing concepts. Education tends to do what it has tended to do.
What does consilience have to do with all this? It’s the opposite of that entrenchment as Wilson championed the concept. The OED calls it “The fact of ‘jumping together’ or agreeing; coincidence, concurrence; said of the accordance of two or more inductions drawn from different groups of phenomena.” Wilson called it “the proof that everything in our world is organized in terms of a small number of fundamental natural laws that comprise the principles underlying every branch of learning” That meant we should at least be seeking to learn from other branches, to be less immersed in our own specialty. To utilize “an outside pair of eyes (which) can improve a group’s ability to find novel solutions to a problem”. As business professors Marion Poetz, Nikolaus Franke and Martin Schreier write”:
“Over the course of years of studying innovation, we’ve found that there’s great power in bringing together people who work in fields that are different from one another yet that are analogous on a deep structural level. Such as makeup and surgical infections, surprisingly. Or inventory management and robot games. Or malls and mines.
Bringing in ideas from analogous fields turns out to be a potential source of radical innovation. When you’re working on a problem and you pool insights from analogous areas, you’re likely to get significantly greater novelty in the proposed solutions, for two reasons: People versed in analogous fields can draw on different pools of knowledge, and they’re not mentally constrained by existing, “known” solutions to the problem in the target field. The greater the distance between the problem and the analogous field, the greater the novelty of the solutions…
The authority of inertia or Bas again
Take K-12 for example, how much of the authority for doing things is held by the faculty? Do they get in part to set the paradigm if you will, the rules of the game? This can happen quietly even invisibly. A superintendent of large school district near ETS once gave a fascinating but disheartening presentation on the workings of his budget. Like water running down the hill, the decisions on where to spend money and, therefore, time and other resources follow the path of least resistance and end up doing the same things over and over again.
Testing needs to take the advice of Vijay Govindarajan and forget the past so that it can invent the future. Since they are unlikely to give up the authority on their own, of informed citizens will have to rest the decision-making rights from them by insisting that on the K-12 level no test be administered that is not either a cognitive-based assessment of, by, and for learning a developmentally targeted assessment of other attributes such as conscientiousness and hope. In our next post, will look at it even more concrete plan that people could adopt to make this happen.
I should have read this post before commenting on the previous one! I love the idea of an ‘expert’ versus a ‘trespasser’ (a new term to me but one that I will eagerly adopt). I agree that assessment is beset with trespassers, from politicians to purse-string-holders to journalists and beyond. Similarly, the issue of entrenchment or the silo effect has dogged my entire professional life. ‘We do what we do because this is the way we do it/have always done it’ etc. Any sort of change or innovation is usually treated with suspicion and often downright hostility, but I am definitely in the consilience camp: “there’s great power in bringing together people who work in fields that are different from one another yet that are analogous on a deep structural level”. I see that first-hand with mixing delegates from different assessment contexts – primary schools, secondary schools, universities, the workplace (hospitality to engineering to human resources etc). That’s what so great about the January Jolts – bringing together ideas from across the domain (and beyond) that can help make new and exciting connections and create fresh ideas.