The ‘people should judge people, not tests’ take ignores our biases

Graphic by @bigthink at this link


People. Let’s check out their ability to render valid and reliable judgments.

Those who suggest that tests need to go away usually suggest that people can do the judging — teachers, HR generalists or recruiting specialists, admissions counselors. It is rare in my experience to have either a suggestion of the improvements or replacements for tests that tell us how much a person or group knows about some construct that we say we consider critical to success in our world such as reading, writing, calculation, basic scientific understanding. It is also rare for folks to acknowledge that while the test might go away the making of consequential claims isn’t going anywhere in our world..

Matt Yglesias had an interesting take via Twitter (in a post that no longer exists) on the push against standardized testing:

“This is one reason I’m very suspicious of the push against standardized testing on social justice grounds … alternate methods of screening people seem more biased not less.”

He was referring to a journal article on school counselor bias

But let’s leave the biases of teachers, school counselors, and admissions counselors alone for a day. and Since this is personal history of testing, consider every job interview you ever had. Most of those took place without any supplemental test, I’m guessing. But because of the work of people like Amos Tversky, Danny Kahneman, Richard Thaler and others we know that it is highly likely that whoever interviewed you had scant training in interviewing and did so in a way that included at least some bias.

  • What kind of biases? Here’s a list. If at the end of reading it, you still think that we would like to have people do all the evaluations without the help of any standardized test then that’s OK. In my own personal history, I encountered several of these biases. Some of them were when I was the person being interviewed and felt the confirmation bias or focusing effect coming across the table at me. But most of it came when I was in response of what we called the people processes or human resources at ETS or even before that when I was a consultant on those matters at hundreds of corporations. In those cases, loss aversion, neglect of probability, hyperbolic discounting, and selective perception cropped up all the time. BTW, the biases were on the other side of the table in inetreviewees as well: e.g., Egocentric bias – when people claim more responsibility for themselves for the results of a joint action than an outside observer would and the Forer effect (aka Barnum Effect) – the tendency to give high accuracy ratings to descriptions of their personality that supposedly are tailored specifically for them, but are in fact vague and general enough to apply to a wide range of people. For example, horoscopes.
  • Of course, Self-serving bias, the tendency to claim more responsibility for successes than failures, permeated every work organization that I ever encountered. So did Halo effect – the tendency for a person’s positive or negative traits to “spill over” from one area of their personality to another in others’ perceptions of them. And who can ignore the  Lake Wobegon effect – the human tendency to report flattering beliefs about oneself and believe that one is above average (see also worse-than-average effect, andDunning Kruger; Dunning Kruger is to business executives as hot dogs are to Coney Island,Cheese steaks  to Philadelphia , and  ‘Chowda’ to Boston: Those things just go together. But enjoy taking a look at this list of cognitive biases and consider whether we question the abilities of people to make claims about us as much as we question the features of standardized testing. See you tomorrow for mailbox Monday number 2 and enjoy the rest of the weekend.

 Those other biases borrowed from Wikipedia mostly

  • Hindsight bias sometimes called the “I-knew-it-all-along” effect, is the inclination to see past events as being predictable
  • Fundamental attribution error the tendency for people to over-emphasize personality-based explanations for behaviors observed in others while under-emphasizing the role and power of situational influences on the same behavior.
  • Confirmation bias the tendency to search for or interpret information in a way that confirms one’s preconceptions.
  • Disconfirmation bias – the tendency for people to extend critical scrutiny to information which contradicts their prior beliefs and uncritically accept information that is congruent with their prior beliefs.
  • Selective perception – the tendency for expectations to affect perception.
  • Bandwagon effect – the tendency to do (or believe) things because many other people do (or believe) the same.
  • Bias blind spot – the tendency not to compensate for one’s own cognitive biases.
  • Choice-supportive bias – the tendency to remember one’s choices as better than they actually were.
  • Congruence bias – the tendency to test hypotheses exclusively through direct testing.
  • Contrast effect – the enhancement or diminishment of a weight or other measurement when compared with recently observed contrasting object.
  • Déformation professionnelle – the tendency to look at things according to the conventions of one’s own profession, forgetting any broader point of view.
  • Endowment effect – the tendency for people to value something more as soon as they own it.
  • Focusing effect – prediction bias occurring when people place too much importance on one aspect of an event; causes error in accurately predicting the utility of a future outcome.
  • Hyperbolic discounting – the tendency for people to have a stronger preference for more immediate payoffs relative to later payoffs, the closer to the present both payoffs are.
  • Illusion of control – the tendency for human beings to believe they can control or at least influence outcomes which they clearly cannot.
  • Impact bias – the tendency for people to overestimate the length or the intensity of the impact of future feeling states.
  • Information bias – the tendency to seek information even when it cannot affect action.
  • Loss aversion – the tendency for people to strongly prefer avoiding losses over acquiring gains (see also sunk cost effects)
  • Neglect of probability – the tendency to completely disregard probability when making a decision under uncertainty.
  • Mere exposure effect – the tendency for people to express undue liking for things merely because they are familiar with them.
  • Omission bias – The tendency to judge harmful actions as worse, or less moral, than equally harmful omissions (inactions).
  • Outcome bias – the tendency to judge a decision by its eventual outcome instead of based on the quality of the decision at the time it was made.
  • Planning fallacy – the tendency to underestimate task-completion times.
  • Post-purchase rationalization – the tendency to persuade oneself through rational argument that a purchase was a good value.
  • Pseudocertainty effect – the tendency to make risk-averse choices if the expected outcome is positive, but make risk-seeking choices to avoid negative outcomes.
  • Selective perception – the tendency for expectations to affect perception.
  • Status quo bias – the tendency for people to like things to stay relatively the same.
  • Von Restorff effect – the tendency for an item that “stands out like a sore thumb” to be more likely to be remembered than other items.
  • Zero-risk bias – preference for reducing a small risk to zero over a greater reduction in a larger risk.
  • Social Biases
  • Most of these biases are labeled as attributional biases.
  • Actor-observer bias – the tendency for explanations for other individual’s behaviors to overemphasize the influence of their personality and underemphasize the influence of their situation. This is coupled with the opposite tendency for the self in that one’s explanations for their own behaviors overemphasize my situation and underemphasize the influence of my personality. (see also fundamental attribution error).
  • False consensus effect – the tendency for people to overestimate the degree to which others agree with them.
  • Fundamental attribution error – the tendency for people to over-emphasize personality-based explanations for behaviors observed in others while under-emphasizing the role and power of situational influences on the same behavior (see also actor-observer bias, group attribution error, positivity effect, and negativity effect).
  • Illusion of asymmetric insight – people perceive their knowledge of their peers to surpass their peers’ knowledge of them.
  • Illusion of transparency – people overestimate others’ ability to know them, and they also overestimate their ability to know others.
  • In-group bias – preferential treatment people give to whom they perceive to be members of their own groups.
  • Just-world phenomenon – the tendency for people to believe that the world is “just” and therefore people “get what they deserve.”
  • and overconfidence effect).
  • Notational bias – a form of cultural bias in which a notation induces the appearance of a nonexistent natural law.
  • Outgroup homogeneity bias – individuals see members of their own group as being relatively more varied than members of other groups.
  • Projection bias – the tendency to unconsciously assume that others share the same or similar thoughts, beliefs, values, or positions.
  • Self-fulfilling prophecy – the tendency to engage in behaviors that elicit results which will (consciously or subconsciously) confirm our beliefs.
  • Trait ascription bias – the tendency for people to view themselves as relatively variable in terms of personality, behavior and mood while viewing others as much more predictable.

You made it to the end!!!

from Wikimedia Commons

1 thought on “The ‘people should judge people, not tests’ take ignores our biases

  1. Marianne Talbot

    Bias is great, isn’t it? I could go on for hours, days, weeks. There is just so much of it. It is everywhere. But we mostly tend to ignore it. I particularly enjoy/suffer from the planning fallacy (myself), and the status quo bias (other people, never myself, of course).

    Just to be provocative, take this quote from a Cambridge Assessment article by John Rust – my emphasis added (from 2007, available here: – sorry it looks like I’m shouting):

    “But latterly there has been an increased recognition that, apart from item level bias, MOST OF THE BIAS FOUND IN ASSESSMENT IS EXTRINSIC TO THE TEST OR EXAMINATION ITSELF. More often, differences in test scores between groups come about as a result of the impact of real differences in society. Bias in, and the consequent adverse impact of, school examination results CAN TO A LARGE EXTENT BE ACCOUNTED FOR BY DIFFERENCES BETWEEN LOCALITIES IN THE QUALITY OF SCHOOLING, OR OF PARENTAL, PEER AND TEACHER EXPECTATION AND SUPPORT. These are themselves dependent on the impact of social policy on local demographics.”

    I am not sure this is true – any more? – it might have been what was BELIEVED at the time, but I’d like to think society has moved on a bit in the last 15 years – but education will likely take a while to catch up…

Comments are closed.