Before he disgraced himself by dating and then marrying the adopted daughter of his previous partner, Woody Allen was an artistic hero of mine. I read all of his stories, listened to the recordings of his standup, and stood online to catch one of the first screenings whenever a new film of his opened. No more. Since that breach of morality, Woody became one more filmmaker and no longer an icon. It might help if he made movies as funny as his early ones too.
However, as an English major, I was schooled in the importance of avoiding the biographical fallacy; judge the work of the person not their life however tawdry and disappointing it may be. So, I do retain the piece of wisdom that I derived from him that seemed most pertinent to my own life that was something that he actually said to his collaborator Marshall Brickman: “Eighty percent of life is showing up.” His dictum had face validity; it appeared to fit what I had already discovered through my own accomplishments and misadventures. The 80% rule seemed to buttress belief in the efficacy of being conscientious.
Early on in my ETS career, I had the enormous good fortune to meet two of the preeminent researchers in the world on attributes that were deemed noncognitive: Rich Roberts and Patrick Kyllonen. One of their areas of research — conscientiousness — raised again the Woody Allen line because conscientiousness is not only essentially about showing up but as these guys were discovering this personality attribute possesses a criticality in life success. As I mentioned at the end of yesterday’s post, conscientiousness and other such factors might explain the success of some individuals who do not have high test scores on one of our major standardized formats like SAT, GRE, LSAT. Kim, Poropat, and MacCann (in a book edited in part by Rich) offer as a “broad general definition, conscientiousness describes individual differences in people’s approach to tasks and task completion. Highly conscientious people are well-organized, self-directed, and goal driven, whereas people low in conscientiousness can be spontaneous, careless, and disorganized.” The illustration below purports to show a hierarchy of conscientiousness.
As the above authors noted in their chapter back in 2016, “four facets of conscientiousness appear consistently across multiple investigations: orderliness, industriousness, responsibility, and self-control.” Some readers of this blog may have encountered the concept of grit that “refers to one’s perseverance and passion for long-term goals” that was popularized by Angela Duckworth. (Her Ted talk on grit is one of the most popular.) She argued that grit was a separate construct from conscientiousness but its pertinence in the national conversation about education dipped after subsequent studies found that grit did not significantly predict any of several educational outcomes beyond the effect of personality. (Remember that this is science and the work on figuring out the significance of grit — if any — in achievement continues as noted in this brief article.) Conscientiousness on the other hand proved predictive in a very specific way that is part of my personal history of testing.
How personal? The only academic award I ever gained in my mottled academic career was for perfect attendance at my eighth grade graduation in 1965 from St. Francis of Assisi Grammar School in Ridgefield Park New Jersey. Wasn’t the perfect attendance award a score indicating conscientiousness in a way? Every day you had a chance to show up and I showed up on all the days. Forty years later when I learned from my friends Rich Roberts and Patrick Kyllonen the extraordinary importance of conscientiousness, its importance to helping people achieve became even clearer. That’s why I bucked all of the guardians of ETS’s image and traditions – Legal, Finance, R&D solons, Marketing did I mention Legal — to help another friend, Steve Robbins, arrange a study in which he demonstrated a positive correlation between conscientiousness and performance reviews among customer service representatives at ETS. I wish Steve had included one more question on the survey: have any of you ever won a perfect attendance award from your grammar school?
As early as 2005, ETS R&D proposed a ‘New Constructs initiative that included conscientiousness. Why does this matter? For two reasons:
- this trait plays a central role in academic success (that is) reliably established and widely recognized
- it’s not part of the standard battery of assessments associated with the kind of claims that change a person’s life; e.g., college admission, job selection and promotion, etc.
If we desire tests that help people to be able to make claims about all of the positive attributes they bring to the situation, then we should not just assess cognitive factors. I’m going to keep on bringing up that 13% of students who are in the lowest quartile the jury scores 4.0 GPAs. Was there another cohort rejected because of their GPA that could have done the same thing due to these additional factors like conscientiousness? Was their own exclusion due to the fact that there was no test score they could show indicating this additional and somewhat elusive capability?
That 2005 presentation suggested three products associated with these noncognitive or psychosocial factors; one for K-12, one for higher education admission, and one for community college. They didn’t happen. And the next generation didn’t happen when it was proposed. If you go to the ETS.org website today, you won’t find any assessment of conscientiousness or any of the other so-called five factors of personality that have a significant influence on individuals’ ability to succeed such as extraversion, agreeableness, and openness.
This is not a criticism of ETS; they just found it too difficult to encompass these other factors in an organization that for the better part of century focused on reading, writing, and arithmetic. They did invest in a significant amount of research in this area, but the tests didn’t emerge. I got involved over and over again because the previous CEO thought that a Chief Learning Officer should be helping to spur innovation.
That initial presentation remains revealing. The Value Proposition of the three proposed new tests would be to “improve academic performance via non-cognitive factors (research based) through increased and enhanced Engagement, Learning Skills, and School Climate. In those days, we were also still talking about these tests as being supportive of No Child Left behind (remember that?) and Title I requirements. Part of the thinking was that having such assessments would work to the benefit of all parties: the teacher would know more, the parent would have a different basis upon which to hold discussions and help their kids, and the kids would learn more about themselves as well.
BTW these attributes are not fixed. In my last years at ETS, I worked exclusively over in R&D in one of the most engaging tasks assigned to me was to comment on the paper by Michele Martin-Raugh, Kevin Williams, and Jennifer Lentini on ‘The Malleability Of Workplace Relevant Noncognitive Constructs’. In plainer terms, Michele, Kevin, and Jennifer who were all colleagues at ETS at the time conducted an enormous analysis of whether attributes and capacities like teamwork, leadership and intrapersonal skills such as personality and motivation could change. Their interest was “How can noncognitive constructs be effectively measured, and what are the mechanisms that effectively drive change?” Their answer was yes they can be measured and yes there are certain things that help them to change. However, part of the measurement was to have really good assessments.
My disappointment that ETS didn’t get into that business went beyond the fact that I was personally invested in it through my association with Rich and Pat and then later on with Steve Robbins. In fact, Steve and I along with Mohammed Kousha traveled to our then subsidiary, Prometric, in Baltimore and sold their CEO on the idea of offering a new and innovative workplace assessment. Of course, that CEO said he would do it if he could sell the assessment for five dollars a test and ETS to my knowledge hasn’t been in the five dollar test business for a long time if ever. So, that didn’t happen.
My disappointment is that ETS is a big name. If that kind of an organization was promoting these sorts of assessments, then other people might pay more attention to them and those who are disadvantaged by assessments that are purely cognitive may get more of a chance to show who they are and what they can do. It’s not that these kinds of assessments don’t exist. There are lots of other noncognitive or psychosocial factors that researchers continue to study to determine what effect they might have on achievement and overall success in life. For example, one paper that Rich Roberts cowrote looked at whether Extraversion in association with pleasure orientation leads to greater problem-solving ability. It does!!! There are dozens of assessments looking at the categories of emotional intelligence, empathy, self-esteem, motivation, self-directed learning and other areas. And I’m not talking up phony baloney ones like Myers-Briggs, which will have more to say about in a subsequent post. These assessments have solid psychometrics behind them.
So, why don’t we ask for more of the resources that are spent on summative tests to be applied to these kinds of assessments? Why don’t we complain? We know their importance. Several years ago, the economist and intellectual, Tyler Cowen in his book Average is Over concluded that one of the things that will continue to matter most in our world is conscientiousness. He calls out this “critical quality in laborers” that will become even more important in the new economy. But if you don’t have a way of telling people where they stand in that regard then it is difficult for them to use it as part of their profile or figure out how to improve it. Being in favor of testing doesn’t mean accepting the status quo. We need more formative tests of factors like conscientiousness and hope. The latter one will be the subject of tomorrow’s post.
At least, I hope so.
Ah, conscientiousness. I have often thought that the quality of I value most in others (as well as in myself) is the certainty that they will do what they say they are going to do. This is pretty hard to measure in education, I think, but easy in the workplace – but they are already hired by then, of course… And a lot of people don’t do what they said they were going to do, which is infuriating, inefficient, and sometimes costly. So it would be great if we could develop a valid and reliable (and fair etc) test for it, please!