Dog’s Breakfast or What My Algorithm Hath Wrought

According to Merriam-Webster, a Dog’s Breakfast is defined as “chiefly British. : a confused mess or mixture.” I don’t know about the British part because we certainly heard the phrase in our exclusively Irish household where I knew that it meant something that was thrown together. (The first use of the term cited by OED is from Ireland but the two isles have more in common than either wants to admit) And I knew that my use the term today it would give me a fairly legitimate excuse to post a picture of our deeply beloved and dearly departed Australian Terrier, Mr. Bingley. But the dog’s breakfast part is the confused mess or mixture that results from grabbing every scrap of information that is thrown at me through all the various feeds and sites that I frequented during my time in the testing field.

I’ve been collecting links and documents and charts on the subject of testing ever since I started at ETS 20 years ago. Therefore, the dark Angels of the Internet have been collecting information on what I’ve been collecting so that they could throw more of the same at me as a kind of click bait when they were peppering me with advertisements to buy another wine cooler or to support some hapless candidate limping for political office.

What that means is that after all those years and now staring down the barrel of only eight days to go in our 30 when January Jolts I have…

Sooooooooo Many Links Related to Learning and Testing That I’m Just Not Going to Be Able to Weave into This Narrative over the Next Eight Days

For example, I’d love to talk about the weird reality that people believe in learning styles but don’t believe in testing. Instead, I’ll just point you to this excellent YouTube video The Biggest Myth in Education and proudly state that the debunking of learning styles happened at ETS. Not out of any meanness. You have to remember that the people working there in research and development are scientists. They were really trying to find out whether such a thing existed or not and they couldn’t unearth any evidence in their research that there is such a thing as learning styles. Subsequent studies over the years have come up with the same conclusion, which hasn’t prevented some otherwise smart people from peddling this pish and lots of other people from buying it.

I’d like to write more about the testing situation in Europe in part because it contrasts so greatly with our own mess of denial and misinformation. But here’s one from the country of which I am proud citizen, Ireland: Standardised tests provide an important snapshot of pupil achievement (via @IrishTimes) 

here’s a sample from that article:

“Hence, the more important question to ask is not should we test or not but how to do so fairly to ensure confidence in the results? Ironically (because this is often overlooked), the answer to this question is implied in their official name: “norm-referenced standardised achievement tests”.

…

The word “standardised” underscores the point that these tests are constructed and intended to be administered, and the results interpreted, in a standard manner.

As detailed in the manuals that accompany the test booklets, the expectation is that every teacher, irrespective of context, follows strictly a set of directions regarding test preparation, time allocation and supports offered during testing and so on, when administering the tests.

This is essential if fair comparison is to be made subsequently based on the results and why practices such as “teaching to the test” is so problematic; in effect, it can invalidate the test results.

The final point to note is that these are achievement tests – designed to measure what a pupil has learned in specific areas of the curriculum – English reading, mathematics and Irish reading (if attending a gaelscoil) – based on what is taught in school.

…

So, rather than castigating norm-referenced standardised achievement tests out of hand, it would be prudent to consider them in context and rather than inflate pupils’ stress and anxiety unnecessarily remind pupils, and each other, that these tests provide a useful and important snapshot of their achievement, nothing more or less.”

There’s some really interesting stuff there, but it’s a cute little Irish bridge too far for this venture. Similarly, the impossibility of taking on a discussion of testing in China even though that is an area where I had some involvement especially around English language testing. If you think our situation is tough in terms of tests inducing anxiety, you have another think coming.

The weird crackdown on tutoring by the Chinese government puts parents in a terrible bind there. Parents are NOT complaining about the test. They are complaining about not being able to access the tutors. Here is the article:

China Targets Costly Tutoring Classes. Parents Want to Save Them. https://nyti.ms/3lcFMEo

“ For Chinese students hoping to get a spot at a prestigious university, everything hinges on the gaokao, a single exam that many children are primed for before they even learn how to write.”

“If this criteria for selecting students doesn’t change, it’s hard to change specific practices,” said Ms. Tu, whose research is focused on wealth and education in China. Parents often describe being pressured into finding tutors who will teach their children next year’s curriculum well before the semester begins, she said.

But in a final and fulfilling flourish, I am canceling, obstructing, and obliterating in any way I can the parts of my digital trail like cookies, history, and ‘likes’ that might bring me more links on the SAT.

Four years, the great river that is the Internet has washed up on my sure this kind of flotsam and jetsam

A defense of standardized tests

And

The Persistent Grip of Social Class on College Admissions https://nyti.ms/3fItpfa

“another sign of the persistent pull of social class, a recent working paper from authors affiliated with the Student Narrative Lab at Stanford shows that essay content, when quantified through a computer program, is more highly correlated with household income than SAT scores are.”

But those pieces with actual references to the science are the exceptions when it comes to the SAT/ACT because the incoming tides are much more likely to carry effluvia like the sort of misinformation like written by Patrice Apodaca. Pieces like this are why I despair of ever having an evidence based conversation about testing. (BTW does anyone out there think that editors keep on publishing anti-SAT things because it’s the sort of stuff that readers like, which puts it in the same category as eight ways to trick yourself into exercising in the New York Times and 173 ways to thrive in 2022 in the New York times and the Miss Manners column in the Washington Post taking up the critical question of whether it’s okay to just a knowledge of dog and not its owner when you’re out on a walk.

One of the greatest psychometricians in the world whom I won’t embarrass by naming him here sent me a very kind note after my SAT posts and asked why it was that retired guys like us were the ones writing these defenses instead of the people who are still creating the tests? I have no idea, and I hope that this next defense prompted by Patrice Apodaca’s column will eliminate the itch to protest ill-informed and actually harmful tirades like this piece once and for all.

Part of what gets me is faux Apodictic nature of these harangues.

Yes, Apodictic is the sort of word that might show up on the SAT and there’s nothing wrong with having an enriched vocabulary that allows you to precisely describe the sort of person who presents their views as “Clearly established or beyond dispute” when they are anything but as I show now show in this Step by misstep account of the points made in the above,.

“I’ve been rooting for the downfall of the SAT for a long time.”

What? I recognize that this is completely consistent with the dominant tone of discourse on platforms like Twitter where everybody acts like they are one of the Sharks or Jets and they want to prove their toughness with an opening provocative statement. But how are we supposed to take seriously someone who begins an argument with this admission of a bias? How can any of us avoid confirmation bias if our predisposition is to root against something and, therefore, because of that advocacy only accept information that supports what they already believe? To this septuagenarian curmudgeon, such an attitude (which is now pretty pretty widespread) constitutes one of the most serious threats to our civilization. I’ve learned a great deal about the deficiencies of the SAT by actually paying attention to what the better critics say. Sorrowfully — and I mean that I cannot include Patrice Apodaca in that complement because of the way in which she continues:

“The SAT contributes next to nothing to deep and meaningful learning, but it’s part of a culture in which schools have felt pressed to tailor their curricula to improve standardized test scores.”

Let’s return to my point made in an earlier one of these January jolt blog posts that many kids are not going to take the SAT at all. Therefore, it’s doubtful that any school tailors their instruction or curriculum to the SAT. I can say with absolute certainty since ETS had California as a client in designing their in-state tests that our country’s most populous state certainly does not tailor their curricula to improve standardized test scores. Besides being wrong such a statement is extraordinarily insulting to people in Sacramento who worked very hard, very rigorously, and very independently to come up with what they believe will be the best curriculum to prepare their students for the real world. I had the privilege of sitting in on some discussions as a rank outsider and came away deeply impressed. They’re not insisting that their students do well on the math portion of the SAT. There insisting that their students learn math and science in such a way that they can succeed in the new world economy. There really serious about it and they insist that anybody working with them be just as serious.

I think the same is true in Texas where ETS did work, Virginia, etc. As to the SAT contributing next to nothing to deep and meaningful learning, it’s a test of whether somebody has learned certain constructs. Does the author mean that those constructs don’t matter? Reading? Vocabulary? Math?

Of course, like almost every other critic Apodaca trots out the notion that the SAT isn’t good at predicting who will do well in college. Just not true but it keeps getting repeated. See the details of how this statement is at best ignorant and at worst dishonest by looking again at What do the SATs measure? And Do the SATs measure success? From a writing perspective alone, would think that these critics would be embarrassed to keep on using the same material over and over again. At least, have the ingenuity and self-esteem to come up with something original.

Here’s another chestnut:

“SAT actually exacerbates inequality because it favors those who can afford expensive test prep”

 First, most estimates place that boost from test prep to be an actual 20 points. Of course, very clever strategy of the test prep organizations to give an extraordinarily hard test as the first exam to their clients which they will score poorly on and then given the regular test which they will score better on is why this illusion persists. The con of test prep deceives observers; the reality of research points out that there is inequality in outcomes but that’s an output of a classist capitalist society and not the test. I am all for figuring out ways to eliminate that inequality but scrapping the SAT or even blaming the SAT for it is not going to be helpful.

It should be acknowledged that not everyone is happy with the UC decision.

This is rich. That is some first-class dissembling. The truth is that the University of California system’s Faculty Senate originally voted to keep the SAT. Through political maneuvering, other powers reversed the original decision.

And…

And…

I give up. Nothing I write here is going to change the minds of those who are determined to hate testing. My audience is actually those who want to know what’s really going on, who are open to discovering for themselves the ambiguities and possibilities that testing can present as part of a different way of looking at learning and development. And at the end of this month be getting to that.

The good news is my exorcism of testing proceeds apace: that’s the last time I ever going to write anything about SAT deniers. If you know the movie, I’m well past point where Linda Blair’s head spins around.

Instead, I’d much rather show you another take on hope that we wrote about last week, which I recommend highly that particular subject to be of interest. I found this paper entitled The Magic Of Hope: Hope Mediates The Relationship Between Socioeconomic Status And Academic Achievement to be of great interest. My greatly compressed take away is that there is a rather strong correlation between somebody’s socioeconomic status and the level of hope they feel especially as we define it in our earlier post being not only having the will to succeed but also being able to think of and access a number of ways to get you over your obstacles. That may not seem surprising or earthshaking in any way, but we already know that our level in possession of this kind of hope is malleable; it can be changed. In fact, we can change it with the right sort of help. I don’t want to sound like Pollyanna here (although as a kid I was much taken with Hayley Mills in the movie) and suggest that all these problems are surmountable by simply having a different attitude. They are not. But since the problems of inequality and racism and poverty are not  going to go away in the short term how someone deals with the problems definitely can make a marginal difference and a marginal difference is better than no difference at all.

Here’s how the authors of the study phrased their most important finding:

“Increasing a student’s SES is difficult, expensive, and highly impractical on a grand scale (J€antti et al., 2006). However, increasing a student’s hope can be inexpensive, highly effective, and can be employed at all levels of education (see Weis & Speridakos, 2011). Finding that hope not only mediates this relationship, but that it also accounts for the majority of the variance in academic achievement that SES and hope together explain, suggests that adolescents from low-SES backgrounds may be able to achieve as much as their higher SES counterparts (all else being equal), if they receive a little extra assistance via a hope intervention.” Let’s end there for today — with and on  hope