Harbingers of the replication crisis

I am going to post a series of tweets by James Heathers (who if you are not following on Twitter, you are really missing out) highlighting a series of passages dating back over half a century. They serve as warnings that, had they been heeded, would have left many of my peers in my corner of the sciences feeling a lot different about the soundness of the work we cite in our own research and teach to students:

Thesis: 'methodological crisis' in science is NOT the sudden realisation of a problem. It is a well characterised problem which benefited hugely in recognition by a change in how scientists communicate and collaborate. Central issues were outlined clearly before 1970.

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Sterling, 1959. Publication bias, the file drawer problem, and the cult of significance. pic.twitter.com/N8sOnSNrUi

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Cohen, 1962. Social scientific studies are, in general, substantially underpowered. pic.twitter.com/wVwCws1gNu

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Forscher, 1963. The pursuit of publication rather than the pursuit of reliable results. pic.twitter.com/6m5IJMaS0J

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Platt, 1964. Topic-hopping, weak theory, and the role of induction over time. pic.twitter.com/zZUxaNtfxS

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Meehl, 1967. The role of the 'cute', surprising, or counter-intuitive results as an eventual outcome for capitalizing on chance. pic.twitter.com/XrManBcdNH

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Lykken, 1968. The weakness of statistical significance in isolation, the need for replication, the central importance of methods, and a whole lot more. pic.twitter.com/e5UgZepmft

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Does all of the above form a coherent body of work that people read at the time? No idea. Probably not.

But - these papers do address, straightforwardly and in better prose than we're allowed to write now, the heart of issues that a lot of people feel blindsided by at present.

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Let's not only heed those who tried to warn us over a half century ago, but heed those who are warning us now. I will not be around in a half century, but I will be around long enough to suss out if we're going to be actually progressing as a science or if we are just going to run over the same old ground.

Getting back to flexible measures:

The Buss Teacher-Learner approach seems to fit. Thankfully one set of authors was unwittingly honest enough to demonstrate.

The Buss Teacher-Learner method was another flexible measure among aggression researchers. Thankfully these authors provided us analyses for almost all the various ways the DV could be measured. Guess which version was the basis for their article. pic.twitter.com/OPEh8LSWZj

— James Benjamin (@AJBenjaminJr) October 22, 2018

We're more likely to see the CRTT used these days, but that more updated approach has been shown to be problematic. Those of us who genuinely care about aggression research need to do better. We need to know that our techniques are reliable, that they can be validated, that their administration and analysis can be standardized. Until then, we have no idea what we are really measuring, and simply finding a p<.05 isn't cutting it.

RIP Bernardo "Bernie" Carducci

Bernardo Carducci passed away late in September. Both of us are alums of California State University, Fullerton. He attended and graduated from CSUF a number of years before I did, but he was a friend of several of the faculty there and he would often talk to his former mentors and to those of us presenting as students back in the day. I did not see him in person very often, but recall a man who was extremely outgoing and full of life. His work on shyness, while outside my normal specialty area, was of some intrinsic interest to me. My last contact with him had been over composing a couple chapters for an encyclopedia he was in the process of editing. He was one of those rare individuals who was adept at coming across via electronic media much as he might in person. Hopefully encyclopedia will serve as a part of his legacy. He will be missed.