Motivation

No matter our background, no matter our vocation, there has to be something that gets us up in the morning. For me, lately, that is anger.

At what? Let's just say that the crisis which goes by many names (replication crisis, replicability crisis, methodological crisis) felt like a punch to the gut - and one I just did not see coming. As I digested what had happened and what was happening, I had to change my perspective about a field that defines a significant part of my identity. Initially I was a bit sanguine. Then as reality sank in, I got pissed off. After all, to maintain any semblance of integrity, I had to alert students in many of my classes that there were whole sections of textbooks that they were probably best ignoring, or viewing only as cautionary tales. That meant accepting that students would ask me, what was real, and for me to not necessarily feel like I had a satisfactory answer. A substantial chunk of work in my corner of our aching science seems to needlessly scare the hell out of people, and that work is not aging well. In fact the moral panics over video games and violence or screen time and any of a number of purported negative psychological health outcomes remind me of the moral panics that I grew up with: Dungeons and Dragons was supposed to damage teens psychologically, as were the lyrics of songs from many of my favorite bands of the time (remember that I enjoyed and still enjoy punk and punk-derived music from the late 1970s-mid 1980s). At the time I would see people make causal claims from correlational data (or merely out of thin air) and I would just think, "bullshit." One could say that I did become an educator, and maybe that questionable life choice is an outcome of questionable life choices I made in my youth, including my pop culture interests and activities of the day. I am pissed at a system of dissemination of our work that relies on the funds (at least indirectly) of our citizens but which once published becomes the property of some conglomerate that then sells the content back to the citizens at an insane profit, and sometimes with peer review and editorial standards that differ little from what most of us rightfully deride as predatory journals.

Thankfully, from punk I got both the attitude and the politics. The attitude is the easy part. The politics actually took a good deal of thought. And so here I am again. It would be easy to adopt a pose of casual contempt or indifference and merely sneer as I preview a textbook or read the latest journal article. That's not me. I actually care. So maybe a little anarchy (not in the sense of chaos!) will do us some good about now. Things get shaken up a bit and if that leads to the sort of changes we need (more open communication and archiving our work, more equality and equity in the profession as opposed to rigid hierarchies) I'm in. Reading much of what is coming out of the open science proponents is the equivalent of putting on an old familiar Black Flag or Dead Kennedys LP. Hell, sometimes I do both, especially if I am at the office on a weekend and can crank up the volume. The punks at their best were angry and thoughtful. They wanted to knock down stuff, but they also were also wanting to replace whatever was knocked down with something better (which was of course ever an open-ended question as to what that would entail). Whatever form that something better takes, I hope for a science that truly gives itself away in the public interest, rather than get coopted into some neoliberal facsimile of open science that merely repeats the same mistakes of the past. Doing what I can, as an educator and scholar who has little privilege or leverage to offer other than adding to the voices in the proverbial wilderness is enough for now. That gets me up in the morning, like clockwork.

Everything Went Black

I nicked that title from an old Black Flag album, from when they were temporarily not Black Flag due to a legal battle with their former label. Apparently issuing Damaged under Black Flag's new label really pissed off the suits at MCA. Eventually the label owned by MCA went under and Black Flag returned with a vengeance. Among bands in the hardcore scene circa the mid-1980s, Black Flag did not fit comfortably. It is not clear that they were even punk by the time they released some of my favorite recordings in 1984 and 1985. The band had moved into much more experimental territory, we elements of metal, and more importantly free jazz thrown into the mix. Add to that a very confrontational set of artists who clearly did not relish the ever present threat of violence at shows where their new sounds were increasingly alienating their core audience.

The way I saw it at the time, although I may not have worded it as such is that there was a crisis in the punk scene. The old formulas just did not seem to work any more, and openly admitting so was a good way to get sucker-punched, or stomped. One would certainly be shunned even if a punch were never thrown. So the old formulas remained in place, and punk became "another meaningless fad" (to nick a line from Dead Kennedys). What to do when what appeared to work before no longer does? One answer is to ignore it or wish it away. I certainly watched enough people come and go who did that back in the day. Another approach was to abandon what no longer worked and move in a different direction - ideally still embodying the ideals of the movement. Black Flag were quite adept at doing so for a few years. So was Flux (formerly Flux of Pink Indians), whose last album, Uncarved Block, was unlike any UK anarchopunk LP at the time. Probably should mention Chumbawamba while I am at it. There is something refreshing about searching for a new path when the old one has turned into a dead end. It happens in the arts, the sciences, and in life. As someone who was never more than one of the scenesters during the 1980s punk era, I knew it was time to follow some different muses when it became obvious that all that was left at the clubs and parties were folks who had the style and the attitude down, but who never really understood the ideas or the politics.

On some level, what I recall from a formative part of my early years serves as an allegory for what has gone on in my aching corner of the sciences as a methodological crisis has continued to unfold. There is so much I would love to write about. Problems in my little corner of the psychological sciences are the same ones affecting the rest of our aching field. Unfortunately when I am passionate about something that actually matters to me, I write with the heat of a thousand suns. Although that heat may not be aimed at one specific person or group, there is the chance it will be treated as such, placing me in a position that I find uncomfortable. Having to scrub this blog of content in order to prevent a situation from escalating is something I will not go through again. That is simply not tenable given the time it takes me to write, along with my numerous other commitments. When you are not a person of privilege (in the academic world, I and the institutions where I work are truly among the unprivileged), consequences hit twice as hard as for anyone else. Don't feel bad. I don't. Just the way it is. If you want to feel anything, feel anger. Then do something to make academic life more equitable. I guess I never really left my punk roots, and perhaps there is a reason I do have a good deal of empathy for those among psychology's reformers who advocate burning everything to the ground.

I am honestly not sure what I am going to do with this blog. I considered just deleting it altogether and look for other avenues to work out ideas, look at some problems that desperately need to be looked at, etc. Maybe that's the way to go. Maybe I will figure out a way to write as I wish. Time will tell.

Harbingers of the replication crisis

I am going to post a series of tweets by James Heathers (who if you are not following on Twitter, you are really missing out) highlighting a series of passages dating back over half a century. They serve as warnings that, had they been heeded, would have left many of my peers in my corner of the sciences feeling a lot different about the soundness of the work we cite in our own research and teach to students:

Thesis: 'methodological crisis' in science is NOT the sudden realisation of a problem. It is a well characterised problem which benefited hugely in recognition by a change in how scientists communicate and collaborate. Central issues were outlined clearly before 1970.

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Sterling, 1959. Publication bias, the file drawer problem, and the cult of significance. pic.twitter.com/N8sOnSNrUi

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Cohen, 1962. Social scientific studies are, in general, substantially underpowered. pic.twitter.com/wVwCws1gNu

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Forscher, 1963. The pursuit of publication rather than the pursuit of reliable results. pic.twitter.com/6m5IJMaS0J

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Platt, 1964. Topic-hopping, weak theory, and the role of induction over time. pic.twitter.com/zZUxaNtfxS

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Meehl, 1967. The role of the 'cute', surprising, or counter-intuitive results as an eventual outcome for capitalizing on chance. pic.twitter.com/XrManBcdNH

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Lykken, 1968. The weakness of statistical significance in isolation, the need for replication, the central importance of methods, and a whole lot more. pic.twitter.com/e5UgZepmft

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Does all of the above form a coherent body of work that people read at the time? No idea. Probably not.

But - these papers do address, straightforwardly and in better prose than we're allowed to write now, the heart of issues that a lot of people feel blindsided by at present.

— 🏴James Heathers🏴 (@jamesheathers) October 27, 2018

Let's not only heed those who tried to warn us over a half century ago, but heed those who are warning us now. I will not be around in a half century, but I will be around long enough to suss out if we're going to be actually progressing as a science or if we are just going to run over the same old ground.

Getting back to flexible measures:

The Buss Teacher-Learner approach seems to fit. Thankfully one set of authors was unwittingly honest enough to demonstrate.

The Buss Teacher-Learner method was another flexible measure among aggression researchers. Thankfully these authors provided us analyses for almost all the various ways the DV could be measured. Guess which version was the basis for their article. pic.twitter.com/OPEh8LSWZj

— James Benjamin (@AJBenjaminJr) October 22, 2018

We're more likely to see the CRTT used these days, but that more updated approach has been shown to be problematic. Those of us who genuinely care about aggression research need to do better. We need to know that our techniques are reliable, that they can be validated, that their administration and analysis can be standardized. Until then, we have no idea what we are really measuring, and simply finding a p<.05 isn't cutting it.

RIP Bernardo "Bernie" Carducci

Bernardo Carducci passed away late in September. Both of us are alums of California State University, Fullerton. He attended and graduated from CSUF a number of years before I did, but he was a friend of several of the faculty there and he would often talk to his former mentors and to those of us presenting as students back in the day. I did not see him in person very often, but recall a man who was extremely outgoing and full of life. His work on shyness, while outside my normal specialty area, was of some intrinsic interest to me. My last contact with him had been over composing a couple chapters for an encyclopedia he was in the process of editing. He was one of those rare individuals who was adept at coming across via electronic media much as he might in person. Hopefully encyclopedia will serve as a part of his legacy. He will be missed.

How France Created the Metric System

Although we mostly ignore the Metric System in the US, it is a standard of measurement that influences so much of our lives, and even more so the lives of our fellow human beings. This quick BBC article highlights how the Metric System came into being, as well as the difficulty in its mainstreaming - as the article notes, it took about a century. It was truly an accomplishment with revolutionary origins, and one that truly changed the world.

Following up on Wansink

Andrew Gelman is on point in this post. I will give you this clip as a starting point:

I particularly liked this article by David Randall—not because he quoted me, but because he crisply laid out the key issues:

The irreproducibility crisis cost Brian Wansink his job. Over a 25-year career, Mr. Wansink developed an international reputation as an expert on eating behavior. He was the main popularizer of the notion that large portions lead inevitably to overeating. But Mr. Wansink resigned last week . . . after an investigative faculty committee found he had committed a litany of academic breaches: “misreporting of research data, problematic statistical techniques, failure to properly document and preserve research results” and more. . . . Mr. Wansink’s fall from grace began with a 2016 blog post . . . [which] prompted a small group of skeptics to take a hard look at Mr. Wansink’s past scholarship. Their analysis, published in January 2017, turned up an astonishing variety and quantity of errors in his statistical procedures and data. . . . A generation of Mr. Wansink’s journal editors and fellow scientists failed to notice anything wrong with his research—a powerful indictment of the current system of academic peer review, in which only subject-matter experts are invited to comment on a paper before publication. . . . P-hacking, cherry-picking data and other arbitrary techniques have sadly become standard practices for scientists seeking publishable results. Many scientists do these things inadvertently [emphasis added], not realizing that the way they work is likely to lead to irreplicable results. Let something good come from Mr. Wansink’s downfall.

But some other reports missed the point, in a way that I’ve discussed before: they’re focusing on “p-hacking” and bad behavior rather than the larger problem of researchers expecting routine discovery.

That is I think how we should be focusing here. This is partially about scientists engaging in questionable behavior, but the focus should not be to pillory them. Rather, we should ask ourselves about a research culture that demands we find positive results each time we run a study. News flash: we're going to get a lot of findings that are at best inconclusive if we run enough studies. We should also focus on the fundamentals of research design along with making sure that any instruments used for measurement (whether behavioral, cognitive, attitudinal, etc.) are sufficiently reliable and have been validated. When I asked in an earlier post about how many Wansinks there are, I think I would want to clarify that question with a statement: the bulk of these scientists who could be the potential next Wansink are often well-intentioned individuals who are attempting to adapt to a particular set of environmental contingencies [1] (ones that reinforce positive results, or what Gelman calls routine discovery), and who are using measures that are quite frankly barely warmed over crap. In my area of social psychology I would further urge making sure that the theoretical models we rely on for our particular specialty areas are really ones that are measuring up. In aggression research, it is increasingly obvious to me that one model I relied on since my grad school days really needs to be rethought or altogether abandoned.

As we move forward, we do need to figure out what we can learn from the case of Brian Wansink, or anyone else for whom we might encounter a checkered history of questionable findings. I would recommend focusing less on the shortcomings of the individual (there is no need to create monsters) and focus instead on the behaviors, and how to change those behaviors (both individually and collectively).

[1] I am no Skinnerian, but I do teach Conditioning and Learning from time to time. I always loved that term, environmental contingencies.