Interesting podcast on Open Science and its enemies

Now that I have a couple of moments to breathe, I've been able to spend a bit of time on Bluesky (which is where a lot of academic Twitter landed after Elon Musk took over the platform and made it far worse), and reconnect with some folks whose work I respect. I have been gathering that there might be some trouble in paradise among the community advocating for Open Science. I've seen some chatter about a preprint that offers a very broad definition of what may be considered questionable research practices that have at least some in the community suggesting that the definition really is too broad. I may come back around to that, but if and when I have the time to do more than give the preprint an initial reading. Instead, I'll focus on a podcast that I had never heard of before entitled The Error Bar. It's a clever title, and the host, Nick Holmes, certainly strikes me as witty and open-minded. He is planning a three-part series on what he refers to as Open Science and its Enemies. This week's podcast is called the p-circlers. We can think of p-circling as reverse p-hacking according to Dr. Holmes. The upshot is that p-circlers hone in on a finding they do not like and then look for ways to make the finding seem suspect or so trivial as to not even be worth examining in the first place. Holmes focuses on one finding that seemed to create a stir, and a preprint that ends up resorting to p-circling behavior in order to explain away the findings as much ado about nothing. If you have 26 minutes or so to spare, it is a worthwhile podcast episode, and hopefully will provoke some thought. Since I have some proverbial skin in the game when it comes to presenting open science practices in my undergraduate methods courses, I want to make sure that our actions really do move our respective disciplines and sub-disciplines forward, rather than simply weaponize a series of recently developed tools for post-peer-review and in turn lead us to making the same mistakes as our predecessors. Anyway, give this episode a listen. I don't think you'll be disappointed.

P.S.: If you do not like listening to podcasts, Nick Holmes also blogs his podcasts. Here is the blog post for the episode on p-circlers.

Another moral panic debunked: The "woke" university

Judd Legum has a reasonably good explainer of the current moral panic that is targeting colleges and universities: the myth that these institutions function primarily for indoctrinating students into some sort of "woke" ideology. In a sense, this is simply a rehash of earlier moral panics targeting colleges and universities. In my day as an undergraduate student, there were still plenty of pundits who persisted to claim that universities were hotbeds of communist indoctrination - a notion I found quite amusing at the time. Later, I'd read that there was a noticeable downward trend in course offerings on Marxism and a downward trend in Marxist scholarship by the 1980s. My guess is that what really drove the moral panic of my day was student-led efforts to urge universities to divest from South Africa's Apartheid regime. Today's moral panic appears to have its origins primarily in DEI initiatives by universities and university systems starting late last decade, and more recently in student protests over the war in Gaza which is somewhat reminiscent of the anti-Apartheid protests from the 1980s and early 1990s. 

As Legum notes, many of the individuals who seem to be responsible for our current moral panic over "woke" ideology in universities happen to be billionaires (often in the tech sector) and political pundits on streaming services or in our legacy media. There are the usual anecdotes to make the reader's or viewer's blood boil. Anecdotes intended to provoke outrage aside, is there any data to back up the claims that our universities are indoctrinating our students? The answer turns out to be no.

Legum mentions an organization called Open Syllabus. The data collected by Open Syllabus can be quite helpful. Obviously the data in this case come from publicly available syllabi, which could be a limitation, but it at least gives us numbers to test the claim that our universities are too "woke" to effectively educate adult learners. It turns out that there are very few course syllabi that include terms like Critical Race Theory (or race more broadly), transgender (or gender more broadly) and so on. In other words, as Legum notes, it appears possible and even probable that students, even at elite universities, can go through a four year degree program without ever encountering any of the concepts that apparently cause our far-right billionaire class and punditry class to quake in their comfy slippers. We could certainly have a conversation as to whether a lack of exposure to structural racism, the concept of gender as a social construct, etc., is beneficial or detrimental to students who are preparing for careers in which they will likely work with and supervise a diverse set of individuals. There is certainly room for debate about the necessity and effectiveness of DEI initiatives in terms of fulfilling a university's primary mission and objectives. Clamping down on DEI programs, courses of study, and student speech with scant to near-non-existent evidence is far from conducive to an environment in which ideas are freely exchanged and challenged. 

The moral of the story is simple: if a claim is too outrageous to be true, it probably is. At that point, it is a really good idea to question the source or sources of the claim or claims, and do some digging to see if there is any trustworthy data to support their claim. If not, it is best to discount the claim as not valid and move on.

The problem of retracted articles continuing to receive citations

Now that I am mostly done with grading for the semester, it is time to turn my attention to a phenomenon that has bothered me for quite a long time, and will most likely continue to bother me. It is not uncommon for articles to get retracted for any of number of reasons. Sometimes, an error gets caught that invalidates the claim made by the authors, sometimes fraud is involved, and sometimes there is plagiarism involved. It happens. Life should go on. In an ideal world, once the article is retracted, that should be the end of its useful life. That article should receive no more citations. We don't live in an ideal world. I know. I am being Captain Obvious about that. The reality, as this article in Retraction Watch points out, is far more complicated, and concerning. Retracted articles may get cited less once the retraction is made public, but they still can rack up quite a number of citations.

I can think of a handful of reasons why an article might still be cited post-retraction:

1. In the months and year immediately after the retraction, papers by citing authors may have already been accepted for publication. It is quite likely that the citing authors would have no knowledge that a retraction was in the works. These are good-faith citations, and should be treated as such. The impact the retraction has on the papers by the citing authors may or may not have significant ramifications, depending on how much of their argument was anchored by the retracted article, or depending on if the effect size data from the retracted article was included in a meta-analysis. Usually the ramifications are fairly negligible, but we shouldn't always assume that to be the case.

2. The retracted article is cited as part of an argument for why the work of a lab, a principal investigator, and/or that PI's collaborators should be viewed with a very healthy dose of skepticism. Under such circumstances, we should expect that the citing authors will explicitly label the article or articles in question as retracted.

3. The retracted article is cited as part of a debate about whether or not a specific theoretical model is still viable in the face of one or more of a theorist's articles being retracted. Although I am not certain I would want to build much of a case that a theory is debunked because the theorist was either negligent or fraudulent in at least one instance, I can appreciate how such examples can provide some context. Under such circumstances we should expect that the citing authors will explicitly label the article or articles in question as retracted.

4. The retracted article is cited as a cautionary tale of what not to do. Retracted articles can be rich case studies in their own right, and guide scientists to steer clear of the sorts of mistakes or misdeeds that can lead to a retraction. No two retractions are exactly alike, although there are some overlapping patterns. My favorite retraction narratives are ones where the authors of a botched article actively work to get the offending article retracted. That said, when citing a retracted article as a cautionary tale, the article in question will be explicitly labeled as retracted.

Beyond those examples, I can't think of a defensible reason to continue citing an article that is for all intents and purposes removed from the public record (aside from the retraction notice specifying the reason for the retraction). I suppose it is possible that a citing author has cited the retracted article before it was retracted, and continues to cite it in their own future papers out of force of habit. That is not a good look. I see that sort of behavior as a sign that authors are not keeping current with the literature in their own areas of expertise. I can also imagine situations where the citing authors cite a retracted article in bad faith in order to further a theoretical or political agenda. That is also not a good look, and the societal ramifications are very, very concerning when citing authors simply hope that their readers don't bother to notice that they are basing their argument on findings that have been retracted for good reason. 

There is a simple solution for those of us who want to avoid citing retracted articles. The Google Chrome browser has a PubPeer extension that is really good at flagging retracted articles. Those who install and use that extension can be alerted to retracted articles well ahead of time, and that will give them ample opportunity to learn why the retraction happened and opportunity to find another article to cite instead. Be smart and use some simple tools that are freely available. In the process, you will help make your particular science more trustworthy. That's a good thing.