Friday, January 29, 2021

Research Confidential: How Self-Correcting is Science?

The question is arguably rhetorical. Science in and of itself is not self-correcting. It takes living, breathing human beings to notice something is wrong, take the time and make the effort to report what is wrong to relevant stakeholders (e.g., journal editors, relevant university adminstrations, etc.), and then have good reason to believe that the relevant stakeholders will show due diligence, correct or retract flawed papers as needed, and otherwise hold those responsible for the flaws, whether due to sheer incompetence or fraud, accountable. In an ideal world, that is how it would work. In this world, it's considerably more complicated, and often more than a bit disheartening.

If you are a regular reader of this blog, you are quite aware of our favorite media violence researcher who is notorious for some of the worst papers published in that particular niche area of psychology - Qian Zhang of Southwest University. I have documented, over the last couple years or so, some of the most insane tables with means, standard deviations, and test statistics that simply are impossible to interpret. I have reported test statistics that, based on the degrees of freedom reported, would have to be incorrect. I have reported discrepancies between degrees of freedom for test statistics and the sample size reported. I have documented evidence of potential plagiarism and self-plagiarism - the latter due to the tendency for Zhang to rely heavily on copying and pasting from one paper to another. I have also found some amusing typos that resulted from Zhang's tendency to copy and paste tables from paper to paper. I've tagged Zhang's work as I have documented here (for your convenience) and on PubPeer under a pseudonym. 

Dr. Joe Hilgard has gone considerable further than have I. He's blogged about his own experiences in documenting problems with Zhang's work in great detail, and the efforts he's made to contact journal editors along with officials at Zhang's university, offering painstaking evidence of the problems he has discovered. You can read about Joe Hilgard's efforts, and the decidedly mixed and disappointing outcome of his efforts here. You should really take to the time to read Hilgard's post as it is thorough and damning. The short version? Some journal editors responded rather well, and in one case very quickly to retract two papers that were clearly unsound. Other journal editors have either stonewalled or ignored Hilgard's concerns. Zhang's university cleared him of wrongdoing, chalking it all up to Zhang being "deficient in statistical knowledge and research methods." So in other words, the university writes it off as "the guy's merely an idiot, but hey, let's just give him a remedial stats course and call it even." I agree with Hilgard that the university's failure to take action is not that surprising, as universities seem to be in the business of taking care of their own, especially if the researcher in question might be bringing in grants or other forms of prestige. So the guy maybe fudges some numbers and has no idea what random assignment means. There's nothing to see here. Move along.

My take on the matter is that the most charitable view that one could take based on the body of Qian Zhang's work is that this is a researcher who is grossly incompetent, but that a more probably defensible case can be made that his activities are on some level fraudulent. I am more inclined to the latter less charitable view. I've seen too much. Regardless, this is research that should have never made it past peer review. I agree with Hilgard that this body of research is very problematic given that as long as it remains published, it will distort our understanding of what is actually happening with stimuli such as video games that contain violent content on outcome variables such as aggressive behavior or cognition. Meta-analyses are especially vulnerable given that some of the reported findings by Zhang rely on large samples. Those results could artificially inflate effect sizes, leading meta-analysts and those consuming meta-analyses to believe that an overall effect is stronger than it actually is. 

This is one of the dark alleys I mentioned a few years ago. And given what Hilgard has experienced and what I've experienced in my own way, it's one that few leave with any sense of hope for the state of this particular are of psychological inquiry. If blatantly problematic papers, ones where the problems are so obvious that a beginning methods student could discover them, cannot be retracted within a short window of time, what is going on with work in which potentially fraudulent data analyses are more cleverly presented? What else is out there that cannot be trusted? That is something that should cause us all to lose some sleep.

One final thought for anyone thinking of collaborating with Zhang: don't. If you absolutely cannot help yourself, insist on seeing the data before agreeing to be part of that particular project. I'd say that is a safe practice regardless of the situation. If I take on a statistician for a project, or someone who is at least better versed in a particular statistical method than I am, I insist on sending the data set or database, and I expect that the statistician on the project will double check my work and ask difficult questions as needed. That can save a lot of grief, assuming that the statistician involved is actually looking at what is being sent. One of the tragedies for some of Zhang's coauthors is that they've never had access to the data sets to which they lent their names and reputations, nor were they apparently allowed access. That is not how we do science, folks.

In the meantime, more papers are in the pipeline to be published by this particular author, and it will become more of a struggle to keep up with the dross that is likely to be found in any of those papers. Again, that is something that should cause us all to lose some sleep.

Wednesday, January 6, 2021

The attempt to thwart the certification of electors: some quick thoughts

I'm pretty exhausted at the moment, given the unexpectedly long day we experienced here in the US, and perhaps around the world. This is the first time that I have ever witnessed something like this in our nation's capital. I am aware this happens in fragile democracies elsewhere. So, we're really not that special, I suppose. So I offer a few thoughts. Perhaps I will expand on those later.

I think it is interesting to be reminded that the last time the Capitol Building had been breached by insurrectionists was during the War of 1812. Thanks to Sen. Cory Booker for the history lesson. In each case many insurrectionists carried flags devoted to a single despot. Back then, it was a King of England. Today, it is actually a sitting President. The difference then is that this particular day of violence was instigated not by a foreign adversary but someone who had sworn to uphold the US Constitution. 

I've been reading The Authoritarian Nightmare by John Dean and Bob Altemeyer. I suspect that authoritarianism explains much of what we saw this January 6, 2021, and really events leading up to this moment. The authors rightly point out that often we focus on the despot in situations like this, and that is important. But, every despot needs willing followers. In our case Trump is the despot. He would probably score as an authoritarian if he were to complete any of a number of authoritarianism questionnaires. He's highly social dominant as well, which is pretty obvious to all. His followers tend to wish for some fantasized past that either no longer exists or never really quite did, are submissive to an authoritarian leader's demands, and are willing to accept or participate in aggressive and violent acts if encouraged by authoritarian leaders. Trump has been practically shouting via Twitter (his favorite means of communication) and rallies that he won by a landslide and that the election was stolen. He spouts a lot of conspiracy theories for how such a highly unlikely thing could happen. His followers are enraged as he would want. He's been encouraging them for a bit now to come to a rally he'd hold in DC during the Congressional certification of Biden as the duly elected President and Harris as the duly elected Vice President. He spouted more of the usual, along with a few others sharing the podium with him. He encouraged the crowd to march to the Capitol Building, which they did. And the rest is history. 

That's the short version of the story, anyway. It's probably a bit more complicated, but that appears to be the gist. Inside the mind of those willing to do his bidding, the Congressional leadership has become the enemy. Trump has quite a number of elected Congressional members willing to do his bidding, minus the violence as well. We are witnessing that, which is why certification of the Electoral Vote will stretch into the wee hours overnight.

I would watch for what Trump says and does. Chances are that he will demand even more extreme actions from his followers in the days leading up to Biden's swearing in on January 20th. I would certainly hope FBI officials are monitoring the usual places where his followers congregate on social media (Parler, Gab, etc.). His true believers still believe, and are still almost certainly willing to act.

Friday, January 1, 2021

Our political divide as explained by a neuroscientist and a political scientist

Happy New Year

 

Well, when we rang in 2020, I seriously doubt any of us was thinking of SARS-CoV-2 (or COVID-19 as it is also known) would turn our personal and professional lives upside down. As I was wishing you all a happy new year last year, I was expecting to be going to a couple professional conferences (NSSA and SIPS) and would be participating in some way with repliCATS. I expected to do my usual AP Psychology Reading in Tampa in June. Those did happen, in a way, but all of those activities ended up virtual. It appears to be virtual conferences and so on for the upcoming year as well, at least until the vaccines that are very painfully slowly being rolled out are administered to enough people to allow us to discover the emerging contours of our new normal. In some ways, I miss in person conferences, but I don't miss the travel expenses I racked up over the years. After universities on my level stopped paying conference travel and hotel in advance, folks like me loaded up credit cards, awaited partial reimbursement, and then found that in the interim that the interest accrued alone had wiped out any benefit to the partial reimbursements. Having a couple years where I am actually paying down credit cards is a real boon. I may just decide to abandon all travel going forward and stick to virtual options. Such is life. Those of us who toil in smaller regional colleges and universities in the US probably have more in common with our peers in the Global South than we might realize.

As for the blog? The pandemic really had some impact on the amount of content I posted. Flipping labs from seated to online was a huge task, and that took a lot out of me. That didn't stop me from posting items of interest. That said, some of what had my interest in 2019 had more or less played itself out. There are only so many critiques of a specific lab's papers one can write before it really does become overkill. I am hoping that a broader point of using tools available for detecting problems (Statcheck, GRIM, SPRITE) gets through, and that we see more folks use those tools and report problems as they find them. I am sure I will find some items of interest to discuss. Anytime I see an injustice in my field, I certainly want to find time for it. I am hoping that as time permits, I can discuss some projects that I effectively shelved in 2020. Those need to be dusted off and completed. Starting something and not finishing it tends to leave a bad taste in my mouth, so it's close to time to get back to it. 

I do have a challenging spring semester ahead of me. I am once more dealing with lab courses online that are better suited for in-person instruction. That will again suck up some time, and give me some heartburn. I agreed to some overload for the spring. I am glad I did as it looks like one of my streams of income (some adjuncting for a nearby community college) is going to dry up, at least for a while. If my other income streams remain constant, it won't be too much of a problem. Unfortunately, in the academic world, there is far too much uncertainty. 

This will be a challenging year. At least this time around, I have an idea of what is headed my way. That's something of an advantage. 

I hope each of you has a better year than last year. Take care and stay safe.

Prebunking as a strategy to reduce vulnerability to disinformation

 This is an interesting application of innoculation theory.

Check it out:

Roozenbeek, J.; van der Linden, S; Nygren, T. (2020). Prebunking interventions based on “inoculation” theory can reduce susceptibility to misinformation across cultures. The Harvard Kennedy School (HKS) Misinformation Review. https://doi.org/10.37016//mr-2020-008