Saturday, May 4, 2019

Imagine (a methods thought experiment)

Imagine a university in an emerging economic power becomes a major research institution early in the decade. Not surprisingly, some researchers within that university quickly take advantage of their university's new status to further their own careers. Fair enough. Who can blame them.

Now let's imagine that one of those researchers publishes work in your area of expertise. It's a large enough university that it would hardly be a surprise. Let's go further. Imagine that you first encounter an article by said researcher. The article seems riddled with multiple errors. You get concerned, as you should, and write the author. In your email, you ask for original data and analyses. You want to be able to reproduce what this researcher claims to have found. You never get a response. You move on with your life.

A couple years later, you realize that said researcher has published multiple articles that have a similar pattern of errors. Sometimes you see identical analyses all the way down to the exact test statistics. We're talking errors in basic descriptive statistics (means and standard deviations, marginal means) as well as inferential statistics (e.g., t-tests and F-tests). These are blatant errors. They are so obvious that you cannot miss them if you do more than a cursory scan of the abstract and discussion section of each article. How are you feeling now? What do you do? Those are questions I struggle with a lot these days. I am not sure my approach was necessarily great - that is to go public - but I do know that I could not just sit still.

This is a real scenario. I personally think that a lot of people should be pissed. I think that the citizens of the country in question should be pissed. After all, their taxes paid for an extended international excursion where some of this dodgy research was presumably worked on. Their tax dollars are funding grants for said author's research even now. I think coauthors should be pissed - especially if the researcher in question did not share data with coauthors. In particular I would advise those coauthors to read about Diederik Stapel and how he went about pulling the proverbial wool over a lot of people's eyes until he eventually got caught. I don't know if this particular situation is in the same league as Stapel, nor will I make that claim (as I have no evidence to do so), but I would not be able to rule that out absent actual data sets to look at (as those would be the evidence needed to rule out foul play). If I were a fellow scientist, I would be pissed, as this author's methodology and analyses are so poorly reported that the body of this author's work could be used to potentially delegitimize a whole area of research - all it takes is a few well-informed lay people to figure out that something is not quite adding up. We have enough problems in the psychological sciences without gross incompetence and potential fraud to worry about.

If you wonder about a few of my seeming obsessions as of the last several months, you at least understand where I am coming from. I am very concerned about the legitimacy of my field first and foremost. Whether or not the author in question publishes research that is in accordance with my professional opinions is not material. For the record, in at least one case, the author was initially preaching to the proverbial choir. In other cases, the author in question had a much harder sell. I am an open-minded skeptic who can be persuaded by good arguments based on solid data and analyses. So that just goes with the territory. We are supposed to be truth seekers and truth tellers who follow the evidence wherever it may lead.

So for now, I will continue to blog about periodic articles from this particular researcher and this particular researcher's lab. Heck, in one way or another I have been discussing this particular problematic research for around three years. Why stop now? This last year I have been much more public. As of yet I see no reason to stop. Neither this blog nor my Twitter account get much in the way of traffic. I am hardly someone who would be taken "seriously" by those who are at the top of the hierarchy in my field, and in my area of expertise. I am, after all, "just an educator" at a regional university no one has heard of. I do think the truth matters, and I am hopeful that the truth will come out.

As the late journalist John Ross used to say, "la lucha sigue, y sigue, y sigue..." This struggle to get the truth out is one that will not stop. Nor will I.

No comments:

Post a Comment