Sunday, November 4, 2018

New beginnings

I am slowly figuring out the next phase of the research portion of my career. This week some students and I embark on something that has been in the works for a few months. We are going to pilot an experiment examining one version of a word completion task (WCT) used in aggression research. We chose a 22-item task that is seeing some recent use as we needed to collect, enter, and analyze data in a very short time period.

We are examining if there is a difference in word completion rate and aggressive word completion rate in timed (3 minute) and untimed conditions. We are also interested in the internal consistency of the WCT that we are using.

We randomly assigned two classes of introductory psychology students to one of the two time conditions (timed or untimed). Students are receiving informed consent statements, and after they have consented, are presented with the WCT with either instructions that they will be timed or no time instructions. We also will collect some demographic data (e.g., age, sex).

We will code completed words as either aggressive or nonaggressive. Nonwords will be treated as non-responses for this pilot study. We will then use t-tests to compare overall word completion rate between conditions and aggressive words completed between conditions. We will use Cronbach's Alpha as a measure of internal consistency.

Our findings will be reported initially at my department's Psych Symposium.

At the moment we're making no predictions. This is purely exploratory work. I am hoping to shed some light on how vulnerable this version of the WCT is to measurement error and to get some idea if some of the flexibility in administration of this instrument has consequences for completion rates.

I expect that any feedback we receive will be incorporated into a larger scale study.

I have some other ideas in the works, based on my interests in some of the difficulties that might exist with some of our aggression measures. I am concerned that as an area within the psychological sciences that we collectively are not doing enough to determine that our measures are uniformly administered and sufficiently vetted as reliable (and validated) prior to being used in published work. We can and should do better. Hopefully with this instrument we did not do too badly. We'll know more soon enough.

Monday, October 29, 2018

Harbingers of the replication crisis

I am going to post a series of tweets by James Heathers (who if you are not following on Twitter, you are really missing out) highlighting a series of passages dating back over half a century. They serve as warnings that, had they been heeded, would have left many of my peers in my corner of the sciences feeling a lot different about the soundness of the work we cite in our own research and teach to students:

Let's not only heed those who tried to warn us over a half century ago, but heed those who are warning us now. I will not be around in a half century, but I will be around long enough to suss out if we're going to be actually progressing as a science or if we are just going to run over the same old ground.

Wednesday, October 24, 2018

Getting back to flexible measures:

The Buss Teacher-Learner approach seems to fit. Thankfully one set of authors was unwittingly honest enough to demonstrate.

We're more likely to see the CRTT used these days, but that more updated approach has been shown to be problematic. Those of us who genuinely care about aggression research need to do better. We need to know that our techniques are reliable, that they can be validated, that their administration and analysis can be standardized. Until then, we have no idea what we are really measuring, and simply finding a p<.05 isn't cutting it.

Saturday, October 13, 2018

RIP Bernardo "Bernie" Carducci

Bernardo Carducci passed away late in September. Both of us are alums of California State University, Fullerton. He attended and graduated from CSUF a number of years before I did, but he was a friend of several of the faculty there and he would often talk to his former mentors and to those of us presenting as students back in the day. I did not see him in person very often, but recall a man who was extremely outgoing and full of life. His work on shyness, while outside my normal specialty area, was of some intrinsic interest to me. My last contact with him had been over composing a couple chapters for an encyclopedia he was in the process of editing. He was one of those rare individuals who was adept at coming across via electronic media much as he might in person. Hopefully encyclopedia will serve as a part of his legacy. He will be missed.

Friday, October 5, 2018

Boom, Footshot! When is a replication not a replication?

By now you've read and considered what Patrick Markey blogged earlier. Patrick Markey and Malte Elson both noticed some anomalies in a paper by Jodi Whitaker and Brad Bushman that had both a bold and audacious premise and some troubling anomalies in the reported results. The premise fits in with something that some media violence researchers worry about and I can imagine more than a few weapons effect researchers have considered: the possibility that a realistic weapon control used in a first person shooter (FPS) gaming context could lead players to become adept shooters. Talk to folks who actually use firearms and they will probably tell you the premise is farfetched. Well, actually there are other words they might use, but we'll keep it PG or at least PG-13 on this blog. But I digress. The findings seemed too good to be true. Markey and Elson eventually get access to the data files, not without a lot of headaches, and what they found was disturbing. Ultimately the editor of the journal that published this masterpiece had to retract it, as it was not clear which data set was used to analyze the data, and some very suspicious code-switching seems to have occurred, allegedly without Bushman's knowledge. Although Bushman, however he managed to do so, was cleared of any wrong-doing, Whitaker ended up losing her PhD (she was a student of Bushman's) and her job.

Bushman initially insisted that the two of them were working on a replication, and later it was just Bushman working on the replication. So there were rumors of a replication for a while. The journal that initially accepted the original ended up ultimately rejecting the replication, but Aggressive Behavior did accept it. That's pretty much the nickel tour of the saga. Markey has looked at the data and has been able to reproduce the analyses and I have no doubt so has Elson. I've looked at the data as well. Basically, the study can only count as a replication if the author found what he and his former coauthor claimed to find the first go around: that a weapon controller used in an FPS game will lead to significantly more headshots than any other condition (regular controller FPS, weapon controller nonviolent shooting game, regular controller nonviolent shooting game, nonviolent game). Bushman did not find that. He had to collapse the weapon controller and regular controller conditions for both FPS and nonviolent shooting games. That had a little something to do with failing to find a significant difference between using a weapon controller in a FPS and a regular controller in a FPS on number of headshots fired with a real gun afterwards. If you look a bit further you'll suss out that there is no significant difference between number of headshots fired after playing either FPS or nonviolent shooting games. Oops. Really the analyses suggest that the findings amount to a whole lot of nothing. And that would be fine. We learn that something that seemed too good to be true really was not true. Life moves on. However Bushman really oversells the findings and continues to claim a replication when none actually occurred. Somehow that got past peer review. So it goes.

Personally, I think it is a good idea to read the papers side by side.

The original paper by Whitaker and Bushman (2014), minus the retracted watermark, can be found at Jodi Whitaker's Researchgate page. If you like watermarks, you can find it at sci-hub as well. It has the journal's retraction notice attached.

The so-called replication by Bushman can be found at the journal home page, but since it is behind a paywall, let's just give you the sci-hub link.

There may be some duplicate content between the documents as well. When I scan the documents side by side using Plagiarism Checker X Pro, depending on the parameters I set, there is anywhere between 19% and 30% duplicate content. Some of that is probably chalked up to the simple fact that there are only so many different ways to describe identical or relatively identical methods between both papers. Some of that may have to do with the end notes where Bushman goes into extended detail how the finding is a replication (even though it is not). The rest? I will leave that up to you to ponder. This particular effort could be a footshot in more ways than one.

The bottom line is that the paper that was retracted was done so for good reason, and that the paper billed as a replication is the furthest thing from a replication. It counts as a publication, but folks, I would not cite it unless looking for a cautionary tale to write about.

Sunday, September 30, 2018

Boom, Footshot!: An Introduction

Here's a really good summary of the recent alleged replication of the retracted Boom, Headshot study by Whitaker & Bushman (2014). I'll want to say more about this one later, but for now let's digest Patrick Markey's words. After all, he and Malte Elson did yeoman's work sleuthing the original study. Suffice it to say, the so-called replication is not really a replication (spoiler alert).

How France Created the Metric System

Although we mostly ignore the Metric System in the US, it is a standard of measurement that influences so much of our lives, and even more so the lives of our fellow human beings. This quick BBC article highlights how the Metric System came into being, as well as the difficulty in its mainstreaming - as the article notes, it took about a century. It was truly an accomplishment with revolutionary origins, and one that truly changed the world.