Research Confidential: Let's Discuss Investigator Allegiance Effects

The meta-analysis on the weapons effect on which I am the primary author examined a number of potential moderator effects. Our list of moderators was far from exhaustive, and I often wondered throughout the process what we might be missing. One of those might be something that Luborsky et al. (2006) and others call an investigator allegiance effect. Early on, when a new phenomenon is presented and examined, there is a tendency for investigators who are very closely affiliated with the principle investigator of the original finding to manage to successfully replicate the original findings, and for independent researchers to perhaps more difficulty in replicating. That pattern appeared to occur in the early years of research on the weapons effect. Several ex-students and post-docs of Leonard Berkowitz (e.g., Frodi, Turner, Leyens, etc., along with some of their own students) appeared to find additional evidence of a weapons effect. Others (e.g., Arnold Buss) often reported replication failures, even when faithfully reproducing the original Berkowitz and LePage (1967) findings - Arnold Buss et al. (1972) is a great example if you look at experiments 5 and 6.

Anyway, I noticed this pattern, and it always sort of gnawed on me. Recently, someone on Twitter (Sanjay Srivastava) suggested to me that what I was looking at was an allegiance effect, and that clinical researchers routinely look for that in meta-analysis. It had honestly never occurred to me. Something I had a gut feeling about not only had a name, but also could be quantified. So, I went to my old CMA database, added a moderator column for Berkowitz Allegiance Effect, and then coded all behavioral studies as yes (affiliated with Berkowitz) or no (not affiliated with Berkowitz). Since Berkowitz and LePage (1967) presumed that the weapons effect occurred under conditions of high provocation, I decided that studies where an aggressive behavioral outcome was a DV and in which there was a high provocation condition were where I should focus my preliminary analyses. Here is what I found:

If you click the picture, you will get a full sized image. Basically, in studies where the researchers were in some way connected with Berkowitz, the mean effect size was pretty strong, d = .513 (.110, .916). The mean effect size for those who were not affiliated with Berkowitz was d = . 192 (-.111, .495). Note that the confidence intervals for the mean effect size for those not affiliated with Berkowitz included zero, which makes sense as many of those authors reported replication failures. It appears that there is indeed an allegiance effect.

I ran a similar analysis on cognitive outcomes - adding Bushman and Anderson to the list, as these two authors drove much of the research literature on what we call a weapons priming effect. Full disclosure: I coauthored two of those articles with Anderson. Here is what I found in CMA:

 One thing to note when you click on this photo is that the mean effect size really does not change regardless of whether or not the study was authored by Anderson, Bushman, or Berkowitz or by someone affiliated with them. There is no apparent evidence for an allegiance effect for cognitive DVs.

Keep in mind that these are only preliminary findings. I have reason to be a bit more confident in the cognitive outcomes than I suspected. That said, simply showing a cognitive effect means little if we cannot consistently observe tangible behavioral change. Behavioral outcomes are truly another matter. One thing to note about the behavioral outcomes. Very few new studies that included a behavioral DV were published after Carlson et al. (1990) published their meta-analysis, which appeared to settle the debate about the weapons effect. There are some problems with that meta-analysis, which I should probably discuss at some point. Bottom line is that I think we as a field squandered an opportunity to continue to check for behavioral outcomes as our standards for proper sample sizes and so on changed. Only three studies that I am aware of have included a behavioral DV this decade, and one of those was conducted by a former student of one of Berkowitz's students (Brad Bushman, who was mentored by Russell Geen). The Bushman study appears to show a clean replication. The other two do not. Make of that what you will.