Patterns

The lab of which I have posted over intensively over the last few weeks is nothing if not consistent. We could even say that the lab has a sort of assembly line for doing research. The basic research design barely changes. The only things that change are perhaps the stimuli, the participants, and occasionally the dependent variable.

If you were to look for a metaphor that I understand, let's think about auto manufacturing. It is not uncommon for a major corporate conglomerate in the auto business to have a common platform across that is shared across various brand names. Change the headlights and grill, the tail lights, and the hubcaps (and perhaps various trim levels) and you can go from a Chevy to a Cadillac, just like that. If you are not wanting to believe me, look at the offerings for SUVs by Chevy, GMC, and Cadillac. You could do the same for Ford or Chrysler products.

An assembly line approach to social-personality research is not in itself intrinsically bad. There is a certain efficiency to that approach that is very much valued in our current capitalistic system. It is an approach that provides consistency as far as expectations are concerned, and is arguably more likely to be rewarded with grant dollars. It is what it is. We could even argue that a certain expertise is built in the process. In an auto assembly line, we swap out the headlights and grill, the tail lights, and the hubcaps, and perhaps some other creature comforts in the cabin and we go from branding the final product as an entry level vehicle or something more as a luxury vehicle. Regardless, we know how it will perform, we know what to expect in terms of quality control, and in terms of resale value. Maybe that is not how the Party in China would want to publicly frame things, butt it is how our global  capitalistic system works.

In the lab I have been critiquing, we can swap out the stimuli (violence in video games, violence in films, presence or absence of weapons), participants (preadolescent children, adolescents, or adult college students), and perhaps the operational definition of aggressive cognition (reaction time on a Stroop task or reaction time on some equivalent of a lexical decision task), but expect some consistency in results. In and of itself not necessarily a bad thing. Where things get tricky is in the process whenever we change one or more of the elements. My hunch for a while is that the lab in question has used a template, swapped in or out whatever needed to be swapped in or out, and reported findings as quickly as possible. Again, given the incentive system in capitalist economies, it is to be expected. Efficiency is rewarded. Inefficiency, not so much. The end result is the same. There is no honest way to mistake a Chevy for a Cadillac, no matter how hard one might try to rebrand it.

I am less concerned with a lab using a template for conducting research than I am the end result. It is one thing if there are a series of experiments that are conducted largely the same way, save for the requisite changes to IVs, samples, and DVs.  It is another thing when there are some serious mistakes based on relying on a template. What if the same test statistics are found in multiple experiments? What if an earlier sample is included in a larger study without readers understanding what was done and why? What if the same errors that made Chevy products unreliable back in the day (any one recall the Chevy Cavalier?) are found in equivalent Cadillac products (for example the Cimarron)?

What I am imparting is a lesson I learned a long time ago. It is one thing to be consistent. It is another to be consistently wrong. I have been concentrating on a lab whose products have been consistently wrong for a reason. That has less to do with the lab itself than on the fact that each lab has a consistent pattern of behaviors that itself has led to non-replicable findings that have led us to our current crisis. It is very possible to be consistently wrong. Changing the way variables are measured or changing the nature of the sample does not change that reality.

The past cannot be changed. At best, we can try to correct the record. We can change how we act in the here and now, as well as in the future. As we evaluate ourselves and others, we can ask about our patterns of doing our work, and if those patterns are truly serving us well, and more importantly - are they serving our science and the public well? If I seem obsessed, it is because that is what keeps me awake at night. Arguably, I need a new hobby. In the meantime, I have questions. Those questions deserve answers. Science is not self-correcting. It takes real living and breathing human beings to ask tough questions, as well as real living and breathing human beings to make conscious choices to own up to mistakes that they have made. Doing so is hardly easy. In an ideal world, doing so would be done in a sense where some kind of forebearance were the rule. We are far from that ideal. However, that is the ideal that matters most to me.