Another article from the Zhang lab was published very recently. I do have to say that the format of the manuscript is well-done compared to many of the earlier papers. There are some initial concerns I will voice now. I may come back to this one later when and if the moment arises.
The literature review notes a number of meta-analyses that purport to provide support for media violence causing aggressive outcomes. The authors do offer a quick summary of several other meta-analyses that show that the average effect sizes from media violence research are negligible. Then the authors quickly state that the evidence is "overwhelming" that violent media leads to aggression. Um....not quite. There is actually a serious debate in which the extent that any link between exposure to violent content in films, video games, etc. and actual aggressive behavior, and the impression I get is that at best the matter is far from settled. The evidence for is arguably underwhelming rather than overwhelming. But hey, let's blow through all that research and at least check off the little box saying it was cited before discarding its message.
I am not too keen on the idea of throwing participants out of a study unless there is a darned good reason. Equipment malfunctions, failures to follow instructions, suspicion (i.e., guessing the hypothesis) would strike me as good reasons. Merely trying to get an even number of participants in treatment and control condition is not in and of itself a good reason. If one is inclined to do so anyway and state that the participants whose data were examined were randomly chosen, then at least go into some detail as to what that procedure entailed.
I will admit that I do not know China particularly well, but I am a bit taken aback that 15 primary schools could yield over 3,000 kids who are exactly 10 years of age. That is...a lot. Those schools must be huge. Then again, these kids go to school in a mega-city, so perhaps this is within the realm of possibility. This is one of those situations where I am a bit on the skeptical side, but I won't rule it out. Research protocols would certainly clarify matters on that point.
I am not sure why the authors use Cohen's d for effect size estimates for main effect analyses and then use eta square for the remaining ANOVA analyses. Personally I would prefer consistency. It's those inconsistencies that make me want to ask questions. At some point I will dive deeper into the mediation analyses. Demonstrating that accessibility of aggressive thoughts mediates the link between a particular exemplar of violent media and aggression is the great white whale that aggression researchers have been trying to chase for a good while now. If true and replicable, this would be some potentially rare good news for models of aggression derived from a social cognition perspective.
It is not clear if there were any manipulation checks included in the experimental protocols, nor if there was any extensive debriefing for suspicion - i.e. hypothesis guessing. In reaction time experiments that I ran, as well as any experiments using the competitive reaction time as a measure of aggression, it was standard operating procedure to not only have manipulation checks and an extensive debriefing with each participant, as problems like suspicion could contaminate the findings. Maybe those are procedural practices that have been abandoned altogether? I would hope not.
One of the most difficult tasks in conducting any media violence experiment is ascertaining that the violent and nonviolent media samples in question are as equivalent as possible except of course level of violent content. It is possible that the cartoon clips the authors use are perfectly satisfactory. Unfortunately we have to take that on faith for the time being.
At the end of the day, I am left with a gut feeling that I shouldn't quite believe what I am reading, even if it appears relatively okay on the surface. There are enough holes in the report itself that I suspect a well-versed skeptic can have a field day. Heck, as someone who is primarily an educator, I am already finding inconvenient questions and all I have done is give the paper an initial reading. This is my hot take on this particular paper.
Assuming the data check out, what I do appear to be reading thus far suggests that these are effects that are small enough to where I would not want to write home about them. In other words, in a best case scenario, I doubt this paper is going to be the one to change any minds. It will appeal to those needing to believe that violent content in various forms of mass media are harmful, and it will be either shrugged off or challenged by skeptics. I guess this was good enough for peer review. It is what it is. As I have stated elsewhere, peer review is a filter, and not a perfect filter. The rest is left to us who want to tackle research post-peer review.
Tuesday, May 7, 2019
Monday, May 6, 2019
Patterns
The lab of which I have posted over intensively over the last few weeks is nothing if not consistent. We could even say that the lab has a sort of assembly line for doing research. The basic research design barely changes. The only things that change are perhaps the stimuli, the participants, and occasionally the dependent variable.
If you were to look for a metaphor that I understand, let's think about auto manufacturing. It is not uncommon for a major corporate conglomerate in the auto business to have a common platform across that is shared across various brand names. Change the headlights and grill, the tail lights, and the hubcaps (and perhaps various trim levels) and you can go from a Chevy to a Cadillac, just like that. If you are not wanting to believe me, look at the offerings for SUVs by Chevy, GMC, and Cadillac. You could do the same for Ford or Chrysler products.
An assembly line approach to social-personality research is not in itself intrinsically bad. There is a certain efficiency to that approach that is very much valued in our current capitalistic system. It is an approach that provides consistency as far as expectations are concerned, and is arguably more likely to be rewarded with grant dollars. It is what it is. We could even argue that a certain expertise is built in the process. In an auto assembly line, we swap out the headlights and grill, the tail lights, and the hubcaps, and perhaps some other creature comforts in the cabin and we go from branding the final product as an entry level vehicle or something more as a luxury vehicle. Regardless, we know how it will perform, we know what to expect in terms of quality control, and in terms of resale value. Maybe that is not how the Party in China would want to publicly frame things, butt it is how our global capitalistic system works.
In the lab I have been critiquing, we can swap out the stimuli (violence in video games, violence in films, presence or absence of weapons), participants (preadolescent children, adolescents, or adult college students), and perhaps the operational definition of aggressive cognition (reaction time on a Stroop task or reaction time on some equivalent of a lexical decision task), but expect some consistency in results. In and of itself not necessarily a bad thing. Where things get tricky is in the process whenever we change one or more of the elements. My hunch for a while is that the lab in question has used a template, swapped in or out whatever needed to be swapped in or out, and reported findings as quickly as possible. Again, given the incentive system in capitalist economies, it is to be expected. Efficiency is rewarded. Inefficiency, not so much. The end result is the same. There is no honest way to mistake a Chevy for a Cadillac, no matter how hard one might try to rebrand it.
I am less concerned with a lab using a template for conducting research than I am the end result. It is one thing if there are a series of experiments that are conducted largely the same way, save for the requisite changes to IVs, samples, and DVs. It is another thing when there are some serious mistakes based on relying on a template. What if the same test statistics are found in multiple experiments? What if an earlier sample is included in a larger study without readers understanding what was done and why? What if the same errors that made Chevy products unreliable back in the day (any one recall the Chevy Cavalier?) are found in equivalent Cadillac products (for example the Cimarron)?
What I am imparting is a lesson I learned a long time ago. It is one thing to be consistent. It is another to be consistently wrong. I have been concentrating on a lab whose products have been consistently wrong for a reason. That has less to do with the lab itself than on the fact that each lab has a consistent pattern of behaviors that itself has led to non-replicable findings that have led us to our current crisis. It is very possible to be consistently wrong. Changing the way variables are measured or changing the nature of the sample does not change that reality.
The past cannot be changed. At best, we can try to correct the record. We can change how we act in the here and now, as well as in the future. As we evaluate ourselves and others, we can ask about our patterns of doing our work, and if those patterns are truly serving us well, and more importantly - are they serving our science and the public well? If I seem obsessed, it is because that is what keeps me awake at night. Arguably, I need a new hobby. In the meantime, I have questions. Those questions deserve answers. Science is not self-correcting. It takes real living and breathing human beings to ask tough questions, as well as real living and breathing human beings to make conscious choices to own up to mistakes that they have made. Doing so is hardly easy. In an ideal world, doing so would be done in a sense where some kind of forebearance were the rule. We are far from that ideal. However, that is the ideal that matters most to me.
If you were to look for a metaphor that I understand, let's think about auto manufacturing. It is not uncommon for a major corporate conglomerate in the auto business to have a common platform across that is shared across various brand names. Change the headlights and grill, the tail lights, and the hubcaps (and perhaps various trim levels) and you can go from a Chevy to a Cadillac, just like that. If you are not wanting to believe me, look at the offerings for SUVs by Chevy, GMC, and Cadillac. You could do the same for Ford or Chrysler products.
An assembly line approach to social-personality research is not in itself intrinsically bad. There is a certain efficiency to that approach that is very much valued in our current capitalistic system. It is an approach that provides consistency as far as expectations are concerned, and is arguably more likely to be rewarded with grant dollars. It is what it is. We could even argue that a certain expertise is built in the process. In an auto assembly line, we swap out the headlights and grill, the tail lights, and the hubcaps, and perhaps some other creature comforts in the cabin and we go from branding the final product as an entry level vehicle or something more as a luxury vehicle. Regardless, we know how it will perform, we know what to expect in terms of quality control, and in terms of resale value. Maybe that is not how the Party in China would want to publicly frame things, butt it is how our global capitalistic system works.
In the lab I have been critiquing, we can swap out the stimuli (violence in video games, violence in films, presence or absence of weapons), participants (preadolescent children, adolescents, or adult college students), and perhaps the operational definition of aggressive cognition (reaction time on a Stroop task or reaction time on some equivalent of a lexical decision task), but expect some consistency in results. In and of itself not necessarily a bad thing. Where things get tricky is in the process whenever we change one or more of the elements. My hunch for a while is that the lab in question has used a template, swapped in or out whatever needed to be swapped in or out, and reported findings as quickly as possible. Again, given the incentive system in capitalist economies, it is to be expected. Efficiency is rewarded. Inefficiency, not so much. The end result is the same. There is no honest way to mistake a Chevy for a Cadillac, no matter how hard one might try to rebrand it.
I am less concerned with a lab using a template for conducting research than I am the end result. It is one thing if there are a series of experiments that are conducted largely the same way, save for the requisite changes to IVs, samples, and DVs. It is another thing when there are some serious mistakes based on relying on a template. What if the same test statistics are found in multiple experiments? What if an earlier sample is included in a larger study without readers understanding what was done and why? What if the same errors that made Chevy products unreliable back in the day (any one recall the Chevy Cavalier?) are found in equivalent Cadillac products (for example the Cimarron)?
What I am imparting is a lesson I learned a long time ago. It is one thing to be consistent. It is another to be consistently wrong. I have been concentrating on a lab whose products have been consistently wrong for a reason. That has less to do with the lab itself than on the fact that each lab has a consistent pattern of behaviors that itself has led to non-replicable findings that have led us to our current crisis. It is very possible to be consistently wrong. Changing the way variables are measured or changing the nature of the sample does not change that reality.
The past cannot be changed. At best, we can try to correct the record. We can change how we act in the here and now, as well as in the future. As we evaluate ourselves and others, we can ask about our patterns of doing our work, and if those patterns are truly serving us well, and more importantly - are they serving our science and the public well? If I seem obsessed, it is because that is what keeps me awake at night. Arguably, I need a new hobby. In the meantime, I have questions. Those questions deserve answers. Science is not self-correcting. It takes real living and breathing human beings to ask tough questions, as well as real living and breathing human beings to make conscious choices to own up to mistakes that they have made. Doing so is hardly easy. In an ideal world, doing so would be done in a sense where some kind of forebearance were the rule. We are far from that ideal. However, that is the ideal that matters most to me.
About those Stroop task findings (and other assorted oddities)
I am hoping you all are generally familiar with the Stroop Interference Task. It has been around a long time. If you need a refresher, this will at least be a start. In my neck of the research woods, some form of an Emotional Stroop Task is occasionally used. It works just like the original task. Individuals are presented with multiple trials in which a word is paired with a color and the individual is supposed to name the color. Of interest is reaction time to naming the color. If there is some sort of interference, individuals should respond more slowly to the stimuli than they would otherwise.
In aggression research, an example might help. Here is a version of a Stroop task in which individuals are primed with either weapons or neutral objects:
And here are the results:
Notice that in the weapon priming condition, participants took significantly longer to respond to colors paired with aggressive words than neutral words. Contrast that with those participants in the neutral objects condition.
So far so good.
Now, here is an example of a Stroop task in which the prime stimulus is movie violence. Check out the results:
Notice something odd. Presumably, if violent movies were priming aggressive thoughts, the reaction times for aggressive words would have been significantly higher than for non-aggressive words in the violent movie condition. Instead, reaction times are lower. Strange.
Of course, there is some other weirdness. We get overall mean reaction times for aggressive and non-aggressive words in Table 2. Those could not possibly be right if the means and standard deviations are correct in Table 3. Again, something is just not adding up. I could nitpick a bit and ask why there are no decimal points for the means in each cell, but there are decimal points for standard deviations. That is a very, very minor complaint. The larger complaint is that the authors are not going to be able to say that violent movies increase accessibility of aggressive thoughts based on what they presented using a Stroop task, and that there is a huge (I mean HUGE) disconnect between the overall mean reaction times in Table 2 and the cell means provided in Table 3. After that, all we get are mean difference scores. I don't necessarily have a problem with that, although in Table 5, there is no way that the cell means provided are going to square with the marginal means for violent and non-violent movie conditions. That is also problematic. And if those difference scores are to be believed, it would appear that short-term exposure to a violent movie might actually suppress accessibility of aggressive thoughts among the subsample measuring high in trait aggressiveness, which is not what the authors wish to argue.Again, that is highly problematic. I suppose that the authors made some typos when they were writing up the paper? There is no way of knowing without access to the data set.
Also, Table 4 is just odd. It is unclear just how a MANCOVA would be appropriate as the only DV that the authors consider for the remaining analyses is a difference score. MANOVA and MANCOVA are appropriate analytic techniques for situations in which multiple DVs are analyzed simultaneously. The authors fail to list a covariate. Maybe it is gender? Hard to say. Without an adequate explanation, we as readers are left to guess. Even if a MANCOVA were appropriate, Table 4 is a case study in how not to set up a MANCOVA table. Authors should be explicit about what they are doing as possible. I can read Method and Results sections just fine, thank you. I cannot, however, read minds. Zhang et al. (2013) have placed us all in the position of being mind-readers.
Again, this is a strange article in which the analyses don't make much sense. Something is off in the way the descriptive statistics are reported. Something is off in the way that the inferential statistics are reported. The findings perhaps "looked" superficially good enough to get through peer review. It is possible that depending on the set of reviewers, the findings were ones that fit their own pre-existing beliefs - in other words some form of confirmation bias. I can only speculate on that, and will simply leave it at that. Speculation is not ideal of course.
This particular journal is not exactly a premier journal, which is hardly a knock on it. A number of low-impact journals do no worse, as near as I can tell, then the more premier journals when it comes to the quality of peer review. What matters is that this is an example of an article that slipped through peer review that arguably should not have. Given that this same lab has produced a number of subsequent articles with many of the same problems in more premier journals, I am justifiably worried.
Consider this a cautionary tale.
Reference:
Zhang, Q. , Zhang, D. & Wang, L. (2013). Is Aggressive Trait Responsible for Violence? Priming Effects of Aggressive Words and Violent Movies. Psychology, 4, 96-100. doi: 10.4236/psych.2013.42013
And here are the results:
Notice that in the weapon priming condition, participants took significantly longer to respond to colors paired with aggressive words than neutral words. Contrast that with those participants in the neutral objects condition.
So far so good.
Now, here is an example of a Stroop task in which the prime stimulus is movie violence. Check out the results:
Of course, there is some other weirdness. We get overall mean reaction times for aggressive and non-aggressive words in Table 2. Those could not possibly be right if the means and standard deviations are correct in Table 3. Again, something is just not adding up. I could nitpick a bit and ask why there are no decimal points for the means in each cell, but there are decimal points for standard deviations. That is a very, very minor complaint. The larger complaint is that the authors are not going to be able to say that violent movies increase accessibility of aggressive thoughts based on what they presented using a Stroop task, and that there is a huge (I mean HUGE) disconnect between the overall mean reaction times in Table 2 and the cell means provided in Table 3. After that, all we get are mean difference scores. I don't necessarily have a problem with that, although in Table 5, there is no way that the cell means provided are going to square with the marginal means for violent and non-violent movie conditions. That is also problematic. And if those difference scores are to be believed, it would appear that short-term exposure to a violent movie might actually suppress accessibility of aggressive thoughts among the subsample measuring high in trait aggressiveness, which is not what the authors wish to argue.Again, that is highly problematic. I suppose that the authors made some typos when they were writing up the paper? There is no way of knowing without access to the data set.
Also, Table 4 is just odd. It is unclear just how a MANCOVA would be appropriate as the only DV that the authors consider for the remaining analyses is a difference score. MANOVA and MANCOVA are appropriate analytic techniques for situations in which multiple DVs are analyzed simultaneously. The authors fail to list a covariate. Maybe it is gender? Hard to say. Without an adequate explanation, we as readers are left to guess. Even if a MANCOVA were appropriate, Table 4 is a case study in how not to set up a MANCOVA table. Authors should be explicit about what they are doing as possible. I can read Method and Results sections just fine, thank you. I cannot, however, read minds. Zhang et al. (2013) have placed us all in the position of being mind-readers.
Again, this is a strange article in which the analyses don't make much sense. Something is off in the way the descriptive statistics are reported. Something is off in the way that the inferential statistics are reported. The findings perhaps "looked" superficially good enough to get through peer review. It is possible that depending on the set of reviewers, the findings were ones that fit their own pre-existing beliefs - in other words some form of confirmation bias. I can only speculate on that, and will simply leave it at that. Speculation is not ideal of course.
This particular journal is not exactly a premier journal, which is hardly a knock on it. A number of low-impact journals do no worse, as near as I can tell, then the more premier journals when it comes to the quality of peer review. What matters is that this is an example of an article that slipped through peer review that arguably should not have. Given that this same lab has produced a number of subsequent articles with many of the same problems in more premier journals, I am justifiably worried.
Consider this a cautionary tale.
Reference:
Zhang, Q. , Zhang, D. & Wang, L. (2013). Is Aggressive Trait Responsible for Violence? Priming Effects of Aggressive Words and Violent Movies. Psychology, 4, 96-100. doi: 10.4236/psych.2013.42013
Sunday, May 5, 2019
Some more oddness (Zhang, Xiong, Tian, 2013)
Here's another relevant oldie but goodie by Zhang's lab:
What is odd about this one - aside from Table 3, which is just a mess - is that there is no way that the simple effects F for Boys would be statistically significant given the degrees of freedom. Recall that there are 74 participants in the experiment. That chews up one df. Now let's look at the df for each comparison that would have to be computed, regardless of whether or not it was reported. There are three main effects and a DV which we'll simply note is the difference in reaction time between aggressive and non-aggressive words in a Stroop task. Main effect for movie type costs 1 df, main effect for gender costs another df, and main effect for trait aggressiveness as defined in the article costs 2 df. There are the interactions of movie type x gender (1 df), movie type x trait aggressiveness (2 df) and gender x trait aggressiveness (2 df). The three-way interaction will take another 2 df. That leaves us with 62 df. A significant F-test for simple main effects would need to be practically 4.00. The obtained F in Table 4 for Boys is well below that threshold. And yet the authors claim statistical significance for that particular subsample. The authors want to claim a significant movie type x gender interaction, but have no significant simple main effects. That does not compute, folks.
This, by the way, is one of the better articles from this lab in terms of - at least on the surface - data reporting.
One more thing to note: the authors improperly use terminology commonly used by aggression researchers incorrectly. They are not measuring aggressive attitude levels. They are measuring accessibility of aggressive cognition (or thoughts, in lay terms). I see that mistake enough times to acknowledge that it is hardly unique to Zhang's lab. It is just an annoyance, I suppose.
The article can be read here.
File under articles that probably should not be cited without adequate clarification from the authors.
Reference:
Zhang, Q. , Xiong, D. and Tian, J. (2013) Impact of media violence on aggressive attitude for adolescents. Health, 5, 2156-2161. doi: 10.4236/health.2013.512294
What is odd about this one - aside from Table 3, which is just a mess - is that there is no way that the simple effects F for Boys would be statistically significant given the degrees of freedom. Recall that there are 74 participants in the experiment. That chews up one df. Now let's look at the df for each comparison that would have to be computed, regardless of whether or not it was reported. There are three main effects and a DV which we'll simply note is the difference in reaction time between aggressive and non-aggressive words in a Stroop task. Main effect for movie type costs 1 df, main effect for gender costs another df, and main effect for trait aggressiveness as defined in the article costs 2 df. There are the interactions of movie type x gender (1 df), movie type x trait aggressiveness (2 df) and gender x trait aggressiveness (2 df). The three-way interaction will take another 2 df. That leaves us with 62 df. A significant F-test for simple main effects would need to be practically 4.00. The obtained F in Table 4 for Boys is well below that threshold. And yet the authors claim statistical significance for that particular subsample. The authors want to claim a significant movie type x gender interaction, but have no significant simple main effects. That does not compute, folks.
This, by the way, is one of the better articles from this lab in terms of - at least on the surface - data reporting.
One more thing to note: the authors improperly use terminology commonly used by aggression researchers incorrectly. They are not measuring aggressive attitude levels. They are measuring accessibility of aggressive cognition (or thoughts, in lay terms). I see that mistake enough times to acknowledge that it is hardly unique to Zhang's lab. It is just an annoyance, I suppose.
The article can be read here.
File under articles that probably should not be cited without adequate clarification from the authors.
Reference:
Zhang, Q. , Xiong, D. and Tian, J. (2013) Impact of media violence on aggressive attitude for adolescents. Health, 5, 2156-2161. doi: 10.4236/health.2013.512294
Saturday, May 4, 2019
Imagine (a methods thought experiment)
Imagine a university in an emerging economic power becomes a major research institution early in the decade. Not surprisingly, some researchers within that university quickly take advantage of their university's new status to further their own careers. Fair enough. Who can blame them.
Now let's imagine that one of those researchers publishes work in your area of expertise. It's a large enough university that it would hardly be a surprise. Let's go further. Imagine that you first encounter an article by said researcher. The article seems riddled with multiple errors. You get concerned, as you should, and write the author. In your email, you ask for original data and analyses. You want to be able to reproduce what this researcher claims to have found. You never get a response. You move on with your life.
A couple years later, you realize that said researcher has published multiple articles that have a similar pattern of errors. Sometimes you see identical analyses all the way down to the exact test statistics. We're talking errors in basic descriptive statistics (means and standard deviations, marginal means) as well as inferential statistics (e.g., t-tests and F-tests). These are blatant errors. They are so obvious that you cannot miss them if you do more than a cursory scan of the abstract and discussion section of each article. How are you feeling now? What do you do? Those are questions I struggle with a lot these days. I am not sure my approach was necessarily great - that is to go public - but I do know that I could not just sit still.
This is a real scenario. I personally think that a lot of people should be pissed. I think that the citizens of the country in question should be pissed. After all, their taxes paid for an extended international excursion where some of this dodgy research was presumably worked on. Their tax dollars are funding grants for said author's research even now. I think coauthors should be pissed - especially if the researcher in question did not share data with coauthors. In particular I would advise those coauthors to read about Diederik Stapel and how he went about pulling the proverbial wool over a lot of people's eyes until he eventually got caught. I don't know if this particular situation is in the same league as Stapel, nor will I make that claim (as I have no evidence to do so), but I would not be able to rule that out absent actual data sets to look at (as those would be the evidence needed to rule out foul play). If I were a fellow scientist, I would be pissed, as this author's methodology and analyses are so poorly reported that the body of this author's work could be used to potentially delegitimize a whole area of research - all it takes is a few well-informed lay people to figure out that something is not quite adding up. We have enough problems in the psychological sciences without gross incompetence and potential fraud to worry about.
If you wonder about a few of my seeming obsessions as of the last several months, you at least understand where I am coming from. I am very concerned about the legitimacy of my field first and foremost. Whether or not the author in question publishes research that is in accordance with my professional opinions is not material. For the record, in at least one case, the author was initially preaching to the proverbial choir. In other cases, the author in question had a much harder sell. I am an open-minded skeptic who can be persuaded by good arguments based on solid data and analyses. So that just goes with the territory. We are supposed to be truth seekers and truth tellers who follow the evidence wherever it may lead.
So for now, I will continue to blog about periodic articles from this particular researcher and this particular researcher's lab. Heck, in one way or another I have been discussing this particular problematic research for around three years. Why stop now? This last year I have been much more public. As of yet I see no reason to stop. Neither this blog nor my Twitter account get much in the way of traffic. I am hardly someone who would be taken "seriously" by those who are at the top of the hierarchy in my field, and in my area of expertise. I am, after all, "just an educator" at a regional university no one has heard of. I do think the truth matters, and I am hopeful that the truth will come out.
As the late journalist John Ross used to say, "la lucha sigue, y sigue, y sigue..." This struggle to get the truth out is one that will not stop. Nor will I.
Now let's imagine that one of those researchers publishes work in your area of expertise. It's a large enough university that it would hardly be a surprise. Let's go further. Imagine that you first encounter an article by said researcher. The article seems riddled with multiple errors. You get concerned, as you should, and write the author. In your email, you ask for original data and analyses. You want to be able to reproduce what this researcher claims to have found. You never get a response. You move on with your life.
A couple years later, you realize that said researcher has published multiple articles that have a similar pattern of errors. Sometimes you see identical analyses all the way down to the exact test statistics. We're talking errors in basic descriptive statistics (means and standard deviations, marginal means) as well as inferential statistics (e.g., t-tests and F-tests). These are blatant errors. They are so obvious that you cannot miss them if you do more than a cursory scan of the abstract and discussion section of each article. How are you feeling now? What do you do? Those are questions I struggle with a lot these days. I am not sure my approach was necessarily great - that is to go public - but I do know that I could not just sit still.
This is a real scenario. I personally think that a lot of people should be pissed. I think that the citizens of the country in question should be pissed. After all, their taxes paid for an extended international excursion where some of this dodgy research was presumably worked on. Their tax dollars are funding grants for said author's research even now. I think coauthors should be pissed - especially if the researcher in question did not share data with coauthors. In particular I would advise those coauthors to read about Diederik Stapel and how he went about pulling the proverbial wool over a lot of people's eyes until he eventually got caught. I don't know if this particular situation is in the same league as Stapel, nor will I make that claim (as I have no evidence to do so), but I would not be able to rule that out absent actual data sets to look at (as those would be the evidence needed to rule out foul play). If I were a fellow scientist, I would be pissed, as this author's methodology and analyses are so poorly reported that the body of this author's work could be used to potentially delegitimize a whole area of research - all it takes is a few well-informed lay people to figure out that something is not quite adding up. We have enough problems in the psychological sciences without gross incompetence and potential fraud to worry about.
If you wonder about a few of my seeming obsessions as of the last several months, you at least understand where I am coming from. I am very concerned about the legitimacy of my field first and foremost. Whether or not the author in question publishes research that is in accordance with my professional opinions is not material. For the record, in at least one case, the author was initially preaching to the proverbial choir. In other cases, the author in question had a much harder sell. I am an open-minded skeptic who can be persuaded by good arguments based on solid data and analyses. So that just goes with the territory. We are supposed to be truth seekers and truth tellers who follow the evidence wherever it may lead.
So for now, I will continue to blog about periodic articles from this particular researcher and this particular researcher's lab. Heck, in one way or another I have been discussing this particular problematic research for around three years. Why stop now? This last year I have been much more public. As of yet I see no reason to stop. Neither this blog nor my Twitter account get much in the way of traffic. I am hardly someone who would be taken "seriously" by those who are at the top of the hierarchy in my field, and in my area of expertise. I am, after all, "just an educator" at a regional university no one has heard of. I do think the truth matters, and I am hopeful that the truth will come out.
As the late journalist John Ross used to say, "la lucha sigue, y sigue, y sigue..." This struggle to get the truth out is one that will not stop. Nor will I.
Friday, May 3, 2019
A thread on authoritarianism
Above is the beginning. Worth reading the thread in its entirety. It serves as a good reminder that there is an important person-situation interaction that must be taken into account. In this case, we have to examine the sorts of issues our society's elites are constantly going on about, as their actions (as individuals, corporations, political parties, etc.) can either contain or exacerbate the authoritarian inclinations among that subset of the population that...well...has authoritarian inclinations. We live at a time in which arguably the language used by major media organs (e.g. Fox News), a major political party (GOP), and a White House occupant fan the flames of fear and loathing that pushes those who were already inclined toward authoritarianism over the edge. We've seen this play out before elsewhere historically, as well as in the current moment. I study authoritarianism primarily from the perspective as a social-personality psychologist. I only study from one angle. As Dr. Federico reminds us, the reality is considerably more nuanced.It's taken me a while to read this and formulate some comments, but ultimately my reaction is as negative as most of the others I've seen [thread]. (1/n) https://t.co/qXV9hwNMeY— Christopher Federico (@ChrisPolPsych) May 3, 2019
Wednesday, May 1, 2019
Correcting the scientific record does take a toll on the people involved
May Day was once an international labor day. In the US we tend to avoid so much as a mere acknowledgement of this date and its meaning. It is one worth remembering.
In that spirit, I would like to offer a recent blog post by James Heathers. Part of the labor involved in the sciences includes cleaning up the messes that fellow scientists leave behind when they are grossly incompetent or commit outright fraud. That labor is rarely recognized, is often demonized, and is often done after hours in relative isolation. I don't see that as a sustainable way for individual scientists to live, nor is this a sustainable model for the sciences if we wish them to remain legitimate.
This is as good a time as any to question the incentive structure that allows poor quality work and fraudulent work to slip through the cracks. This is as good a time as any to challenge the publish or perish ethos that has come to define institutions that are not even traditionally research-oriented. The problems revealed by the replication crisis in my field are international in scope and require international solutions. We need to have funding in place for those who are doing the necessary cleanup work as well as structures in place that reward openness - especially when inevitable mistakes get made. We are so far away from where we need to be. We should not see people who are genuinely trying to do the right thing burn out. They need collective support, ASAP.
A wiser man than me once said something about "having nothing to lose but your chains." I think there may be something to that - especially for those of us who toil in relative obscurity and with relative job insecurity.
In that spirit, I would like to offer a recent blog post by James Heathers. Part of the labor involved in the sciences includes cleaning up the messes that fellow scientists leave behind when they are grossly incompetent or commit outright fraud. That labor is rarely recognized, is often demonized, and is often done after hours in relative isolation. I don't see that as a sustainable way for individual scientists to live, nor is this a sustainable model for the sciences if we wish them to remain legitimate.
This is as good a time as any to question the incentive structure that allows poor quality work and fraudulent work to slip through the cracks. This is as good a time as any to challenge the publish or perish ethos that has come to define institutions that are not even traditionally research-oriented. The problems revealed by the replication crisis in my field are international in scope and require international solutions. We need to have funding in place for those who are doing the necessary cleanup work as well as structures in place that reward openness - especially when inevitable mistakes get made. We are so far away from where we need to be. We should not see people who are genuinely trying to do the right thing burn out. They need collective support, ASAP.
A wiser man than me once said something about "having nothing to lose but your chains." I think there may be something to that - especially for those of us who toil in relative obscurity and with relative job insecurity.
Subscribe to:
Posts (Atom)