Recently, I bookmarked Malte Elson's website, FlexibleMeasures.com. As of now, Elson's work involves examining the competitive reaction time task (CRTT) and the Go/No-Go Task. I am familiar with the CRTT in part because I worked in a lab that relied on the CRTT for our assessment of aggressive behavioral outcomes, and in part because it is (as Elson duly notes) arguably the most common measure of aggressive behavior in lab research. His concerns are ones that we should take seriously, and the article that he and his colleagues published four years ago is one I consider required reading.
My interest is in another increasingly common measure - this time involving assessment of accessibility of aggressive cognition. Although there are plenty of methods available (each, I suspect with its own benefits and shortcomings), I am noticing more and more a reliance on variations of a word completion task WCT). The idea is simple enough. Participants are presented with a series of word fragments and are required to complete the word fragments to form words. Some of those words may be aggressive and some may be non-aggressive. For example KI__ could be completed as KICK or KILL (aggressive) or KISS or KIND (non-aggressive).
There are at least two versions of the WCT of which I am aware. One was developed by Craig Anderson and was first used in the late 1990s, and the first published results using Anderson's WCT appeared in 2003 and 2004 (note that I was a coauthor of the latter article, and I utilized the WCT in one of the experiments). Anderson's WCT contains 98 items, only half of which allow for the opportunity to complete the fragments as aggressive. Anderson usually recommends using the proportion of aggressive words completed (aggressive words/total words completed) as the measure of accessibility of aggressive thoughts. More recently, Brad Bushman developed a shorter version of the WCT that includes only 22 items, each of which can be completed to form aggressive or non-aggressive words.
Please make sure to read through Anderson's description of the task as he developed it. The WCT is flexible to the extent that it can be administered in such a way that it is timed or untimed, and that one could use either the proportion of aggressive words completed or the total number of words completed as the measure of accessibility of aggressive thoughts. Although Bushman, as far as I am aware, provides no guidance on how to administer or score the version of the WCT that he developed, I would wager that it is potentially just as flexible as far as administration and scoring. Obviously the concerns Elson might voice regarding the CRTT are ones I might now voice regarding the WCT. In addition, we have very little evidence of the reliability of the WCT. I am currently in the process of cataloging all studies in which either Anderson's or Bushman's WCT was used to measure accessibility of aggressive cognition. I routinely examine each article for any psychometric information, such as internal consistency. I found, so far, one published article in which reliability data for a version of Anderson's WCT is reported. Unfortunately, the coefficient alpha is only for a 63-item version of the WCT, rather than the full 98-item WCT. The alpha appears to be acceptable (.81), but there did not appear to be much of a rationale for why a shorter version was used. I am having to assume that since the study was based on adolescents the number of items was limited. I hate assuming. Thus far I have not found any other reliability data. However, my search is only in its infancy. As far as Bushman's, thus far nothing regarding its psychometric properties is available - at least in print. The only information I have is from an unpublished data set, in which an experiment did not appear to work. I examined the WCT's internal consistency as a means of understanding what might have gone wrong, and was a bit taken aback by my discovery that the coefficient alpha was only .53 (regrettably, as that data set is not my own, I am precluded from publishing a full research report - but I do think that potential users of Bushman's 22 item WCT should have some fair warning). That could be a fluke. However, I just do not know, and that bugs me.
As of now, I see two problems: 1) the WCT is apparently administered and scored "flexibly" (i.e., inconsistently) and 2) we don't have a good handle on the reliability of either instrument. Hence, I am concerned about the validity of the WCT as a measure of aggressive cognition. Hopefully, my efforts to catalog the available research will enable me to at least understand the contexts in which the WCT is used and some idea of its psychometric properties. I am hoping to do some new research examining the reliability of both versions of the WCT in order to gain a better idea of the psychometric soundness of each instrument, and if needed begin the process of developing a more sound version of the WCT than currently exists.
My motivation here is hopefully fairly obvious. I like the idea of a WCT, as it has the potential to enable researchers to conduct cognitive research in traditional lab settings without the necessity of expensive software, as well as potentially useful in ecological valid contexts or in applied contexts. I need to know that when I use either instrument, that it is one that will yield valid results (i.e., true, regardless of whether or not we can reject the null). At the moment, I am a friendly skeptic. I would like to be confident that either or both of these instruments work as intended, but I am not so sure currently.
The usual conflict of interest statement: I have coauthored articles with both Anderson and Bushman at various points in my career. This post is not intended as an expression of either good or ill-will toward either of these individuals. I respect their work. That said, I do believe that truth comes before personal loyalties, and that includes the truth about instruments that these individuals have developed.
No comments:
Post a Comment