Tuesday, October 5, 2021

Converting Standard Error to Standard Deviation

This is just one of my brief posts that I hope is obvious to most folks who would bother to read this blog. It's just a technique that is part of my meta-analyst's toolbox, can be calculated on the back of a napkin, and comes in handy when authors report means and standard error instead of the usual standard deviation term we would want to estimate Cohen's d. The formula is simple.

I've used it a bit more often than I would have imagined. It comes in handy. One common mistake meta-analysts can make is to erroneously use the SE term instead of SD in the denominator to compute Cohen's d, which would inflate the effect size for the hypothesis test in question. Needless to say, you want to avoid that.

Credit to Cochrane Handbook for Systematic Reviews of Interventions.

Monday, October 4, 2021

A grim day for another weapons effect paper

Sometimes a specific lab becomes the gift that keeps on giving. If the work is good, we are the better for it. If the work is questionable, our science becomes less trustworthy not only to the public, but to those of us who serve as educators and fellow researchers. As is true in other facets of life, there are gifts we would really rather return. 

Which brings us to a certain researcher from Southwest University: Qian Zhang. In spite of several recent retractions, I have to give the man credit. He remains prolific. A recent paper was recently uploaded on a preprint server, on a variation of weapons effect research that is quite well known to me. The author was even kind enough to upload the data and the analyses at osf.io, which is to be commended. 

That said, there are clearly some concerns with this paper. I will only discuss a few in this post. My hope is that others who are far more facile at error detection and have enough fluency in Mandarin can pick up where I will likely leave off. The basic premise of the paper is to examine if playing with weapon toys will lead children to show more accessibility of aggressive cognition (or think more aggressively, if that is easier on the eyes and tongue) and show higher levels of aggression on the Competitive Reaction Time Task (CRTT). As an aside, I seem to have some difficulty with the acronym, CRTT, and often misspell it when I tweet about the task. But I digress. These sorts of experiments have been run primarily in North America (specifically, the US) and Europe, and not so much outside of those limited geographical regions. Research of this sort outside of the US and Europe could be potentially useful if done well. Usually, experiments of this sort are done to examine only behavioral outcomes (Mendoza's 1972 dissertation is arguably an exception, if we code the variation of a TAT as a cognitive measure), so the idea of also examining cognitive outcomes could be potentially beneficial. 

As I read through the paper, I noticed that there were 104 participants in total. The author contends that he used an a priori power test to determine sample size at 95% power, using G-Power 3.1. That caught my attention. I dusted off my meta-analysis (Benjamin et al., 2018) and looked at effect size estimates for various distributions that we were interested in examining at the time. One of those distributions specifically included studies in which toy weapons were used as primes. The naive effect size is not exactly overwhelming: d = 0.32. That is arguably a generous estimate, once we include various techniques for measuring the impact of publication bias, and a good-faith argument can easily be made that the true effect size for this particular type of prime is close to negligible. But let's ignore that detail for a minute. Let's pretend we exist in a universe in which the naive effect size of d = 0.32 is correct. The authors argue that an N of 52 would suffice, but that their "sample size of (N=102 [sic])" was more than sufficient to meet 95% power. If you ever run any study in G-Power, you have to choose your analysis, enter the info required, and you are given a sample size estimate. One complication with G-Power is that it never directly allows us to enter an effect size for Cohen's d. It does give us Cohen's F. Computing Cohen's d from Cohen's F is quite easy: d = 2*F, and F = d/2. So, if I know that the effect size for my research question of interest is d = 0.32, I divide by 2 and can plug that into G-Power for Cohen's F, and then make sure I have my other info correct, including number of conditions, covariates, etc. When I do all that, based on the experiment as described, with a Cohen's F of 0.16, it becomes clear that the experiment would require a sample of at least 510 students. Now let's say that the author merely made a mistake and plugged in the number for Cohen's d by accident. The sample would still have to be about 129 in order to meet the requirements of 95% power, and really given the intention to randomly divide an equal number of males and females into treatment and control conditions, the author should shoot for 132 students. In order for the argument of 95% power to be met in this study, we'd have to assume a Cohen's d of approximately 1.00. There may be individual studies in the literature that would yield such a Cohen's d, but of the available sample of studies? Not so much. So, we have another low power experiment in the research literature. It's hardly the end of the world. 

What grabbed my attention was the research protocols described in the experiment. For the time being, I will take the author at his word that this was an experiment in which random assignment was involved (this author has once been flagged for failing to disclose that participants chose which treatment condition they were involved in, which was, shall we say, a wee bit embarrassing). The way the treatment and control conditions are described seems standard enough. What was odd was what happened after the play session ended. The children were first given a semantic classification task. I admit that I've had to do a double take on this, as some of the wording is a bit off. I am increasingly thinking that what the authors did was use a series of aggressive and neutral pictures and had children respond to them as quickly as they could. The author had made some mention of aggressive and neutral pictures also being used as cues, which threw me, because that would have seemed more like an experiment within an experiment. At minimum, there would have been needless contamination. Then the children participated in a CRTT where they set noise blasts at 70 to 100 db. Those controls were set from 0 (no noise) to 4 (100 db). The authors reported their means and standard deviations. I then initially looked at the means for treatment and control condition using GRIM, which is a nifty online tool for flagging errors. The results were, to say the least, initially looked grim. However, I was reminded that there is the issue of granularity that I might have overlooked. So, even though there is one scale, the trials each count as independent items. So, an N=26 for one cell is, with the 13 out of 25 trials that the author included in the data set (in which participants had an opportunity to send noise blasts after a loss), effectively an N=338. So I went and opened up the SPRITE test link and entered the same mean info, along with the minimum and maximum scale values (0 and 4, respectively), the target mean for each cell I was interested in, and SPRITE would report that each of the two cells measuring boys failed to arrive at a solution for at minimum the standard deviation. In each case, the standard deviation was reported by SPRITE to be too low. I can get reproductions of possible distributions for the other two cells. I then downloaded the data set to see what it looked like. Much of it is in Mandarin, but I can make some educated guesses about the data in each column. I turned my attention to the "ANCOVA" analysis. It actually looked like a MANOVA was run. Perhaps a MANCOVA (but as I am admittedly not literate in Mandarin, it's hard to really know without taking time I don't have yet to put some terms into Google Translate and sort that all out). That's a project for later in the month. I could see the overall mean for the aggressive behavioral outcome, as measured by the CRTT and entered it, its standard deviation, and overall N in SPRITE and noticed it also could generate some potential score distributions. Still, given the failure to generate some potential distributions in SPRITE, it's not a good day to have posted a preprint. At minimum, there is some sort of error in reporting, whatever the cause.

I do need to take some time to sit down, try to reproduce the analyses, etc. Of course, it goes without saying that successfully reproducing a data set that has been in some way fabricated is going to add no new information. That said, I am satisfied that the findings as presented for the CRTT analyses intended to establish that weapon toys could (at least most specifically for the male subsample) influence aggressive behavioral outcomes may also be potentially questionable. This is a paper that should not make it past peer review in its present form. 

Note that I have not yet run this through Statcheck, although in recent years Zhang's lab has become more savvy about avoiding obvious decision errors. I made an effort as of this writing to run the analyses as they appeared in the pdf, and the report came back with nothing to be analyzed. I will likely have to enter the analyses by hand on a word document and then reupload at a later date. 

Please also note that the author appears not to have counterbalanced the SCT and CRTT measures to control for order effects. That strikes me as odd. The very superficial discussion about debriefing left me with a few questions as well. 

Note: Updated to reflect some more refined analyses. Any initial mistakes with GRIM are my own. I am on solid ground with the SPRITE runs, and I think my own concerns about the lack of statistical power, failure to counterbalance, etc. are on solid ground.

Tuesday, September 28, 2021

One of my pet peeves when it comes to cognitive priming tasks in aggression

I'm probably going to come across as the late Andy Rooney for a moment. You know what I hate? Some of the apparent flexibility in how cognitive priming tasks get measured in my specialty area: aggression. I've spent some time with these sorts of tasks during my days as a doctoral student in Mizzou's aggression lab in late 1990s. The idea is fairly simple. We prime participants (typical traditional first-year college students) with stimuli that are either thought to be aggression-inducing stimuli (e.g., violent content in video games, images of weapons, violent lyrical content, etc.) or neutral stimuli (e.g., non-violent content in video games, images of non-weapons such as flowers, nonviolent lyrical content, etc.), and then get reaction times to aggressive and non-aggressive words. I'm probably most familiar with the pronunciation task, in which participants see, for example, an image for a few seconds (weapon or neutral object), followed by a target word that participants read aloud into a microphone, and the latency is recorded in milliseconds. The lexical decision task is similar, except in addition to reacting to aggressive or neutral words, the participants also must decide on whether or not what they are seeing is a word or non-word. At the end of the day, we get reaction time data, and look for latency measured in milliseconds.

For a prime to work, we expect that the relative latency for aggressive words will be lower than for neutral words in the treatment condition when compared to the relative latency for aggressive versus neutral words in the control condition. That's the pattern we found in both experiments in Anderson et al. (1998) and Lindsay and Anderson (2000), for example. The difference in latency between aggressive words and non-aggressive words was significantly larger, and in the predicted direction, in the weapon condition than was the case between aggressive words and non-aggressive words in the neutral prime condition. We could conclude that weapons appeared to prime the relative accessibility of aggressive cognition, or we could say aggression-related schemata or whatever nomenclature you might prefer. 

The way I was trained, and the literature I tended to read worked largely the way I just described, regardless the stimuli used for primes and regardless of the target words or concepts the experimenters were attempting to prime. In our case, comparing the relative difference in reaction time latencies between responses to aggressive and non-aggressive words gave us a basis for comparison across treatment condition, and took into account some of the noise we would likely get in the data, such as individual differences in reaction time speed. 

Lately I have seen in my corner of the research universe papers published in which the authors only publish reaction time latencies for aggressive words, even though they admit in their published reports that they did have reaction time data for non-aggressive words. They appear to be getting statistically significant findings, but I find my self asking myself a question: so you find that participants respond faster to aggressive words in the treatment condition than in the control condition. That's nice, but what do those reaction time findings for aggressive words alone really tell us about the priming of the relative accessibility of aggressive cognition? In more lay terms, you say you found participants respond faster to aggressive words, but compared to what? I have also seen the occasional paper slip through in which the authors attempt to have it both ways. They'll use raw aggressive word reaction times as their basis for establishing that there is a priming effect, but their other hypothesis tests actually do use what I see as a proper difference score between aggressive and non-aggressive words. Oddly enough, in one presumably soon-to-be retracted number, when the authors use the approach I was taught, the effect size for the treatment condition becomes negligible, and the authors have to rely on subsample analyses in order to make some statement about the treatment condition actually priming the relative accessibility of aggressive cognition. Now, when I see subsequent research where only the reaction times for aggressive words are reported, I wonder if what I am reading is to be trusted, or if something is being hidden from those of us relying on the accuracy of those reports. 

That is the sort of thing that can keep me awake at night.

Tuesday, September 21, 2021

The jamovi MAJOR module

I've been switching over most of my data analyses to jamovi (as a very tentative step toward learning R), as well as switching my instruction to jamovi. Overall, I love the interface, and will probably say more about it later. For now I just want to say a few brief words about the MAJOR module, which is meant to interface with Metafor, which is a meta-analytic package for R. MAJOR is very intuitive, and I've found it relatively easy so far to reproduce basic analyses from distributions from prior meta-analysis I've worked on. It produces helpful forest plots and funnel plots. It meets my basic needs. When it comes to publication bias, I wish there were more options available. As of now, MAJOR offers Fail-safe N (and my advice has been that friends do not let friends use Fail-safe N) and Egger's test to detect potential publication bias. I hope the developers of MAJOR plan on adding on more publication bias options, even if just trim-and-fill analyses (fixed and random). That said, meta-analysis has moved way beyond any of the above techniques, and I'd love to see other publication bias techniques included, (PET-PEESE comes to mind). That would be helpful. Otherwise, I am quite happy with what I've been able to do so far. Kudos to the developers of MAJOR for what they have done so far.

Friday, August 13, 2021

It's been quite a week

On August 6th, Tom Mars' temporary injunction against State Act 1002 of 2021 was a tentative success. The parents who sued the state with his representation were able to convince a judge to order a temporary injunction against the law, thus preventing its enactment for the time being. The law would have prevented local and state run agencies and governments from enforcing mask mandates. The law was signed by our Governor, Asa Hutchinson, last May. It seemed like the legislature and the Governor spiked the proverbial football at the time. Cases of COVID-19 had gone down and would continue to go down for a few more weeks. I thought the law was stupid at the time. It coincided with other laws which restricted any of my state's Governors from enacting common-sense state of emergency mandates. The optics of the Governor actually signing State Act 1002 of 2021 would look bad no matter what. Given the spike in cases that has led to a record number of children in ICUs and ventilators during yet another wave of COVID-19 cases, the whole thing smells of hubris. 

There was no good news or relief for those few school districts that had started school right around the end of July or the start of August. But once the temporary injunction was filed, the dominoes started to fall. A number of school districts across the state, including some that are considered to be "conservative" (whatever that might mean anymore) enacted mask mandates. My city's school district was one of those. University systems across the state enacted mask mandates over the course of this week. I think the city of Little Rock mandated masks on public indoor property. Private corporations with footprints in my state have not only enacted mask mandates of their own, but required their employees to show evidence of vaccination as a condition of employment. A standard of conservative governance in the past had been that policy decisions that had been mandated by Federal or state officials should be handed off to localities and private enterprise. Something has changed in the last few years. The conservative orthodoxy that I understood and even respected, albeit grudgingly, has been supplanted with something different. When it comes to public health and safety, apparently now the new orthodoxy is that the state can actively prevent local governments and state agencies, as well as private sector businesses from doing what is necessary to protect their employees, customers, etc. 

I have no doubt that our current state's Attorney General (Leslie Rutledge) will appeal the injunction against Act 1002. After all, she does have some political aspirations, including a run for the office of Governor. I am actually surprised she has waited as long as she has. I am no legal eagle, but based on how I understood the temporary injunction ruling, assuming the state's Supreme Court is even remotely functional, the law is dead in the water. We'll wait and see. Localities not wanting blood on their hands are not waiting. Nor are those in the private sector, even as our legislature plots to treat the corporations that employ the vast majority of our residents as ones that are only able to act under the whims of a command economy. We're not late-period USSR, or at least we should not be.

From what I've seen with parents on FB groups is that they're voting with their feet if they have the luxury of doing so (usually those with some financial means, which results in those who are relatively upper middle class). School districts that seem to be promoting health and safety my see an increase in students, and an increase to their funding. Colleges and universities that demonstrate a willingness to act in the interest of the health and well-being of their students will hopefully be rewarded for their efforts, regardless of how this current legal battle plays out. At least students and where relevant their parents will recall the systems who were willing to stand and be counted. 

In the meantime, we're recording consistently around 3k COVID-19 cases per day in my state, and the hospitalization rate and ventilator rate are both off the charts. Our vaccination rates have started to bump up a bit. I honestly don't know how much of that is due to the proverbial "fear of God" being put into some very reluctant folks given our current dire situation, and how much of that is due to private corporations and business just outright mandating vaccinations. 

Regardless, I am hoping that one of the lessons we learn from this debacle is that politicizing any efforts to mitigate a pandemic is simply an awful idea and should be avoided at all costs.

We got lucky, for now

Early last week, a lawyer, Tom Mars, filed a lawsuit in a state court to seek at least a temporary injunction against the Arkansas state legislature's efforts to forbid local state agencies (such as school boards) from independently mandating masks or facial coverings as a means of mitigating the spread of the Delta variant of COVID-19. The temporary injunction succeeded. I had expected our state's Attorney General to mount an immediate appeal to the state's Supreme Court. So far, that has not been forthcoming. I am under the impression that the Governor has sought outside legal counsel in order to make the next move. If the external counsel is smart, the state-level leaders (if we can call them that) have been informed that the nature of the temporary injunction is one that suggests that the Tom Mars suit will succeed on its merits. We'll have to wait and see.There is reason for optimism.

Not every state agency will issue a mask mandate, even as this latest wave of the pandemic ravages our state. But many will. My city's school district acted quickly and on Monday mandated masks for all who are faculty, staff or students in situations in which there are two or more people inside a building. This is pretty consistent with what was policy last academic year. That's as good as we could have expected. That mandate will be reviewed in late September. Depending on how things look, the mandate may be extended or it may be allowed to expire. My best guess, based upon projections by our own state health experts, is that we'll still want a mask mandate for yet another 60 days. I was glad to see that most of the state's university systems chose to put mask mandates in place. That was wise, given that I suspect that if the decision was placed in the hands of individual faculty senate bodies, there would have been gridlock, given that faculty themselves are polarized (which I realize is counter-intuitive to the usual narratives concerning faculty ideology).

The bottom line is that the vast majority of us are getting what we want and need. There are very understandably skittish students and parents (I and my youngest daughter among them) who needed assurances that students once more could have in-person events in as safe an environment as possible. We're probably a silent majority in our state and locality. Faculty and staff get assurances that they and their families will be kept as safe as is possible at the K-12, college, and university levels. For that, I am grateful. Ideally, we'd have some mandate on our university and college campuses regarding COVID-19 vaccination. Suggesting such an action is politically incorrect in this environment. I was never good at following party lines, and that is especially true with our current ruling party in my state. As it stands, our state legislators (at least those from the GOP, which is the majority party), are even trying to interfere in the decisions private businesses can make regarding vaccination mandates for employees, which seems maddening, given how much the GOP once fetishized all things private sector. 

This is just my opinion, and one should take it for what it is. I am convinced that an uncontrolled pandemic is bad for private businesses, for public universities and colleges, and for public schools.  Refusing to do even the minimum needed to keep students safe? Parents will look for, and pay for, other options. As someone who relies on private businesses for survival? I will patronize those businesses that do what is necessary to mitigate spread. A restaurant that offers an indoor dining experience in which there is physical distancing and a mask requirement for customers, and where employees are required to be vaccinated? They have my business, regardless the cost. That's a promise. I am sticking with our current medical practitioners largely because the larger system took this pandemic so seriously that all employees from doctors to CNAs have to be vaccinated. Our other system in my area seems to be fairly non-seriousness of the current threat. They've seen the last of us for a while. Part of living in a capitalist society is that we do get some choices, and those businesses and agencies that do due diligence to keep us safe will earn future business. Those who refuse will eventually lose out. States and regions refusing to protect their residents in the name of public safety will lose out. I wish corporations with some footprint in these states would do a better job of educating state legislators (including reminding them that their donations have been based on doing the bidding of these corporations), and that various levels of chambers of commerce would reach out and educate local political officials. I also wish that those same chambers of commerce would make it very clear that those unwilling to protect public heath, and hence protect the interests of the private sector, would be out of a political career as of November 2022. That's quite an ask. I can hope.

In the meantime, I have some tools now to keep my students and my family safe that I was not expecting even when I last posted. There is reason to hope. We got lucky, or so it appears. An angry group of parents of kids who have as of yet no access to a COVID-19 vaccine, who have immuno-compromized family, etc., we've had some tentative victories. Let's not get complacent. We've learned that our legislature is only willing to follow our system of economics and governance when it is convenient to our state legislature's party line (this should seem very similar to what those operating under the USSR during its period of collapse experienced). "Good" party members benefited for a while as those who were not suffered. Once we devolve to that point, it's no longer clear if there is even a social order. I don't follow party lines of any sort. So here we are. Benefit students. Accept that businesses who rely on public trust (all businesses, as it turns out) have what they need to build trust during what we hope is a once in a century pandemic.This is not rocket science, by any stretch.

Tuesday, August 3, 2021

A bit of a postscript to the preceding

Some other observations:

There are times when I wish I had more legal expertise. The reason I say that is because I suspect we're not that many student (or faculty and staff) hospitalizations and deaths at institutions that either willingly refuse to mandate vaccines or masks during this particularly virulent phase of the pandemic (with an R0 of about 8 or 9, the Delta variant is already one of the most infectious viruses known to humanity), or whose hands were tied by state legislators and/or governors from someone attempting to file a lawsuit. Whether or not something like that can succeed is another matter, but I would not be surprised if some angry family members seek a legal remedy, under those tragic circumstances. Early on in the pandemic, we were essentially flying blind. At this juncture, we can see very clearly what is coming our way. Someone will be held accountable eventually for preventable risk.

 On the topic of risk, I would expect insurance premiums to go up. Insurers are going to price in risk as long as this pandemic continues, and try to recoup some of their lost profit margins from paying considerable portions of the bills of those who ended up hospitalized with COVID-19. I'm sure colleges and universities have some sort of insurance that they pay to shield them, and those premiums will go up. Smart insurers are going to jack up their rates in areas where vaccination rates are low and infections rates continue to remain high. 

There is stereotype about faculty being practically a hive mind. Those believing that have never been to a faculty committee meeting. I suspect that the divisions regarding how to deal with the pandemic (or simply to refuse to deal with the pandemic) are now very noticeable on college and university campuses. I will only provide an anecdote, but I think it will illustrate what I have in mind. I just rolled off a committee that among other things was tasked with providing some faculty-led guidance on how to find solutions to ease some of the suffering of our students. Although in ordinary years, I would cringe at solutions such as retroactive withdrawals or Pass/Fail options for students to select once they see their grade reports, These last three semesters have been very far from ordinary. Our campus offered those options for students for Spring and Summer 2020. Once the Fall semester was well underway, and it became obvious that a new wave of cases was building up, students approached us to offer these solutions once more. There was a lot of pushback from that committee, and in the end, we'd come up a couple votes shy of what the student leadership was telling us the students needed. Some of that pushback was expected - it amounted to "I walked uphill in the snow in 100 degree heat each way and I did just fine. These students can too." But some that pushback was really ideological and seemed to parrot talking points I've seen on Facebook groups aligned with Trump. I noticed that ideological pattern of pushback emerge on all matters concerning this pandemic as time went on. I'd already decided to roll off that particular committee simply because it was a significant time commitment, and I'd devoted about half of my career at my current university to serving on that committee. The toxicity that has emerged sealed it for me. When it comes to solutions that are pandemic related, this is a committee that might be able to produce half-measures or fail to act at all.* The ideologues are a minority but very vocal, and smart enough to sort out how to use parliamentary procedure to their advantage. All of that is to say that when it comes to things like revisiting mask mandates, if a lawsuit succeeds in my state to strike down a law that prevents our university from doing so, that decision about whether or not to mandate masks indoors will need to be made by senior administration. There is too much division among faculty. It's probably a safe assumption that the same applies among students as well. And it's probably not just my own campus. 

Once basic public health and safety measures got politicized early on during the pandemic, any hope for the public to more or less band together (the usual small groups of outliers notwithstanding) and use some good horse sense went away. Of course that politicization would spill over onto college and university campuses. After all, our campuses are merely microcosms of our own aching society. Regrettably, I can envision an era in which there are going to be sets of faculty on any given campus who simply cannot and will not work together. Whatever goodwill that once existed has vanished. That may be gone for a generation or two.

My professional opinion as an educator is that many in-person activities from instruction to recreation are potentially safe enough if those who make up the campus community are required to be vaccinated (medical and religious exceptions a given) and wear masks for now, at least indoors and in crowded outdoor environments. That may not quite be the normal we wanted, but it is what we could potentially achieve for the next couple semesters (for those of us on the semester system - folks on the quarter system can sort out their own math). I'd also recommend college and university systems offering generous leave for faculty and staff who either acquire COVID-19 (probably breakthrough cases if fully vaccinated) or must care for family who are infected, and generous absentee policies for students who test positive for COVID-19. We probably need at least retroactive W or Passing grade policies in place for a bit longer, with the understanding that once we are actually post-pandemic, those options are no longer available. Let's just say that one individual decision I've made is that if students feel sick, I want them to just stay home. I can get them back up to speed later. My stats assignments can be turned in using either SPSS (which my institution insists on using, much to my disappointment) or Jamovi. The latter can be used off campus with no problems. Any other workshop type activities I offer in my hybrid classes can be easily made up. Most, if not all, are already online. I could handle instructions for most basic assignments in an email in most cases. I could handle something in a Zoom office hour or appointment just as easily. Part of this social contract is that if I am feeling sick, I will also stay home. I have technology available that will allow me to do my job remotely, and will take advantage of that if needed. It's not exactly rocket science. COVID-19 is not through with us yet. We have to do what we can to keep each other safe and healthy, using good judgment and the guidance of the scientists who've done the lion's share of the work to figure out what is safe and what is not. Should be simple, right?

*Note - on nearly other matter within the scope of this committee, a lot of very good work got done this past academic year. Its leadership team were great at herding cats, and I can only imagine the migraines the rest of us caused them.