A few months ago, I made note of concerns appearing on PubPeer about a cyber-bullying article. The good news is that the lead author did offer a revised table to make the information more readable. The bad news is there are some problems with that table that still have not been discussed.
As you will notice, this updated table is considerably more readable, so kudos to the authors for that. However, there are still some lingering concerns. The good news is that the reader knows specifically the breakdown between the number of participants who noticed and who did not notice the bullying. It was what I had sussed out from reading the report in sufficient detail, which was comforting. The test statistics, however, are still a bit odd. Now with a total sample of 221 (N=221, for those who need that spelled out), I would have expected a different number for degrees of freedom (df) for these t-test statistics. Total degrees of freedom can be computed on the back of a napkin: N-2. So if there were 221 total participants, N-2=219, not 217. So I do have some questions there. My charitable take is that this revised table was hastily drawn up and a simple typo occurred. It happens to all of us. It is not entirely clear what the confidence intervals refer to. If those are associated with a Cohen's d statistic, why not report the Cohen's d as one would ordinarily do? The Chi-Square statistic is presumably statistically significant, so why not include at the bottom of the table that * denotes p <.05? The t-tests in the table are reported as positive, but in the corresponding paragraph the same t-values are reported as negative. If the authors have reason to believe this does not matter, they need to explain why it doesn't matter. I'd certainly be intrigued to read the argument that would be needed to justify such a perspective.
Since there had been concerns expressed earlier about some inconsistencies in the reporting of means and standard deviations in the original published article, and since there appeared to be at least one effort to correct an incorrectly reported mean in the updated table, I decided to do a bit of a deeper dive, using SPRITE as a way of exploring whether or not the means and standard deviations would be mathematically possible. Please keep in mind this very important disclaimer: the findings I am reporting should not be taken as the gospel. I am merely noting a couple areas of concern that I would strongly advise the authors, the journal editor, and of course any data sleuths to follow up on posthaste. I am also making an assumption I feel uncomfortable making: that the means and standard deviations are based on single items in which the data are integer data. The latter I am confident about, as the test items involved are Likert-scale items. The former is merely an assumption as the research report does not make the number of items for each DV explicit. If I am wrong, then I am prepared to admit it. With that in mind, let's dive in, shall we?
What you will see below are SPRITE results for each cell mean and standard deviation as reported in the above table. Let's see what I discovered.
Finally, this is for the cell Not Noticed/Bully. The reported mean is 2.46, SD = 1.58. Again, Sprite's analysis is that the target SD is too high, and that the maximum SD = 1.50.
The standard deviation for two of the four cells is simply not possible as reported in the paper. That is discomforting for any meta-analyst who might wish to extract effect sizes using reported mean and standard deviation data. One could potentially get around that with reported t-tests, I suppose, if we had any confidence on which reported df were true, and if we knew the intended sign for each t-test.
Keep in mind that the Sprite analyses are only preliminary. Don't take these as the last word. I am basing the Sprite analyses on the assumption that each scale used as a DV is a single item scale, which is reasonable enough absent any other information about the measures used for the DVs. If there are multiple items for either of the DV measures, I will need to re-run the Sprite analyses accordingly. I am also basing the Sprite runs on the reported sample sizes for each scale. If those were reported incorrectly, then these analyses are effectively moot. Those who possess more fine-grained skills than I could likely confirm my work or find more concerns that I am missing. We do the best we can within our limits. The purpose of this post is simply to ask questions, in the hope that we get to the truth, or at least the closest approximation thereof.
No comments:
Post a Comment