The Sublimated Grief of the Left Behind
I follow quite a number of scholars on Twitter. Periodically I see posts of what falls under the broad umbrella of quit lit retweeted. This post is a bit different, and I hope that her perspective offers some much needed food for thought. As someone who has experienced the loss of talented colleagues due to the circumstances the above author faces, this is a post that hit close enough to home to bear mentioning.
Sunday, February 11, 2018
Friday, February 2, 2018
Never treat a meta-analysis as the last word
I mentioned earlier that any individual meta-analysis should never be treated as the last word. Rather, it is best to treat a meta-analytic study as a tentative assessment of the state of a particular research literature at that particular moment. One obvious reason for my stance simply comes down to one of the available sample of studies testing a particular hypothesis at any given time. Presumably, over time, more studies that attempt to replicate the hypothesis test in question will be conducted and ideally reported. In addition, search engines are much better at detecting unpublished studies (what one of my mentors referred to as the "fugitive literature") than they once were. That's partially due to technological advances and partially due to individuals making their unpublished work (especially null findings) available for public consumption to a greater degree. To the extent that is the case, we would want to see periodic updated meta-analyses to account for these newer studies.
The second obvious reason is that meta-analysis itself is evolving. The techniques for synthesizing studies addressing a particular hypothesis are much more sophisticated than when I began my graduate studies, and are bound to continue to become more sophisticated going forward. The techniques for estimating mean effect sizes are more sophisticated, as are the techniques for estimating the impact of publication bias and outlier effects. If anything, recent meta-analyses are alerting us to what should have been obvious a long time ago: we have a real file drawer problem, and the failure to publish null findings or findings that are considered no longer "interesting" is leading us to have a more rose-colored view of our various research literatures than is warranted. Having said that, it is also very obvious that since we cannot quite agree among ourselves as to what publication bias analyses are adequate, and these techniques themselves can potentially yield divergent estimates of publication bias, it is best to use some battery of publication bias effect estimation techniques for the time being.
Finally, there is the nagging concern I have that once a meta-analysis gets published, if it is treated as the last word, future research pertaining to that particular research question has the potential to effectively cease. Yes, some isolated investigators will continue to conduct research, but with much less hope of their work being given its due than it might have otherwise. I suspect that we can look at research areas where a meta-analysis has indeed become the proverbial "last word" and find evidence that is exactly what transpired.Given reasons one and two above, that would be concerning, to say the least. There is at least one research literature with which I am intimately familiar where I suspect one very important facet of that literature effectively halted after what became a classic meta-analysis was published. At some point in the near future, I will turn to that research literature.
The second obvious reason is that meta-analysis itself is evolving. The techniques for synthesizing studies addressing a particular hypothesis are much more sophisticated than when I began my graduate studies, and are bound to continue to become more sophisticated going forward. The techniques for estimating mean effect sizes are more sophisticated, as are the techniques for estimating the impact of publication bias and outlier effects. If anything, recent meta-analyses are alerting us to what should have been obvious a long time ago: we have a real file drawer problem, and the failure to publish null findings or findings that are considered no longer "interesting" is leading us to have a more rose-colored view of our various research literatures than is warranted. Having said that, it is also very obvious that since we cannot quite agree among ourselves as to what publication bias analyses are adequate, and these techniques themselves can potentially yield divergent estimates of publication bias, it is best to use some battery of publication bias effect estimation techniques for the time being.
Finally, there is the nagging concern I have that once a meta-analysis gets published, if it is treated as the last word, future research pertaining to that particular research question has the potential to effectively cease. Yes, some isolated investigators will continue to conduct research, but with much less hope of their work being given its due than it might have otherwise. I suspect that we can look at research areas where a meta-analysis has indeed become the proverbial "last word" and find evidence that is exactly what transpired.Given reasons one and two above, that would be concerning, to say the least. There is at least one research literature with which I am intimately familiar where I suspect one very important facet of that literature effectively halted after what became a classic meta-analysis was published. At some point in the near future, I will turn to that research literature.
Monday, January 29, 2018
Cross-validating in meta-analysis
I thought I'd share a couple techniques I've picked up on that are useful for cross-validation purposes. Keep in mind that the sorts of meta-analyses I am interested in involve experimental designs, and so what I will offer may or may not work for your particular purposes.
If you are estimating Cohen's d from between-subjects designs, the following formula for estimating N is recommended:
N=8+d22v.
Here you simply need to know your estimate of d and the variance (v) for a particular comparison. If you are able to estimate N from the above formula reasonably accurately, you can be confident that your estimate is in the ballpark. Note that this formula works best when your treatment and control group have equal sample sizes. Unequal sample sizes will not yield accurate estimates of N.
The above formula will not work with within-subjects designs. The formula that I know does work for within-subjects designs is the following:
n=2(1−r)v+d22v.
Note that the above formula assumes you will know the exact correlation (r) between your variables, which may or may not be reported or available. I have found that under those circumstances, if I assume r = .5000, that I typically get accurate enough estimates of N from my calculations of d and variance (v). That said, for those in the process of conducting a meta-analysis, I recommend contacting the original authors or principle investigators under circumstances where all you might have to go on is a paired-sample t-test and a sample size (and potentially a p-value). Often, the authors are more than happy to provide the info you want or need either in the form of actual estimates of r for each comparison that they computed, or better yet provide the original data set and enough info so you can do so yourself. That's easy with newer studies. Good luck if the research was published much earlier than this decade - though even then I have been amazed at how helpful authors will try to be. For those cross-validating a meta-analyst's database, if the original correlational info is available, ideally it will be recorded in the database itself for within-subjects comparisons. If not, email the meta-analyst. Again, we should be able to provide that info easily enough.
If you embark on a meta-analysis, keep in mind that others who eventually want to see your data will try to cross-validate your effect size estimates. Get ahead of that situation and do so from the get-go on your own. You'll know that you can trust your calculations of effect size and you will be able to successfully address concerns about those computations as they arise later. Ultimately that's the bottom line: you need to know that you can trust the process of how your effect size calculations are being computed, regardless of whether you are using a proprietary software package like CMA or open access language like R, and regardless of how seasoned you are as a meta-analyst. If you find problems cross-validating, then you can go back and check your code for possible errors. That'll undoubtedly save some heartache and heartburn, but the more important thing is that you can be confident that what you ultimately present to your particular audience is the closest approximation to the truth possible. Ultimately, that is all that matters. Hopefully the above is helpful to someone.
If you are estimating Cohen's d from between-subjects designs, the following formula for estimating N is recommended:
N=8+d22v.
Here you simply need to know your estimate of d and the variance (v) for a particular comparison. If you are able to estimate N from the above formula reasonably accurately, you can be confident that your estimate is in the ballpark. Note that this formula works best when your treatment and control group have equal sample sizes. Unequal sample sizes will not yield accurate estimates of N.
The above formula will not work with within-subjects designs. The formula that I know does work for within-subjects designs is the following:
n=2(1−r)v+d22v.
Note that the above formula assumes you will know the exact correlation (r) between your variables, which may or may not be reported or available. I have found that under those circumstances, if I assume r = .5000, that I typically get accurate enough estimates of N from my calculations of d and variance (v). That said, for those in the process of conducting a meta-analysis, I recommend contacting the original authors or principle investigators under circumstances where all you might have to go on is a paired-sample t-test and a sample size (and potentially a p-value). Often, the authors are more than happy to provide the info you want or need either in the form of actual estimates of r for each comparison that they computed, or better yet provide the original data set and enough info so you can do so yourself. That's easy with newer studies. Good luck if the research was published much earlier than this decade - though even then I have been amazed at how helpful authors will try to be. For those cross-validating a meta-analyst's database, if the original correlational info is available, ideally it will be recorded in the database itself for within-subjects comparisons. If not, email the meta-analyst. Again, we should be able to provide that info easily enough.
If you embark on a meta-analysis, keep in mind that others who eventually want to see your data will try to cross-validate your effect size estimates. Get ahead of that situation and do so from the get-go on your own. You'll know that you can trust your calculations of effect size and you will be able to successfully address concerns about those computations as they arise later. Ultimately that's the bottom line: you need to know that you can trust the process of how your effect size calculations are being computed, regardless of whether you are using a proprietary software package like CMA or open access language like R, and regardless of how seasoned you are as a meta-analyst. If you find problems cross-validating, then you can go back and check your code for possible errors. That'll undoubtedly save some heartache and heartburn, but the more important thing is that you can be confident that what you ultimately present to your particular audience is the closest approximation to the truth possible. Ultimately, that is all that matters. Hopefully the above is helpful to someone.
And now back to meta-analysis.
I briefly led up to this topic a couple months ago (see Prelude). Where we left off was with the problem that inevitably cropped up with narrative reviews. Meta-analysis offered a promising and more objective alternative to reviewing the literature. The premise is simple enough. We we can combine all studies testing a specific hypothesis in order to get an estimate of the overall effect size (essentially the mean of Cohen's d, Pearson's r, etc.), along with 95% confidence intervals. If the confidence intervals do not include zero, the effect can be considered "significant" - that is it's an effect that appears to be noticeable.We can also examine moderators that might impact the mean effect size estimate. Now admittedly I am oversimplifying, but I just want to provide the gist. If you want the full story, I can recommend any of a number of resources (Michael's Borenstein's work is certainly worth reading).
Meta-analyses are often very useful in providing confirmation that multiple tests of the same hypothesis are confirming initial findings, making sense of messy research literatures, and debunking myths. The reason we rarely talk about Type A personality (TABP) any more is thanks to several meta-analyses that showed no relationship between TABP and heart disease, for example. However, it became obvious in a hurry that there were some issues with this new approach to reviewing the literature.
One problem was that effect sizes were estimated using what was called a fixed effects model. The problem with that was the assumption of fixed effects models is that the collection of studies represent a population. The reality is that we merely have a sample of studies whenever we conduct a meta-analysis, and so we moved to using random effects models. Another very obvious problem is publication bias and the proverbial file drawer problem. Journals rarely publish null findings, and those null findings often don't see the light of day. That is a problem because meta-analyses may be overestimating effect sizes. So, a number of approaches to dealing with that problem have been tried, each with its shortcomings. I still remember the days of the Failsafe N. Thankfully we've moved beyond that. For a number of years, the standard has been Trim-and-Fill analyses and funnel plots. Unfortunately, that approach may understate the potential impact of publication bias. A number of other techniques have been developed and utilized, usually individually, such as PET-PEESE, p-curves, and so on. Each of these techniques individually has its advantages and disadvantages, and in the case of p-curves may be limited to a very specific set of circumstances. A more recent approach, and one I prefer, is to use a combination of sensitivity analyses in order to address publication bias effects and attempt to triangulate around a likely estimate of the mean effect size. If we can triangulate around a likely effect size estimate, we can make some tentatively conclusive statements about the severity of publication bias in a literature and about the extent to which we can say that an effect is real. If we cannot triangulate, we can recommend caution in interpretation of the original naive effect size estimate and try to figure out what is going on with a particular research literature.
In the process, we need to look at another way meta-analysis has changed. When I was working on my dissertation, most of us were using proprietary software such as SAS (which was not developed to handle meta-analysis) or d-stat (which is now defunct) to extract effect size estimates and to synthesize the available effect sizes for a literature. Quite a number of us use software such as CMA, which has a lot of nifty features, although comes with its own limitations (its forest plots and funnel plots leave much to be desired, and one needs to be very careful when entering data into its spreadsheet, as some columns that you can create are not merely columns for entering coded data - something I learned the hard way!). As long as the algorithms in these software appear to work the way they are supposed to and as long as one can cross-validate (e.g., estimate n from, say, estimate of d and variance for each study), you're probably okay. Unfortunately, if one wants to do anything more heavy duty than that, you will want to learn how to use R, and specifically R metafor.
One more thing. I always tell my students that a meta-analysis is not the last word on a particular topic. Additional studies are published (or conducted and not published), methodology improves which may challenge conventional wisdom about a particular research question, and techniques for conducting meta-analysis are continuing to evolve. When reading early meta-analyses, do yourself and me a favor and don't diss them, especially when you realize that the authors may not have examined publication bias, or only used published research, or used Failsafe N as their method of addressing publication bias. The authors of those meta-analyses likely did the best they could with the tools available at the time. We can and should do better with the tools we have at our disposal.
I undoubtedly will want to say more. At some point, I'll provide a better draft of this post with some actual citations and linkage. For now, I just wanted to record some of my thoughts.
Meta-analyses are often very useful in providing confirmation that multiple tests of the same hypothesis are confirming initial findings, making sense of messy research literatures, and debunking myths. The reason we rarely talk about Type A personality (TABP) any more is thanks to several meta-analyses that showed no relationship between TABP and heart disease, for example. However, it became obvious in a hurry that there were some issues with this new approach to reviewing the literature.
One problem was that effect sizes were estimated using what was called a fixed effects model. The problem with that was the assumption of fixed effects models is that the collection of studies represent a population. The reality is that we merely have a sample of studies whenever we conduct a meta-analysis, and so we moved to using random effects models. Another very obvious problem is publication bias and the proverbial file drawer problem. Journals rarely publish null findings, and those null findings often don't see the light of day. That is a problem because meta-analyses may be overestimating effect sizes. So, a number of approaches to dealing with that problem have been tried, each with its shortcomings. I still remember the days of the Failsafe N. Thankfully we've moved beyond that. For a number of years, the standard has been Trim-and-Fill analyses and funnel plots. Unfortunately, that approach may understate the potential impact of publication bias. A number of other techniques have been developed and utilized, usually individually, such as PET-PEESE, p-curves, and so on. Each of these techniques individually has its advantages and disadvantages, and in the case of p-curves may be limited to a very specific set of circumstances. A more recent approach, and one I prefer, is to use a combination of sensitivity analyses in order to address publication bias effects and attempt to triangulate around a likely estimate of the mean effect size. If we can triangulate around a likely effect size estimate, we can make some tentatively conclusive statements about the severity of publication bias in a literature and about the extent to which we can say that an effect is real. If we cannot triangulate, we can recommend caution in interpretation of the original naive effect size estimate and try to figure out what is going on with a particular research literature.
In the process, we need to look at another way meta-analysis has changed. When I was working on my dissertation, most of us were using proprietary software such as SAS (which was not developed to handle meta-analysis) or d-stat (which is now defunct) to extract effect size estimates and to synthesize the available effect sizes for a literature. Quite a number of us use software such as CMA, which has a lot of nifty features, although comes with its own limitations (its forest plots and funnel plots leave much to be desired, and one needs to be very careful when entering data into its spreadsheet, as some columns that you can create are not merely columns for entering coded data - something I learned the hard way!). As long as the algorithms in these software appear to work the way they are supposed to and as long as one can cross-validate (e.g., estimate n from, say, estimate of d and variance for each study), you're probably okay. Unfortunately, if one wants to do anything more heavy duty than that, you will want to learn how to use R, and specifically R metafor.
One more thing. I always tell my students that a meta-analysis is not the last word on a particular topic. Additional studies are published (or conducted and not published), methodology improves which may challenge conventional wisdom about a particular research question, and techniques for conducting meta-analysis are continuing to evolve. When reading early meta-analyses, do yourself and me a favor and don't diss them, especially when you realize that the authors may not have examined publication bias, or only used published research, or used Failsafe N as their method of addressing publication bias. The authors of those meta-analyses likely did the best they could with the tools available at the time. We can and should do better with the tools we have at our disposal.
I undoubtedly will want to say more. At some point, I'll provide a better draft of this post with some actual citations and linkage. For now, I just wanted to record some of my thoughts.
Monday, December 11, 2017
Interlude: Loss of Confidence Project
I saw a blurb on a Loss of Confidence Project on Twitter. I like the idea. Those of us in the psychological sciences conduct research that perhaps at the time we thought was well-done from a theoretical and/or methodological standpoint, but realize later that we made some sort of honest mistake. If we can get to a point where we can take ownership for our mistakes, and foster a culture that is forgiving and accepting of what is probably very common, we'll be better off for it. I'll certainly follow this project with great interest.
Friday, November 24, 2017
Intermission: Beware the term paper mills
Every so often, I do a google search to see who is citing work relevant to my lines of research. I was disheartened recently to find that when my name showed up, it was to documents that led me to a number of term paper writing mills. These companies have been around in one form or another for a while. One of the most recent ones I have become aware of I will refrain from adding a link. No need to give them free advertising. I would advise students to think twice before using these services. Since there are multiple versions of essays citing me (each with the same title) it is likely that buying one of those "original" papers will show up on plagiarism software searches. Any student paper uploaded to my Blackboard course sites are added to our plagiarism software database - and since that database belongs to a third-party company, it becomes easy to match if one tries to upload one of these plagiarized documents. And just in case the software doesn't catch it, be on notice that I will be suspicious of any paper in which I miraculously appear in a Google search that links back to one of these paper writing mills.
If you're a student, save yourself the money, your self-respect, and yourself from a failing grade - and possible embarrassment of appearing before an academic integrity committee or worse. In my field, we are quite forgiving of imperfect student work, and student work in which serious mistakes are present. Heck, I make plenty of mistakes of my own all the time. There is no room, however, for forgiveness for fraud any more today than there was in my day as a student.
If you're a student, save yourself the money, your self-respect, and yourself from a failing grade - and possible embarrassment of appearing before an academic integrity committee or worse. In my field, we are quite forgiving of imperfect student work, and student work in which serious mistakes are present. Heck, I make plenty of mistakes of my own all the time. There is no room, however, for forgiveness for fraud any more today than there was in my day as a student.
Friday, November 3, 2017
Prelude
When I was an undergraduate student, and later a graduate student, if I wanted a summary of the state of the literature pertaining to a research question the narrative review was the primary - and in many cases - the only choice. For those needing a refresher, a narrative review is one in which the author or authors pick a set of studies to summarize, and then offer an intensive analysis of what that literature tells us about how well a particular research hypothesis is holding up. As someone who has certainly read my share of narrative reviews, and authored or coauthored a few of my own, such reviews do have a place. If done even remotely well, a narrative review can offer an encyclopedic summary of a research area, or a quick summary of recent research and theory for a particular line of inquiry.
The problem with narrative reviews was that they were ultimately subjective. Everything from the selection of articles to examine to the conclusions drawn was based ultimately on the particular whims of the authors. With such subjectivity, we were bound to find conflicting narrative summaries on any topic imaginable. If one had sufficient expertise in an area, one could quickly get to know the players well enough to suss out the perspective a particular author or team of authors would likely offer. If one were interested in whether or not psychotherapy was effective, for example, any literature review by Hans Eysenck was going to be predictably negative. However, for novices, or those simply wishing to reinforce pre-existing biases, narrative reviews were highly problematic.
Early narrative reviews on the weapons effect are particularly instructive. Depending on whether one read the work of Leonard Berkowitz and his former students or read the work of researchers who were downright skeptical to the point of cynicism, one would either be convinced that a weapons effect was real or that a weapons effect was non-existent. To make things even more frustrating, in the early 1990s, two book chapters in the same volume were published where competing authors examined mostly the same studies and came to radically different conclusions regarding the existence of the weapons effect. For any of us seeking some closer approximation of truth, such a situation was untenable.
The meta-analysis offered a promising alternative to that untenable set of circumstances. I will turn to that topic shortly.
The problem with narrative reviews was that they were ultimately subjective. Everything from the selection of articles to examine to the conclusions drawn was based ultimately on the particular whims of the authors. With such subjectivity, we were bound to find conflicting narrative summaries on any topic imaginable. If one had sufficient expertise in an area, one could quickly get to know the players well enough to suss out the perspective a particular author or team of authors would likely offer. If one were interested in whether or not psychotherapy was effective, for example, any literature review by Hans Eysenck was going to be predictably negative. However, for novices, or those simply wishing to reinforce pre-existing biases, narrative reviews were highly problematic.
Early narrative reviews on the weapons effect are particularly instructive. Depending on whether one read the work of Leonard Berkowitz and his former students or read the work of researchers who were downright skeptical to the point of cynicism, one would either be convinced that a weapons effect was real or that a weapons effect was non-existent. To make things even more frustrating, in the early 1990s, two book chapters in the same volume were published where competing authors examined mostly the same studies and came to radically different conclusions regarding the existence of the weapons effect. For any of us seeking some closer approximation of truth, such a situation was untenable.
The meta-analysis offered a promising alternative to that untenable set of circumstances. I will turn to that topic shortly.
Subscribe to:
Posts (Atom)