Wednesday, September 11, 2019
Coming soon
I am starting to get my head around some relatively new Zhang lab errata. I have questions. Stay tuned.
Sunday, August 25, 2019
Will this time be different?
I had the pleasure of seeing and hearing Sanjay speak these words live in early July at the closing of the SIPS conference in Rotterdam. I really hope those words are heeded. Simply tightening up some methods without addressing the social inequality that afflicts our science (as is the case with so many sciences) is insufficient. If the only people who benefit are those who just happen to keep paying membership dues, we've failed. Open science is intersectional and is a social movement. Anything short of that will be a failure. I hope that I do not find myself in a decade asking the same question that a good friend of mine once asked over three decades ago in his zine, Pressure: "So, where's the change?"
Thursday, August 1, 2019
Some initial impressions about SIPS 2019
I think perhaps the best way to start is with a Twitter thread I posted right as we were about to end:
My experience started with the preconference put together by the repliCATS project. Their travel grant to those willing to participate in the preconference is what made going to SIPS possible for me. During the 5th and 6th of July, I spent the entire work day at the conference site with a team of several other psychologists in various phases of their careers (most were postdocs and grad students). Each team was tasked with the responsibility of assessing the probability of replication for 25 claims. We had a certain amount of time to read each claim, look up the relevant article, look up any other supplementary materials relevant to the task, and then to make our predictions. We then discussed our initial assessments and recalibrated. I found the process engaging and enlightening. What was cool was how each of us brought some unique expertise to the table. Four of the claims we assessed were meta-analytic, and since I was the one person in the room who had conducted published meta-analyses, I became the de facto expert on that methodology. Believe me when I say that I still don't feel like an expert. But okay. So for those claims, my peers quizzed me a good deal and I got to share a few things that might be useful about raw effect sizes, assessment of publication bias, etc. We did the same with others. There were some claims that all of us found vexing. Goes with the territory. I think what I got from the experience was how much we in the psychological sciences really need each other if we are going to move the field forward. I also realized just how talented the current generation of early career researchers truly is. As deflating as some of the research we evaluated was, I could not help but feel a certain level of cautious optimism about the future of the psychological sciences.
The conference itself started mid-day on the 7th of July and lasted through the 9th of July. It was filled with all sorts of sessions - hackathons, workshops, and unconferences. I tended to gravitate to the latter two classes of sessions during the conference this time around. The conference format itself is rather loose. One could start a session, decide "no this is not for me" and leave without worrying about hurt feelings. One could walk in to a session in the middle and be welcomed. It was great to go to a professional meeting without seeing one person wearing business attire. At least that level of pretense was dispensed with, for which I am grateful. I gravitated toward sessions on the last day that had a specific focus on inclusiveness. That is a topic that has been on my mind going back to my student activist days in the 1980s. There is a legitimate concern that by the time all is said and done, we'll manage to fix some methodological problems that are genuinely troubling for the discipline without addressing the problem that there are a lot of talented people who could offer so much who are shut out due to their ethnicity, national origin (especially if from the Global South), sexual orientation, and gender identity. If all we get as the same power structure with somewhat better methods, we will have failed, in my personal and professional opinion. I think the people leading at least a couple of those sessions seemed to get that. I hope that those who are running SIPS get it too. Maybe an inclusiveness hackathon is in order? I guess I am volunteering myself to lead that one just by blogging about it!
This was also an interesting conference in that I am both mid-career and primarily an educator. So, I was definitely part of a small subset of attendees. Personally, I felt pretty engaged. I can see how one in similar circumstances might end up feeling legitimately left out. There was some effort to have sessions devoted to mid-career researchers. We may want that expanded to mid-career educators as well. We too may want to be active participants in creating a better science of psychology, but our primary means of doing so is going to be in the classroom and not via published reports. Those of us who may one day become part of administration (dept chairs, deans, etc.) or who already occupy those position should have some forum to discuss how we can better educate the next generation of Psychology majors at the undergraduate level so that they are both better consumers of research and better prepared for the changes occurring that will impact them as they enter graduate programs (for that subset of majors who will do so).
It was also great to finally meet up in person with a number of people with whom I have interacted via Twitter DM, email, and sometimes via phone. That experience was beautiful. There is something about actually getting to interact in person that is truly irreplaceable.
Given the attendance at the conference, I can see how much of a logistical challenge the organizers faced. There were moments where last minute room changes did not quite get communicated. The dinner was one where participants were mostly underfed (that is probably more on the restaurant than the organizers, and I can chalk that up to "stuff happens" and leave it at that). Maintaining that sense of intimacy with a much larger than anticipated group was a challenge. But I never felt isolated or alone. There was always a session of interest. There was always someone to talk to. The sense of organized chaos is one that should be maintained.
I did find time to wander around the city of Rotterdam, in spite of my relative lack of down time for this conference. The fact that it was barely past the Summer Solstice meant that I got some good daylight quality photos well into the evening. I got to know the "cool district" of Rotterdam quite well, and definitely went off the beaten path in the process. I'll share some of those observations at another time.
Overall, this was a great experience. I will likely participate in the future, as I am finding ways of using what I learned in the classroom. As long as participants can come away from the experience thinking and knowing that there were more good sessions than they could possibly attend, it will continue to succeed. If the organizers are serious about inclusiveness, they will have something truly revolutionary as part of their legacy. Overall, the sum total of this set of experiences left me cautiously optimistic in a way that I have not been in a very long time. There is hope yet for the psychological sciences, and I got to meet some of the people who are providing the reason for that hope. Perhaps I will meet others who did not attend this year at future conferences. I'll hold out hope for that as well.
The conference was different from any conference I have ever experienced. For those wanting to get a feel for what SIPS is about, a good place to begin might be to check out the page for this year's conference. Rotterdam was a good location in part because the city slogan is make it happen. SIPS is an organization devoted to actively changing the way the science of psychology is done, and is formatted in such a way that those participating become active. This is a conference for people who really want to roll up their sleeves and get involved.1. With #SIPS2019 about to come to a close Wednesday, I have a few observations - brief for now. I love the format and have felt more actively engaged in a conference than I have in a very long time.— James Benjamin (@AJBenjaminJr) July 8, 2019
My experience started with the preconference put together by the repliCATS project. Their travel grant to those willing to participate in the preconference is what made going to SIPS possible for me. During the 5th and 6th of July, I spent the entire work day at the conference site with a team of several other psychologists in various phases of their careers (most were postdocs and grad students). Each team was tasked with the responsibility of assessing the probability of replication for 25 claims. We had a certain amount of time to read each claim, look up the relevant article, look up any other supplementary materials relevant to the task, and then to make our predictions. We then discussed our initial assessments and recalibrated. I found the process engaging and enlightening. What was cool was how each of us brought some unique expertise to the table. Four of the claims we assessed were meta-analytic, and since I was the one person in the room who had conducted published meta-analyses, I became the de facto expert on that methodology. Believe me when I say that I still don't feel like an expert. But okay. So for those claims, my peers quizzed me a good deal and I got to share a few things that might be useful about raw effect sizes, assessment of publication bias, etc. We did the same with others. There were some claims that all of us found vexing. Goes with the territory. I think what I got from the experience was how much we in the psychological sciences really need each other if we are going to move the field forward. I also realized just how talented the current generation of early career researchers truly is. As deflating as some of the research we evaluated was, I could not help but feel a certain level of cautious optimism about the future of the psychological sciences.
The conference itself started mid-day on the 7th of July and lasted through the 9th of July. It was filled with all sorts of sessions - hackathons, workshops, and unconferences. I tended to gravitate to the latter two classes of sessions during the conference this time around. The conference format itself is rather loose. One could start a session, decide "no this is not for me" and leave without worrying about hurt feelings. One could walk in to a session in the middle and be welcomed. It was great to go to a professional meeting without seeing one person wearing business attire. At least that level of pretense was dispensed with, for which I am grateful. I gravitated toward sessions on the last day that had a specific focus on inclusiveness. That is a topic that has been on my mind going back to my student activist days in the 1980s. There is a legitimate concern that by the time all is said and done, we'll manage to fix some methodological problems that are genuinely troubling for the discipline without addressing the problem that there are a lot of talented people who could offer so much who are shut out due to their ethnicity, national origin (especially if from the Global South), sexual orientation, and gender identity. If all we get as the same power structure with somewhat better methods, we will have failed, in my personal and professional opinion. I think the people leading at least a couple of those sessions seemed to get that. I hope that those who are running SIPS get it too. Maybe an inclusiveness hackathon is in order? I guess I am volunteering myself to lead that one just by blogging about it!
This was also an interesting conference in that I am both mid-career and primarily an educator. So, I was definitely part of a small subset of attendees. Personally, I felt pretty engaged. I can see how one in similar circumstances might end up feeling legitimately left out. There was some effort to have sessions devoted to mid-career researchers. We may want that expanded to mid-career educators as well. We too may want to be active participants in creating a better science of psychology, but our primary means of doing so is going to be in the classroom and not via published reports. Those of us who may one day become part of administration (dept chairs, deans, etc.) or who already occupy those position should have some forum to discuss how we can better educate the next generation of Psychology majors at the undergraduate level so that they are both better consumers of research and better prepared for the changes occurring that will impact them as they enter graduate programs (for that subset of majors who will do so).
It was also great to finally meet up in person with a number of people with whom I have interacted via Twitter DM, email, and sometimes via phone. That experience was beautiful. There is something about actually getting to interact in person that is truly irreplaceable.
Given the attendance at the conference, I can see how much of a logistical challenge the organizers faced. There were moments where last minute room changes did not quite get communicated. The dinner was one where participants were mostly underfed (that is probably more on the restaurant than the organizers, and I can chalk that up to "stuff happens" and leave it at that). Maintaining that sense of intimacy with a much larger than anticipated group was a challenge. But I never felt isolated or alone. There was always a session of interest. There was always someone to talk to. The sense of organized chaos is one that should be maintained.
I did find time to wander around the city of Rotterdam, in spite of my relative lack of down time for this conference. The fact that it was barely past the Summer Solstice meant that I got some good daylight quality photos well into the evening. I got to know the "cool district" of Rotterdam quite well, and definitely went off the beaten path in the process. I'll share some of those observations at another time.
Overall, this was a great experience. I will likely participate in the future, as I am finding ways of using what I learned in the classroom. As long as participants can come away from the experience thinking and knowing that there were more good sessions than they could possibly attend, it will continue to succeed. If the organizers are serious about inclusiveness, they will have something truly revolutionary as part of their legacy. Overall, the sum total of this set of experiences left me cautiously optimistic in a way that I have not been in a very long time. There is hope yet for the psychological sciences, and I got to meet some of the people who are providing the reason for that hope. Perhaps I will meet others who did not attend this year at future conferences. I'll hold out hope for that as well.
Wednesday, July 31, 2019
Loose ends
I got back in from Rotterdam late on the 10th of July. It was a great trip and I will talk about it and the aftermath another time.
I've also been clearing my file drawer a bit. Earlier this year saw a narrative review article on the weapons effect published in a small-circulation peer-review journal (National Social Science Journal). I and a former student just learned that a write up of a weapons priming data set got accepted for publication. We are just now going over the proofs. It will be a fun paper to discuss down the road as I am now in this awkward space where I had that one remaining study using a word completion task as a measure prior to realizing that the measure itself had some genuine validity problems (Randy McCarthy and I blogged about some of that earlier this year). My student and I used this as an opportunity to build in a paragraph urging readers to exercise caution in light of some genuine concerns about the lack of validation of not only the AWCT but also other word completion tasks. I am finally writing up an encyclopedic chapter on the weapons effect that is due at the end of August. After that, I think I will finally be done with the weapons effect as a focus of my attention for a good long while. At least that is my hope. I do have the sinking feeling that I could get roped into something to do with that area of inquiry sooner rather than later, but am keeping my fingers crossed that my time has come and gone at last.
Mostly I just want to focus my upcoming academic year on validation research and re-structuring my methods courses so that they incorporate more hands-on activities regarding replicability and open science practices. I got some ideas from SIPS and the repliCATS-sponsored pre-SIPS workshop in Rotterdam. I think it will turn out to be fun.
Onward.
I've also been clearing my file drawer a bit. Earlier this year saw a narrative review article on the weapons effect published in a small-circulation peer-review journal (National Social Science Journal). I and a former student just learned that a write up of a weapons priming data set got accepted for publication. We are just now going over the proofs. It will be a fun paper to discuss down the road as I am now in this awkward space where I had that one remaining study using a word completion task as a measure prior to realizing that the measure itself had some genuine validity problems (Randy McCarthy and I blogged about some of that earlier this year). My student and I used this as an opportunity to build in a paragraph urging readers to exercise caution in light of some genuine concerns about the lack of validation of not only the AWCT but also other word completion tasks. I am finally writing up an encyclopedic chapter on the weapons effect that is due at the end of August. After that, I think I will finally be done with the weapons effect as a focus of my attention for a good long while. At least that is my hope. I do have the sinking feeling that I could get roped into something to do with that area of inquiry sooner rather than later, but am keeping my fingers crossed that my time has come and gone at last.
Mostly I just want to focus my upcoming academic year on validation research and re-structuring my methods courses so that they incorporate more hands-on activities regarding replicability and open science practices. I got some ideas from SIPS and the repliCATS-sponsored pre-SIPS workshop in Rotterdam. I think it will turn out to be fun.
Onward.
Sunday, June 30, 2019
Pride Month May Be Coming To An End
Pride Month may be coming to an end today but the social concerns that brought us Pride Month continue year 'round. So, whatever our specific orientations and gender identities, let's remember that our sciences often fall short as safe spaces for everyone to discuss our research. The US is not an especially inclusivity-friendly nation right now - think of what the occupant in the White House and his allies espouse and put into policy (including undoing progress that took years of difficult work to accomplish). I am well aware that transgender scientists still get discriminated against and are publicly bullied by peers (very obviously so on social media) who really should know better.
In the meantime, enjoy this wonderful post. I love the logo (see below - nicked from the same post). And I love the notion of a more inclusive science and society.
In the meantime, enjoy this wonderful post. I love the logo (see below - nicked from the same post). And I love the notion of a more inclusive science and society.
Saturday, June 29, 2019
Radio Silence
The summer continues, and I am balancing online courses with professional and personal travel. I am also trying to clear my desk of data sets collected during the time I was working on the meta-analysis that got published last year. Those data sets are simple replication studies and I am mainly aiming for journals that are open source and access that are receptive to such work. Nothing prestigious, but that is not the goal. The point is simply to keep data out of the file drawer.
I am thinking through how to get back to the basics: mainly focusing on measurements relevant to my area of expertise. Although such work is often viewed as far from glamorous, it is vital. I think I have found some cool folks to work with. Part of my professional travel includes meeting up with some of these people. I am looking forward to it. Much of what I am doing is learning or relearning.
I hope to blog about my experience at SIPS. That is coming up soon. Probably won't post much of anything until later in July. Between what I was observing from the sidelines as the replication crisis unfolded and some professional experiences I wish could have been avoided, I became convinced that I needed to follow the lead of those who are on the front lines to change the psychological sciences for the better. I'll check out some hackathons and unconferences and soak it in. Then I'll figure out how to incorporate what I learn in my methods courses.
I have not had much to say about Qian Zhang's (Southwest University in China) work lately, but that has more to do with 1) a concern that I would be repeating myself at this point and 2) there are folks better positioned to force the issue regarding some serious problems with that lab's published work. It strikes me as prudent to let those who are better able to do so have that space to make it happen. If it makes sense for me to say something further, just know I will. I am sure that those of you who check this blog out know that I was genuinely appalled at what had slipped through peer review. That has not changed and as far as I am concerned this isn't over by a long shot. Let's see how the process plays out. I'll simply state for the moment that it is a shame, as quality research from outside the US and EU is essential to better understanding the nuances of whatever influence media violence cues might have on aggressive cognitive and behavioral outcomes. What I shared with you all about that set of findings only added to misconceptions about media violence, given that the reporting of the methodology and findings themselves was so poorly done.
For now, it's radio silence. I will return in a few weeks.
I am thinking through how to get back to the basics: mainly focusing on measurements relevant to my area of expertise. Although such work is often viewed as far from glamorous, it is vital. I think I have found some cool folks to work with. Part of my professional travel includes meeting up with some of these people. I am looking forward to it. Much of what I am doing is learning or relearning.
I hope to blog about my experience at SIPS. That is coming up soon. Probably won't post much of anything until later in July. Between what I was observing from the sidelines as the replication crisis unfolded and some professional experiences I wish could have been avoided, I became convinced that I needed to follow the lead of those who are on the front lines to change the psychological sciences for the better. I'll check out some hackathons and unconferences and soak it in. Then I'll figure out how to incorporate what I learn in my methods courses.
I have not had much to say about Qian Zhang's (Southwest University in China) work lately, but that has more to do with 1) a concern that I would be repeating myself at this point and 2) there are folks better positioned to force the issue regarding some serious problems with that lab's published work. It strikes me as prudent to let those who are better able to do so have that space to make it happen. If it makes sense for me to say something further, just know I will. I am sure that those of you who check this blog out know that I was genuinely appalled at what had slipped through peer review. That has not changed and as far as I am concerned this isn't over by a long shot. Let's see how the process plays out. I'll simply state for the moment that it is a shame, as quality research from outside the US and EU is essential to better understanding the nuances of whatever influence media violence cues might have on aggressive cognitive and behavioral outcomes. What I shared with you all about that set of findings only added to misconceptions about media violence, given that the reporting of the methodology and findings themselves was so poorly done.
For now, it's radio silence. I will return in a few weeks.
Wednesday, June 12, 2019
Back to the usual summer routine
Summer started about a month ago for me. A few things typically happen the moment I turn in my final grades. First, I try to decompress a bit. That might mean reading some books, binging some series on Netflix, Hulu, or Amazon Prime, gardening, or day hikes and photography. Some combination of the above is inevitably involved.
Then, I get busy with prepping summer classes, prepping for my usual trip to the AP Psychology Reading site, some family trips (one major due to distance, and a series of day trips for family closer to me), and planning projects and papers for the upcoming year. The latter is becoming less of a priority for me. I have some data sets I need to write up and submit that got neglected during the time when I was heavily involved in a meta-analysis. One was submitted late in April and I don't expect to hear anything for a while. The other two will need to be dealt with sooner or later. Both are replication reports, and those are often not a priority for most journals. I have those outlined. It is just a matter of blocking out time for writing. Beyond that, I have a chapter due at the end of August. It is on a topic with which I have regrettably way too much familiarity (the weapons effect). I can write in a neutral tone, and given the evidence I would need to cite, neutrality is pretty easy for me. I am involved in some collaborative projects that are still largely in their infancy. My hope there is that my collaborators take the lead, as I really have no ambition to take on more than my resources allow.
I have good collaborators, and I expect to learn a great deal from them in the coming year or two. I am heading to SIPS in Rotterdam in early July. Normally I don't do conferences right at the start of a fiscal year, and especially conferences overseas. I have to front way too much money with little hope for reimbursement until several months have passed. An outside travel grant from DARPA made this trip just barely feasible for me financially, and the opportunity to meet with and work with people I know primarily through Twitter direct messages and the occasional phone call or Skype is just too good for me to pass up. I am hoping to take away from the experience some ideas for how to better retool some of my program's methodology courses - ideas that I can float to my departmental colleagues and hopefully implement a bit this coming fall and more completely over the next academic year or two.
A while back I made mention that one of the big lies that we get told in graduate school is that we are supposed to want to work at prestigious institutions, publish in A-list journals, win large grant awards, and essentially be famous. Like a lot of people in my position, I bought into that lie, and I fear that I compromised myself in the process. The last couple years were a bit of a wake-up call for me. If one is really intent on an academic career, there are plenty of ways of doing so and find the experience meaningful and rewarding. The R-1 and R-2 institutions are not necessarily ideal destinations. My full-time gigs have been at institutions that barely emphasize research and that cater to students who may simply want a 2-year degree, rather than a 4-year degree. I do some adjuncting at a community college. I find something meaningful in giving away something interesting or useful about the science of psychology (to paraphrase the late George Miller) at my current position. It is, in short, a calling.
In a sense, I think this summer is more one of self-reflection. I honestly have no idea what the final chapter of my academic career will be. I know I have a couple decades to coauthor it. I can say that if the experiences of the last couple years have awakened me sufficiently to focus on the truth at all costs, that's good enough. I was in a dogmatic slumber until a couple years ago regarding a lot of social priming research. I have been sufficiently awakened from that dogmatic slumber (to paraphrase Kant) based on what I have personally had the chance to analyze and based on my reading of the current state of the literature in my area of expertise. I am fortunate enough to work and live in a community that has been nothing short of kind and accepting to me and my family. That too matters a great deal. Let's just say that part of my self-reflection centers on that particular reality as well.
Then, I get busy with prepping summer classes, prepping for my usual trip to the AP Psychology Reading site, some family trips (one major due to distance, and a series of day trips for family closer to me), and planning projects and papers for the upcoming year. The latter is becoming less of a priority for me. I have some data sets I need to write up and submit that got neglected during the time when I was heavily involved in a meta-analysis. One was submitted late in April and I don't expect to hear anything for a while. The other two will need to be dealt with sooner or later. Both are replication reports, and those are often not a priority for most journals. I have those outlined. It is just a matter of blocking out time for writing. Beyond that, I have a chapter due at the end of August. It is on a topic with which I have regrettably way too much familiarity (the weapons effect). I can write in a neutral tone, and given the evidence I would need to cite, neutrality is pretty easy for me. I am involved in some collaborative projects that are still largely in their infancy. My hope there is that my collaborators take the lead, as I really have no ambition to take on more than my resources allow.
I have good collaborators, and I expect to learn a great deal from them in the coming year or two. I am heading to SIPS in Rotterdam in early July. Normally I don't do conferences right at the start of a fiscal year, and especially conferences overseas. I have to front way too much money with little hope for reimbursement until several months have passed. An outside travel grant from DARPA made this trip just barely feasible for me financially, and the opportunity to meet with and work with people I know primarily through Twitter direct messages and the occasional phone call or Skype is just too good for me to pass up. I am hoping to take away from the experience some ideas for how to better retool some of my program's methodology courses - ideas that I can float to my departmental colleagues and hopefully implement a bit this coming fall and more completely over the next academic year or two.
A while back I made mention that one of the big lies that we get told in graduate school is that we are supposed to want to work at prestigious institutions, publish in A-list journals, win large grant awards, and essentially be famous. Like a lot of people in my position, I bought into that lie, and I fear that I compromised myself in the process. The last couple years were a bit of a wake-up call for me. If one is really intent on an academic career, there are plenty of ways of doing so and find the experience meaningful and rewarding. The R-1 and R-2 institutions are not necessarily ideal destinations. My full-time gigs have been at institutions that barely emphasize research and that cater to students who may simply want a 2-year degree, rather than a 4-year degree. I do some adjuncting at a community college. I find something meaningful in giving away something interesting or useful about the science of psychology (to paraphrase the late George Miller) at my current position. It is, in short, a calling.
In a sense, I think this summer is more one of self-reflection. I honestly have no idea what the final chapter of my academic career will be. I know I have a couple decades to coauthor it. I can say that if the experiences of the last couple years have awakened me sufficiently to focus on the truth at all costs, that's good enough. I was in a dogmatic slumber until a couple years ago regarding a lot of social priming research. I have been sufficiently awakened from that dogmatic slumber (to paraphrase Kant) based on what I have personally had the chance to analyze and based on my reading of the current state of the literature in my area of expertise. I am fortunate enough to work and live in a community that has been nothing short of kind and accepting to me and my family. That too matters a great deal. Let's just say that part of my self-reflection centers on that particular reality as well.
Subscribe to:
Posts (Atom)