Research Lead: Methodological


Replicating the Replicators: Is Psychology in Crisis or Not?

replicators_mainPhoto: Luke Price

[Methodological Psychology]

Science recently published a critique of the findings of the Reproducibility Project: Psychology.  The Reproducibility Project, conducted by the Open Science Collaboration (OSC), gained significant attention last fall for its sobering assessment of the state of psychological science, reporting that less than half of the 100 studies it attempted to replicate did so.

The recent critique, authored by Dan Gilbert, Gary King, Stephen Pettigrew, and Tim Wilson suggests the authors of the original report neglected to take into account a number of key factors in their study design and analysis, including key sources of random error, underpowered replication attempts, and biases in replication methodology. These failures, Gilbert and his co-authors argue, led the OSC to incorrectly portray the rate of replication within psychology as a “crisis.” Members of the OSC replied to the critique, standing by their original conclusion, and offering a critique of their own on Gilbert and his co-authors’ methods and analyses.

Since the critique and response were published, a number of people have attempted to digest the situation in which psychology currently finds itself. Brian Nosek and Elizabeth Gilbert, two of the authors on the original reproducibility report, published a follow-up piece, as did Dan Gilbert and his co-authors here. Uri Simonsohn added his thoughts, as did Sanjay Srivastava. Articles were also published in Nature, the New York Times, The Atlantic and New York Magazine, though Katie Palmer may have summed it up best in her piece for Wired: “Psychology is in Crisis Over Whether It’s in Crisis.”

Disclosure: Dan Gilbert is on The Psych Report’s Advisory Board.


Who’s participating in research on Amazon’s Mechanical Turk?

[Methodological Psychology]

Mimicking 10 facial expressions gets you $1.60. Taking a 90-minute survey on your emotions is worth a buck. Throughout the social sciences, it’s becoming increasingly common for researchers to employ Amazon’s Mechanical Turk–an online marketplace where “workers”are paid to complete tasks offered by “requesters”–in their empirical research. MTurk, as it’s known, is efficient and inexpensive making it an especially attractive research tool. But one of the big questions that remains is who are the workers, or in the case of research, participants, that complete the tasks?

In a recent article, published in Current Directions In Psychological Science, Gabriele Paolacci and Jesse Chandler, review the latest research examining the use of MTurk as a participant pool. Demographically speaking, the MTurk workforce is made up of over 500,000 people from 190 countries, with about 75% of workers living in the United States and India. Paolacci and Chandler report that MTurk offers researchers a participant population that is more diverse than the typical college student population, but still not representative of the population as a whole. According to the authors, “Workers tend to be younger (about 30 years old), overeducated, underemployed, less religious, and more liberal than the general population.” Furthermore, within the US’s MTurk workforce, Asians are overrepresented, while Blacks and Hispanics are underrepresented.

Paolacci and Chandler report workers are motivated primarily by size of the payout, but are also motivated by the intrinsic aspects of the tasks as well. Evidence also suggests that MTurk workers respond just as truthfully, and are similarly attentive as traditional participant samples. However, because obtaining future work on the site is often dependent on how they completed previous work (accurate, on-time) the authors highlight the possibility of demand characteristics within MTurk. Likewise, increased experience completing research tasks, particularly economic games and problems, may lead to a practice effect impacting worker responses. The authors also caution that arbitrary factors in experimental design could impact participant selection, and emphasize the need for researchers to take steps to understand and report the make-up of their participant population.


“Predicting ethnic and racial discrimination: A meta-analysis of IAT criterion studies”

[Social & Methodological Psychology]

The creation of the Implicit Association Test (IAT), in 1998, sparked a new wave of theory and research focused on understanding people’s implicitly held attitudes and beliefs, and how those implicit attitudes and beliefs might affect behavior. One of most researched topics using IAT methodology is racial and ethnic discrimination. Employing the IAT, many researchers seemed to uncover implicit biases and prejudices that explicit measures, specifically because they were explicit, could not uncover. However, a new meta analysis, by Oswald et al. revealed the IAT to be a poor predictor of racial or ethnic discriminatory attitudes and behavior, and in fact no better than an explicit measure. Given these results, Oswald et al. question the construct validity of the IAT, and in turn its ability to predict actual behavior. These new findings raise significant questions about psychology and other domains widespread use and application of IAT based results, specifically with regard to racial and ethnic discrimination, and have broad implications for the legal system that has made significant use of the IAT since its inception 15 years ago.

“The Pervasive Problem With Placebos in Psychology: Why Active Control Groups Are Not Sufficient to Rule Out Placebo Effects”


Proper methodological design is paramount for any scientific study, especially those making causal claims. In a recent article, Boot et al. demonstrate that many psychology studies fail to control for differences in expectation between participants in the control and experimental groups. For many studies, the effects observed in the experimental group may in fact be due to the placebo effect of expectation, rather than the intervention itself, calling into question any causal claims. Through a case study and a simple test of expectations, Boot and colleagues show that expectations do infact differ between the control and experimental groups, and how the failure to control for such differences can lead to faulty or at least premature claims of causation. To help remedy this pervasive problem, the authors point to certain methodological designs that can control for expectations, and call on researchers, reviewers, and editors to set a higher methodological standard when making causal claims for a psychological intervention.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>