Invitation to Decision Markets - www.mtrp.info
michael.h...@gmail.com
Decision markets: Help us choose which 26 out of 41 MTurk experiments published in PNAS to replicate (sign-up form)
We are a group of scientists from several international universities who will be conducting replications of a subset of 41 experimental studies in the social sciences published in the Proceedings of the National Academy of Science (PNAS) in 2015-2018. We will replicate 26 of the 41 studies, and all studies have a positive probability of being selected for replication. The 41 papers were objectively chosen because they were published in this high-profile journal in this time period. They all are online studies involving subjects from Amazon Mechanical Turk, they share a common structure in testing a treatment or interaction effect within or between subjects, and they test at least one clear hypothesis and report a statistically significant finding. We plan to conduct the replications from January 2022 onwards. The selection of the 41 studies does not reflect any subjective belief of ours about whether or not these studies were especially likely to replicate.
Choosing what to replicate: We will run a decision market where we will ask participants – maybe you? – to help us decide which studies to replicate. All studies will have a positive probability of being chosen for replication, but we will only replicate 26 of the 41 studies: the 12 with the lowest decision market prices, the 12 with the highest decision market prices, and 2 randomly selected studies among the remaining 17 studies. In short, decision markets function similarly to prediction markets in that traders bet on the replication outcome and the incentives are to bet according to your beliefs, but not all studies will actually be replicated.
When: The decision markets will open on November 8, 2021 and will be open for two weeks.
How can I sign-up for the decision markets? Registrations to participate in the decision markets are administered via this sign-up form. A PhD degree (or currently being a PhD student) or a comparable level of expertise in the social sciences is required for participating to ensure expertise in the field. Registration to participate in the markets will close on October 29, 2021 at 4pm UTC. Before entering the markets, we will ask you to fill out a survey asking for your replication beliefs. Completing the survey by November 5 constitutes a prerequisite to participate as a trader in the markets.
How will market participation be incentivized? Trading on the decision markets will be incentivized, i.e., traders will be paid based on their performance in the markets. Payments will be made after the markets close and the asset payoffs are determined by the actual replication outcomes for those studies chosen for replication. When signing up for the markets, participants have the opportunity to indicate whether the final payment shall be paid via Amazon gift cards (either for amazon.com, amazon.co.uk, or amazon.de) or PayPal.
We will carry out a data collection based on having 90% power to detect 67% of the effect size reported in the original study (with the effect size transformed into the Cohen's d, to have a common standardized effect size measure across all studies). The criteria for replication is an effect in the same direction as the original study and a p-value < 0.05 (in a two-sided test).
More information can be found on the project webpage: www.mtrp.info, where you can also find details about 41 studies that are in the pool for potential replication,
If you have any questions, please do not hesitate to email mturk-rep...@uibk.ac.at
Best wishes,
Colin Camerer, Yiling Chen, Anna Dreber, Teck Ho, Felix Holzmeister, Suzanne Hoogeveen, Jürgen Huber, Lawrence Jin, Magnus Johannesson, Michael Kirchler, Alexander Ly, Benjamin Mandl, Dylan Manfredi, Gidi Nave, Brian Nosek, Thomas Pfeiffer, Alexandra Sarafoglou, Rene Schwaiger, EJ Wagenmakers, and Viking Waldén