cumulative meta-analytic datasets for direct replications

[Frank Farach]

Some recent discussions in this group --- and Nosek et al.'s articles
on scientific utopia --- have gotten me thinking about different ways
to improve the communal benefits of direct replications. One vision of
scientific utopia I have been playing with is the semi-automated
aggregation of direct replications into cumulative, always-up-to-date,
meta-analytic datasets. These meta-analytic effect sizes and their CIs
would be updated when a new direct replication is published (or
perhaps even preprinted). At first blush, what would be needed to make
this work is: a mechanism for identifying instances of direct
replications; tagging and linking them to the original article;
applying the most appropriate formula(e) for aggregating the primary
effect(s); and publishing the data, perhaps with a brief commentary.
This would yield information that (I would think) would be of direct
interest to the scientific community: the current, cumulative
magnitude and precision of the effect in question. It would also be
trivial to show the historical moving average of the effect size,
which itself might be an interesting way to visualize the impact of
specific studies on the effect.

I know there is great interest in this kind of approach in
evidence-based health care. The UK-based Cochrane Review regularly
publishes updates to its systematic meta-analyses of clinical trials
as evidence accumulates. However, they focus on topical/conceptual
replications (which requires more expertise-based curation) rather
than direct methodological replication (which would be easier to do in
a distributed, open-science manner). Cochrane Reviews are generally
highly regarded in evidence-based health care.

I don't think we are all that far from being able to implement some
components of this idea for selected effects (e.g., controversial or
high-impact effects). The primary barriers are social --- extent of
data sharing, citation practices, and, of course, direct replications
--- not technical.

Perhaps this idea is unrealistic, but I thought I'd just get it out
there and see where it goes. Has anyone come across anything like this
before?

Frank