Antw: [OpenScienceFramework] cumulative meta-analytic datasets for direct replications

[Heather Fuchs]

Hi Frank,

I think this idea sounds really interesting! I think it would be difficult to have such a system be fully automated - e.g., an internet user interface where researchers can simply enter their study information and see the cumulative effect change - just because there are so many subjective decisions that go into a meta-analysis (as with any statistical method). But I could imagine this working in the form of a researcher initivative, where someone very knowledgable in meta-analysis (and at least willing to become knowledgable regarding the effect of interest) looks over the data and enters it into the system in an appropriate manner. I think an added positive effect of this kind of initiative would be to help researchers understand that "negative" effects are normal and still contribute to the cumulative body of knowledge surrounding a certain effect. Overall, I don't think this kind of thing would be completely simple to get started, but I think it would definately be worthwhile!

Sincerely,

Heather

[Brian Nosek]

Hi Frank --

Yes, it has been done before, but only on a small scale.  I haven't found the website again, but there is (or was) one for real time updates of meta-analytic effects for clinical trials.  I saw it perhaps 5 years ago when we were writing a grant to fund open science efforts and part was a proposal to improve data ontologies to make aggregation easier.  

The grant wasn't funded, but the ideas are in the long-term plan for OSF.  If, however, you discover others that have already built the tools, then much better to just link into existing technologies and help make them general use.

brian