Archival survey of replication in published work?

[Roger Giner-Sorolla]

While I have no wish to interfere with the current project, it occurs
to me that another thing we could be doing is research into the
prevalence in a given field of each of the following classes of
published study:

1. Exact replications of prior research (and whether or not this is in
the context of a "replicate and extend" or "replicate but reinterpret"
strategy

2. Conceptual replications

3. Failures to replicate that then establish the reason for failure

4. Failures to replicate without establishing a reason or boundary
condition

In social psychology we can take it as a given that failures to
replicate that don't establish a reason why, are by and large not
published. The one exception I know of proves the rule: Klauer & Musch
2001, non-replicating the evaluative priming effect with pronunciation
task, was published in a cognitive journal, but then Spruyt, de Houwer
et al. 2002 do the more usual thing which is to establish a boundary
condition, and publish in Social Cognition.

Although this suggestion would be a ton of work, it could supplement
the existing replication project by being unarguably representative,
and highlight the following issues:

Are highly cited studies more likely to be replicated outside the
originating lab?
Are there outliers of highly cited studies that have not been
replicated outside the originating lab?
Does media attention correlate *inversely* with replication? (I
suspect it does!)
How much do textbook citations stick to replicated results?
Is experimental/cognitive psych more nonreplication/replication
friendly than e.g. social?

My motivation here comes from observing that more than one person has
replied to this kind of initiative with "But we *do* publish
replications all the time ... replicate and extend that is." It would
be nice to have an answer to this, because the main problem I see from
this way fo doing things is that it's not clear whether a non-
replicated study is that way because nobody tried, or because they
tried, failed, couldn't publish.

[Brian Nosek]

I agree that this is a worthwhile project and could be done in parallel with the "active" replication project that we are currently developing.  Also, defining the project like you have would allow us to put aside some items that we might have been tempted to pursue in the reproducibility project but with a design that was not optimal.  

It might be able to follow the organizational strategy that we establish here, just with a more substantial focus on reviewing the literature and no emphasis on actually conducting the replications.  They'd be very complementary projects.