21 Words that signal disclosure

[Simonsohn, Uri]

Hi list,

 

Within the ongoing discussion about data sharing, more replications, more disclosure, etc., Leif Nelson, Joe Simmons, and I are trying to make a simple, modest, and voluntary proposal targeted to individual authors who are already on board with the idea that disclosure of details of data collection and analyses is necessary for properly interpreting scientific evidence.

 

The idea is that rather than convert skeptics, we can start by enabling those who are already on board to signal the greater confidence we should place on their properly obtained findings.

 

To this end, we are proposing that authors voluntarily add the following 21 word statement to their papers.

“We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”

 

You can read our full pitch, with some visual aids, for using these 21 words here.  It will come out soon in SPSP’s Dialogue  newsletter.

 

Ideally, you will consider adding those 21 words in your next paper,

 

Best,

 

Uri

 

 

 

----------------------------------

Uri Simonsohn

Associate Professor

The Wharton School

http://opim.wharton.upenn.edu/~uws

 

[William Gunn]

I hope this takes off. The Reproducibility Initiative is considering developing an opt-in framework for authors and tracking the article-level metrics for papers which opt-in vs. those which don't, so having a standard phrase which allows us to skip asking an author in the first place would help our efforts immensely.

[Brian Nosek]

It is in at least one paper already:

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2155175

though we got it down to 19 words 

"We report how we determined our sample size, all data exclusions, all conditions, and all measures in the study."

Take that Simonsohn!

[Michael Cohn]

Is the exact text of the statement still open for discussion? I might
suggest that it include something about disclosing all a priori
hypotheses or distinguishing a priori from exploratory ones. It seems
like disclosing all measures is useful mostly as a proxy for that. But
reporting all measures isn't helpful if we don't know which ones were
the specific subjects of the hypotheses. And conversely, we shouldn't
look with suspicion on researchers who ran a lot of additional
measures as long as they're clear about which ones were exploratory or
for archival purposes, and which ones they were really putting their
money on.

- Michael


================================
Michael A. Cohn, PhD
Osher Center for Integrative Medicine
University of California, San Francisco

[Eric-Jan Wagenmakers]

@Michael: I agree that the distinction between exploratory and
confirmatory analyses is very important for the way we draw
conclusions from data. The only way to ensure that this distinction is
acknowledged is through preregistration, as far as I'm concerned. But
maybe the proposed statement is a step in the right direction.

@Uri and Brian: I find the text "We report how we determined our

sample size, all data exclusions (if any), all manipulations, and all

measures in the study." slightly confusing; I first read this
incorrectly as "We report how we determined our sample size, how we
determined all data exclusions (if any), how we determined all
manipulations, and how we determined all measures in the study".

So maybe "We report all data exclusions (if any), all manipulations,
and all measures in the study, as well as how we determined our sample
size."?
[I should add again that the Likelihood Principle and the Stopping
Rule Principle (e.g., Berger & Wolpert, 1988) dictate that when we
monitor the Bayes factor, we do *not* need to determine sample size a
priori. But I guess that "we set out to monitor the Bayes factor until
we find compelling evidence or until we run out of patience, whichever
one comes first" will also suffice.]

I am a little concerned about the wiggle room that the formulation
gives with respect to what defines "a study". Suppose I run three
experiments in a single session; are these part of the same study? I
don't think so. I believe it is honest to indicate which other
experiments were part of the same test session, but I don't think the
results of those other experiments should be reported.

Cheers,
E.J.

--
********************************************
Eric-Jan Wagenmakers
Department of Psychological Methods, room 2.09
University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands

Web: www.ejwagenmakers.com
Email: EJ.Wage...@gmail.com
Phone: (+31) 20 525 6420

“Man follows only phantoms.”
Pierre-Simon Laplace, last words
********************************************

[Timothy Bates]

It's hard to define "all manipulations", and so the 19 words might be more credible. As many of our problems appear to be arising in the manipulations, though, I wonder if one might add "and the blindedness of these conditions". 

Or to make a very simple statement that people who have manipulated the results will not wish to include, perhaps just "We have revealed all decisions that impacted the significance of these findings".

[Ben Blohowiak]

I am partial to the at-a-glance Research Facts label à la the USDA Nutrition Facts label, and would probably be an easy sell on linking it in some overt fashion to a whistleblower hotline or web address; that might give nervous critics some sense of reassurance regarding consequences for fraudulent claims.

Simonsohn seemed to emphasize in the start of this thread that the purpose of a disclosure statement (however exactly worded) is to advance the cause of properly interpreting scientific evidence; hypothetically, communities of practice in which researchers report things like how they determine sample sizes, data exclusions, and which measures they use are communities in which the evidence that researchers generate is more likely (or at least more able) to be interpreted well--presumably by others, though perhaps even by those same researchers. (An enlightened self-interest cause for this may exist, however latent at present.)

Although concerns seem to be on the rise regarding procedures whereby the significance of findings may be determined, transparency in arriving at given p-values and "the proper interpretation of evidence" are not inherently synonymous. If we assume that the former contributes to the latter, my guess is that while we may have our wishes for an invisible hand supporting more clean calculation met by the kind of culture suggested by this sort of disclosure as it is implemented, the onus remains on those seeking greater heuristic or theoretical insight to "go past the p," as it were, which may involve things such as greater public reflection on the measures used and upon the creation of constructs that they are designed to capture information regarding. I can imagine the OSF including as standard such spaces for public reflection for each project.

[Etienne LeBel]

I was similarly slightly confused by the wording used by Uri and Brian and hence prefer E.J.'s slightly revised version:

"We report all data exclusions (if any), all manipulations, and all measures in the study, as well as how we determined our sample
size."

Cheers,

Etienne.

--

 *******************************************

 Etienne P. LeBel, Ph.D.

 Post-doctoral Fellow

 Department of Psychology

 Social Science Centre, RM 7312

 The University of Western Ontario

 London, Ontario, CANADA N6A 5C2

 

http://publish.uwo.ca/~elebel

[Joshua Hartshorne]

I think it's a great idea to model good behavior rather than badger other people about bad behavior.

Just a question: isn't it already obligatory to mention any subject or item exclusions? I always thought it was, and I've heard many other people say so as well. And certainly almost every paper I read describes subject and/or item exclusions, and I always thought this description was meant to be exhaustive. 

Maybe this is something that varies by sub-field?

[Russ Clay]

Hi,

 

Quick question for the group:  Beyond the disclosure statement, would an appendix or a footnote be the most efficient way to state any manipulations, additional procedural tasks, measures, etc. that we want to report for the purpose of full disclosure? Particularly in the case of exploratory analysis on an existing dataset, reporting all of this in the Method section of the manuscript may get a little cumbersome and may be distracting to the main purpose of the paper, but at the same time, we want to portray the research methodology accurately.

 

So, I guess this is more of a style question, but has anyone else found a good way to manage this type of reporting?

 

Thanks,

 

Russ

[Ben Blohowiak]

Russ, my guess is that this could be an issue of granularity in description. One could be accurate in characterizing one's exploratory analyses in general terms in the Method section such that the "what" of the exploration is disclosed even if the more detailed particulars of the "how" remain essentially unknown to readers until they visit footnotes, as you offered, or perhaps until they visit hyperlinked and time-stamped nodes of the OSF itself.

[Brian Nosek]

An important journal in psychology is considering a standard disclosure format that would accompany all submissions and final reports (as a web-portion of the article rather than taking up print pages).  I'll find out if this can be shared to gather comments and critique from the group.