Re: Important Update to Registrations on OSF

[Stanka Fitneva]

Hi,

I am concerned about part of this announcement, which I assume everybody
received. For many of my colleagues, adopting (pre)registration and other
practices promoted by OSF hinges on the condition of their ideas not being
scooped. I personally won¹t even try to persuade grad students that they
should not be worried about their ideas being scooped. At least in my area
- developmental - given the availability of kids in the area, it¹s
extremely hard to complete data collection for a study in a year, let
alone also code and analyze the data. For the replication that we
contributed to RPP we worked full steam for a year. Not a matter of
diligence. Just not that many 4-year-olds around and too many studies
drawing on that population! For many, the concern would extend to being
able to conduct not a single study but a set of studies that would make a
publication.

So setting the term of the embargo to 1 year undermines my plans to
require my students to register their (our) studies. I suppose a lot of
thought has gone in choosing this period but is it possible to revise it
so that it is acceptable to researchers in similar situation like mine?
I¹d suggest setting the embargo period to 4 years as most PhD programs
assume that this is how long active data collection takes. Or perhaps
study administrators can choose the embargo period themselves?

Cheers,
Stanka

Stanka A. Fitneva, PhD
Department of Psychology
Queen's University
Kingston, ON K7L 3N6

phone: 613-533-2363






On 2015-05-28, 4:18 PM, "opensciencefra...@osf.io"
<opensciencefra...@osf.io> wrote:

>Dear Stanka Fitneva,
>
>You are receiving this message because you have an Open Science Framework
>(OSF) account. We are writing to let you know of a change to our
>registrations that may affect how you choose to use that feature.
>
>Registrations create a frozen, time-stamped snapshot of the project and
>its contents. Registrations are used for archiving significant events in
>the research lifecycle, such as pre-registering an analysis plan prior to
>data collection, and they cannot be deleted. As early as June 8, 2015,
>upon creating a registration, you will choose to make it public
>immediately or set an embargo period, which will keep the project
>contents private for up to one year, other than the title, description,
>creation date, embargo period, and contributors. Users will be able to
>end the embargo early, but they will not be able to stop the registration
>from becoming public after the embargo. Instead, to remove a public
>registration, users will have to retract it - similar to retraction of a
>published article. The same information that is available during the
>embargo would remain, but all other project content would be inaccessible
>and replaced with a retraction notice.
>
>This change removes the possibility of having a permanently private
>registration. The change also adds features to promote persistence and
>preservation. OSF registrations will receive persistent identifiers (DOI
>and/or ARKid), and files will be archived to preserve the content.
>Finally, when one project administrator registers the project, all
>contributors are notified of the registration and all project
>administrators will have 48 hours to cancel the registration before it is
>made permanent. We believe that these changes better align the
>registration function with the expectations of what registration should
>do.
>
>For example, Sam initiates a registration of the private project "Destroy
>One Ring" and does not set an embargo period. Frodo, another
>administrator on the project, receives a notification but worries that
>making the registration details public too soon would undermine the
>project. He cancels the registration within 48 hours, but does wish to
>pre-register the project aims and preserve the content for posterity.
>Frodo then initiates a new registration with a 12-month embargo period.
>Frodo and Sam work diligently to complete the project during the embargo
>period.
>
>How will this affect you? The new functionality will not apply to
>existing private OSF registrations - they can remain private. If you
>have any public OSF registrations or ever decide to make those private
>registrations public, you will not be able to make them private. If you
>want to avoid this, then you must set your public registrations to
>private before June 8, 2015. As of June 8, 2015, all new registrations
>will become public immediately or following their embargo date. As such,
>treat registration like journal publication - it is difficult to go back.
>
>Please let us know if you have any questions or would like help
>addressing your current public registrations: sup...@osf.io
>
>Regards,
>
>The OSF Team
>Center for Open Science
>
>We may send you occasional Service-related emails that you may not
>opt-out of (e.g. changes or updates to features of our Services that have
>security or privacy implications, technical and security notices, account
>verification); this is one of those emails.

[Rebecca Willén]

I agree with Stanka. One year is not enough.

I also think it's a bit too early to start employing an embargo. For me, it has been a struggle to get my supervisors' permission to preregister. I don't think I would have convinced them if I couldn't promise that we would keep it private until we ourselves choose to make it public - and that we have the possibility to always keep it private. Because they are bad people? No, because they have not really heard about it before and they don't know of anyone who has done it and they don't have the time to get into what it means or why to do it. It has therefore been important that I could ensure them that everything would be exactly as before - that we have full control and can use it before publication only if we choose to. This way of thinking is of course not optimal, but it's a start.

I totally understand the purpose of the embargo, and I do think it's a good thing. I just think it's a bit too early.

Best,
Rebecca


-----Ursprungligt meddelande-----
Från: openscienc...@googlegroups.com [mailto:openscienc...@googlegroups.com] För Stanka Fitneva
Skickat: den 29 maj 2015 03:59
Till: openscienc...@googlegroups.com
Ämne: [OpenScienceFramework] Re: Important Update to Registrations on OSF

--
You received this message because you are subscribed to the Google Groups "Open Science Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openscienceframe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Nosek]

Hi all --

Tanya, thanks for raising this discussion. A similar one started on
twitter and in my email inbox. I am heading into all day meetings,
but want to put a few things out quickly to help foster the
discussion.

(1) I erred in not having the public discussion about moving
registrations to embargoes prior to our announcing the new policy. My
apologies for that. I am not sure why I had this particular failure
of perspective-taking, but I did. This discussion should have
happened first, rather than the abrupt announcement. A stupid mistake
on my part.

(2) The length of the embargo is easy to change, so I am very happy to
hear discussion about the appropriate embargo period. The one option
that we cannot maintain is "indefinite".

(3) There are two reasons to remove the indefinite option. The first
is that we have been developing an archiving system that will make
registration very robust for preserving the content of the project
upon registration. With that we are making a big commitment to
storage of those data. Allowing indefinite private registration would
make it too easy for anyone to use OSF as an archiving service for any
data (i.e., non research related). Such widespread use would rapidly
break the bank. The second is that this moves aligns OSF with the
standard function of registries, to surface the work that was done (or
intended to be done). We can dig into that more if need be, but
assuming that we are okay with moving to "eventually public" then the
question is:

(4) What should be the maximum embargo period? Some relevant thoughts
for discussion:

(a) We selected a year as a starting point because it is (socially)
easier to make an embargo longer than shorter. There isn't anything
magical about that date.

(b) Besides minimizing storage misuse issues, the main argument for
shorter periods is to have the registry function in its fullest
capacity (increasing discoverability of research) in the most timely
manner possible.

(c) The main arguments for longer periods are potential to be scooped
(whether it is a real or perceived risk) and potential for details of
high-profile human participant studies to be revealed to the
participants before they are complete. It isn't clear how likely the
latter concern is in practice, but like scooping - whether a real or
perceived risk - the effect on people registering their studies is the
same (i.e., not doing it).

(d) Whatever the embargo period, there may be occasions that a user
simply will not use registration because of the need for the material
to go public. In those cases, OSF still provides the date and time
logging of the project history through ordinary, private use. Those
project need not ever go public. In such cases, the user wouldn't
choose to register because they don't want to commit to making the
material public.

There is certainly more that can be productively discussed, but that
may at least provide some fodder. My apologies again for not having
this discussion BEFORE the policy announcement. I can't fathom why
that was not obvious to me yesterday. And, my availability today may
be significantly limited by the meeting events, but hopefully a good
conversation can be had without me.

[I am taking the liberty of adding some others to the cc that have
initiated some discussion of this in other places. Those folks - the
listserv may not be able to receive your replies unless you are a
member of it. If you care to do so, join here:
https://groups.google.com/forum/#!forum/openscienceframework

[Christopher Chambers]

Thanks Brian.

I am copying below the important part of the email I sent to Brian this morning. If embargoes are going to be applied, I think it is important that the investigator can adjust the date to accommodate uncertainties in completion dates (I am not concerned about scooping but rather about participants in a long-term study becoming aware of study hypotheses). I've made a suggestion for how this could be achieved. Interested in any feedback.

best, Chris

---

My main problem with the new OSF policy is that the 12 month embargo is too short and not sufficiently flexible -- as a case in point, my group is soon to embark on a 4-year online study via the Guardian (in ~50,000 people) to test the efficacy of different forms of cognitive control training on eating behaviour (see http://sites.cardiff.ac.uk/cubric/new-european-award-on-self-control-and-the-brain/). To ensure the integrity of the trial it is important that the specific hypotheses underlying the different conditions are not readily available to the participants, so we would not be able to reveal the protocol until data collection is complete (after ~4 years). Added to this is the complication that we don't know precisely how long it will take to collect the data -- it might take 3 years, or 4 years, or even 5 years depending on Guardian site traffic and other unpredictable factors (like attrition, etc). So it wouldn't be possible to even fix a definite embargo date in advance. Notwithstanding these uncertainties I am committed to pre-registering the project.

Therefore, this is what I would suggest: when researchers register a protocol they are required to set an embargo date but they can choose any date they want. Once the project is registered there is some publicly visible part of it (perhaps an ID number and a registration date, plus the embargo date) but no further information can be seen - so no title or other text. The researcher can then adjust the embargo date as they see fit, and this adjusted embargo is added to the publicly visible component whenever an amendment is made. So you could imagine a registration that is visible as "IDXXXX, Registered XX/XX/XX, Original Embargo XX/XX/XX, 1st Amended Embargo XX/XX/XX + stated reason, 2nd Amended Embargo XX/XX/XX + stated reason etc.." This would give researchers the flexibility to use OSF for longer term projects while also preventing permanent private pre-registrations. In theory, of course, a researcher could ensure an effectively permanent private pre-registration in two ways - by setting a release date in the far future (but this would look ridiculous to peers as the embargo date would be publicly visible), or by setting a reasonable future date but continually pushing it back (which poses an administrative burden on the researcher).

If OSF doesn't revise this policy then it will mean that we can't use it to pre-register any of our longer term (and, for us, most important) studies, which would be a real shame.

------------------------------------------------------------
Chris Chambers BSc PhD CPsychol FBPsS
Professor of Cognitive Neuroscience
School of Psychology
Cardiff University
CF10 3AT
United Kingdom

Tel: +44 (0)29-208-70331
Email: chamb...@cardiff.ac.uk
Official site: http://psych.cf.ac.uk/chambers
Blog: http://www.theguardian.com/science/head-quarters
Twitter: @chrisdc77
------------------------------------------------------------

[Daniel Lakens]

I understand the OSF cannot keep storing large amounts of data for ever. I have a backup of my entire Vinyl collection on the OSF in .flac format of 4.2 terrabytes where I was thinking if it might be a bit too much (if you want to be a collaborator on the project, let me know). Just kidding. But really, you are providing a valuable service. If there is a cost, another option would be to explore how much it would cost, and ask people to pay like X dollars per MB they are storing in a preregistration file. 

The second reason is more fundamental. The goal to open all research is worthwhile. However, it only works if it is enforced across the board. Now, the choice is: enter an AllTrials like program where you solve publication bias (great ideal) and pre-register, or don't enter an AllTrials like program (less work!) and don't pre-register (at the OSF, in any case). I'm simply affraid that given that pre-registration is not even necessary to get published, people will rather skip the extra effort and not pre-register than immediately go all the way and share all their pre-registrations. Note that this will not even make a dent in publication bias anyway - people will still do and publish research that is not pre-registered. Furthermore, we have excellent solutions for people who don't want to contribute to publication bias - they can submit to Comprehensive Results in Social Psychology (or similar pre-registration journals). So for the people who want to go AllTrials, there is already a solution. For the people who are still on the fence about open science, this might push them off to the wrong side.

All this will be much less of a problem if the embargo period is flexible, with a maximum of let's say 5 years (perhaps for free pre-registrations, with a pay for 10 year embargo plan). Just some thoughts.

Daniel

[Macartan Humphreys]

Dear all: I share the concerns about the embargo.

Below some suggestions that were in a mail I sent Brian yesterday.

(a) having a really long, eg 10 year universal embargo lifting policy; such long periods could be discouraged but still available

 

(b) allowing negotiated exceptions for privacy periods of more than some period; eg to get around the storage concern groups requiring more than 10 years could make a special request to the governance board (is there one?), sharing with the board, in confidence, the content of the studies

 

or perhaps (c) have automatic lifting of embargoes after every 12 months unless requests for continued gating are made --- this is to make sure that inactive studies turn public.

b and c could be coupled with modest costs or conditions to deter scammers and keep any review burdens small.

Cheers,

Macartan

 

[Rogier Kievit]

I think the suggestion by Chris would cover concern I have (not so much for myself but for the many people I am trying to shepherd towards OSF and related initiatives. My worry is that the one year default sends the (implicit) message that there is one 'type' of research (between subject experimental psychology), ignoring eg longitudinal, large samples, patient samples and more. Inclusivity in this regard is important I think...

[Ruben Arslan]

I want to +1 what Rebecca Willén said, the option to make private pre-registrations made it easier for me to convince the others and gave them time to wrap their heads around it. Now they're proud of having done it and might go public from the go next time, but this helped get them onboard.

Specifically we've got three private pre-registered studies online now. Two of them would have just been made public a month ago. For one the data collection is still ongoing (a diary study, so longitudinal and difficult to meet the planned N).

At the moment, I think it's already clear that a Registered Report is a better badge than a publicly pre-registered study (how many others do you have) and better than an indefinitely privately pre-registered study (just a little better than the status quo of hypothesising in a grant etc.). An embargoed privately pre-registered study would simply be somewhere between public and indefinitely private.

Regarding scooping:

While I feel the timeframe would be too short for some of our projects, and I've been involved in projects that took more than 3 years, I feel that 2 years should be enough of a headstart to prevent scooping, after all if anyone starts a longitudinal study two years after us and manages to publish earlier, it's our own fault for sitting on the data.

Regarding participants gleaning hypotheses:

I think this is a better reason for exceptionally long embargoes (but it would make sense to make this possible e.g. only for experiments with blinding), but I'm also reading something between the lines here that I don't like either:

If the OSF won't do indefinite private data storage, I have a problem. I cannot be convinced that it's safe to put diary data containing information about recent extra-pair sex online, public for anyone. Mask all you want, the problematic unmasker is that person's partner and that person has access to all kinds of information (like when the study was filled out, can reasonably guess what personality profile to look for etc.). 

There's other use cases like this, but this one is real and I think it has the virtue of making it very clear: Not all data can be masked and sometimes the consequences of unmasking would be potentially drastic.
However, I think it'd be good practice to backup our data in a shareable format on the OSF, so that I can easily share it with people who have a legitimate interest in it, cannot "lose it", etc.

Regarding updates to registrations:

Public clinical registries require regular progress updates (i.e. "recruiting"/"completed"/"has results"/" and inactive projects, where contributors missed an update are marked "unknown"). The OSF currently doesn't do this and even if embargoed trials went public, if they did so simply quietly, nobody would know (at least that's how it usually goes in clinical trials and they have more of a semblance of a science police, i.e. Cochrane etc, than the rest of science).

So, embargoes don't really go far enough. Part of what Goldacre & co. criticise is that it's far too easy to have these trials languish in the registry. Nobody checks or calls anybody on it. 

Such trials and studies should stick out like a sore thumb. It's should be no shame to update saying data collection is ongoing, but it should be a shame to let a registration just sit there with no update as to what happened. Maybe you lost interest, and are preventing others from going into it, because they're waiting for your results to inform their planned study, don't want to duplicate efforts? Maybe the results weren't what you predicted and you're hiding it?

I think studies without progress updates should be made very public.

Best wishes,

Ruben C. Arslan

Georg August University Göttingen

Biological Personality Psychology

Georg Elias Müller Institute of Psychology

Goßlerstr. 14

37073 Göttingen

Germany

Tel.: +49 551 3920704

https://psych.uni-goettingen.de/en/arslan

https://formr.org

[Brian Nosek]

Greetings all --

Thanks much for the discussion and feedback. We reviewed this and
direct discussion and settled on starting with an embargo window of up
to 4 years.

We really appreciate all of the thoughtful discussion about it.

Regards,
Brian



P.S. Here is my preregistered prediction. After 3 years of
experiencing preregistration, the incentives to get credit for idea
origination will dominate the concerns about scooping leading to a
strong majority of registrations being public at inception.