A repository of experiments?

[Jon]

Maybe this is a venue to discuss the idea of a repository of
experiment code.

A couple of people (including Alex H) on the psychopy forum have
suggested that it would be good to have a place to upload experimental
code and materials, to allow someone to fully reproduce their study.
It would serve a few purposes:
- makes a study genuinely replicable, given that methods are likely
not enough
- publicises the experiment that you've run
- provides a starting point for new users of a piece of software to
build an experiment

The site would be agnostic to the software used (people could tag the
study by which software it ran on). I think it differs from
psychfiledrawer.org by focussing more on the experimental materials
than any findings. Now I'm thinking about it, maybe it could simply
form a section within that site though.

BUT when I suggested it to some colleagues they were concerned that;
- it could lead people to run studies that they didn't actually
understand
- any errors in studies would propagate rather than be removed by
the next person to try and program it

It seems an interesting question, albeit academic since we don't have
funding to create the site anyway. But maybe it's too much and too
different to try to include with this workshop.

Just thought I'd raise it.

Jon

[Alex Holcombe]

Yes I think the repository of experiments would make science go
significantly faster. It would also make psychophysics more
influential, in that non-expert-programmer psychologists could more
easily use our paradigms. Sure there would be some occasional
catastrophes like people running studies they didn't understand (as
happened here http://neuroskeptic.blogspot.com.au/2011/06/eagle-eyed-autism-no.html
when famous autism researchers got hold of Dakin's software) but I
think the net effect will still be faster progress of science and more
rigorous work (on average) by non-psychophysicists.

The point you make that most experiments today are not easily
replicable without the code is a very important one to me. I'm hoping
that research funders will wake up to this (indeed awareness of
replicability problems in psychology anyway is growing rapidly, thanks
for mentioning www.psychfiledrawer.org) and start to require (with
reasonable exceptions) more that we make our code available.

Without researcher funder mandates, however, I'd expect that few
people would post. Most working scientists are
-slightly or very embarassed by their code
-don't want to take on the burden of fielding questions like "the code
didn't run on my machine"

Often these kinds of new web resources only take off if there is an
existing community website which people are visiting anyway, so that
the site is already getting traffic, causing people to be prompted to
consider posting. For that reason

For believers in open science, their commitment to open science may
trump these concerns (I recently posted some code on my university's
electronic repository associated with our Curr Bio paper), so
hopefully the increasing awareness of the importance of open science
and the problem of replicability will boost the numbers willing to
post. For this reason I'd favor somehow building it into the Psychopy
and Psychtoolbox discussion forums.

Besides research funders, journals can also play a role. Some
progressive journal like JoV could require that code be posted as
supplementary material (not sure where) unless you have a very good
reason not to post it. There's a partially successful movement in
computational sciences to require posting of code, google Victoria
Staddon (sp?) to read about that.

[Jonathan Peirce]

I don't think I'd advocate forcing people to upload their code, and I
don't think any journal would support that for now. The effort to the
user to get their code into readable shape, and the embarassment of
others reading it, is too high for most people.

I think it would have to start as a voluntary process and see how it
develops. Those with the guts to expose their code stand to benefit by
having their work used more than others as the basis for future
experiments. And, potentially, things like PsychoPy's builder interface
will alleviate people's stress about the quality of their code
(similarly Presentation scripts might be simple enough that people
aren't as worried?).

Jon

--
Jonathan Peirce
Nottingham Visual Neuroscience

http://www.peirce.org.uk

This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

[Ariel]

Hi,

I think that this is a great idea, but I don't necessarily see the
need to build anything special-purpose for the sharing of
psychological experiments and see no reason not to already start doing
that. There are plenty of venues that allow people to publish code (in
addition to your own departmental website or lab wiki). For example, I
have used github to make some experimental code (in Matlab/
Psychtoolbox) publicly available: https://github.com/arokem/motion_th.
Initially this was done to facilitate a remote collaboration, but now
this is also something that I can point people to if they would ask me
about the code used to generate my experimental stimuli. Obviously,
this is a location that has a lot of traffic, and could start getting
more traffic from the vision science community.

To pre-empt comments people might make about git being "too high a
barrier on non-technically-minded people", I will point out this:
http://mac.github.com/. Besides, if you can't understand a simple tool
like git, how do you expect to understand the visual system? ;-)

At the risk of echoing some of the remarks made above, I address the
concerns that were raised about sharing code:

1. It could lead people to run studies that they didn't actually
understand.

We should be so lucky that people would actually want to replicate and
run our studies. My code has been on github for more than two years.
It's been forked once and has two "viewers" except for me. The first
part of the sentence ("It could lead people to run studies") seems
like a strong enough pro, that the concern about the second part
shouldn't be as strong. It's not like people can't run studies they
don't really understand if they write their own code to do it. The
visual acuity pixel resolution error could have been made by someone
writing their own code to display this stimulus.

2. Any error would propagate rather than be fixed.

I am not sure I follow the logic of that one. I would think that more
eyes on the code and more people using it would mean less errors in
the long run, rather than more.

3. Scientist are embarrassed to show their code

And they're not embarrassed to be publishing papers based on this
code?! Am I the only one who thinks that's upside down? As pointed out
by David Donoho: "An article about computational science in a
scientific publication is not the scholarship itself, it is merely
advertising of the scholarship. The actual scholarship is the complete
software development environment and the complete set of instructions
which generated the figures."

4. Don't want to take on the burden of fielding the "code doesn't work
on my machine" question.

Who says you have to field anything? Use those emails to train your
spam filter. Or put less aggressively, this again seems like an
advantage of making code public - you'll gain new collaborators!

5. Journals don't accept code as supplementary material:

Actually, JoV seems to accept code as supplementary material, at least
according to their instructions for authors:
http://www.journalofvision.org/site/misc/publication.xhtml#SM

Cheers,


Ariel

On Feb 8, 2:26 am, Jonathan Peirce <jonathan.pei...@nottingham.ac.uk>
wrote:

> I don't think I'd advocate forcing people to upload their code, and I
> don't think any journal would support that for now. The effort to the
> user to get their code into readable shape, and the embarassment of
> others reading it, is too high for most people.
>
> I think it would have to start as a voluntary process and see how it
> develops. Those with the guts to expose their code stand to benefit by
> having their work used more than others as the basis for future
> experiments. And, potentially, things like PsychoPy's builder interface
> will alleviate people's stress about the quality of their code
> (similarly Presentation scripts might be simple enough that people
> aren't as worried?).
>
> Jon
>
> On 07/02/2012 20:07, Alex Holcombe wrote:
>
>
>
>
>
>
>
>
>
> > Yes I think the repository of experiments would make science go
> > significantly faster. It would also make psychophysics more
> > influential, in that non-expert-programmer psychologists could more
> > easily use our paradigms. Sure there would be some occasional
> > catastrophes like people running studies they didn't understand (as

> > happened herehttp://neuroskeptic.blogspot.com.au/2011/06/eagle-eyed-autism-no.html

> > when famous autism researchers got hold of Dakin's software) but I
> > think the net effect will still be faster progress of science and more
> > rigorous work (on average) by non-psychophysicists.
>
> > The point you make that most experiments today are not easily
> > replicable without the code is a very important one to me. I'm hoping
> > that research funders will wake up to this (indeed awareness of
> > replicability problems in psychology anyway is growing rapidly, thanks

> > for mentioningwww.psychfiledrawer.org) and start to require (with

[Christopher Gillespie]

I think a big issue would actually be duplication and tracking of the
programs and the hypothesis that they were testing. I'd imagine that a
well designed website would actually negate the problem of propagating
errors. An obvious way but inevitably controversial would be having a
flagging system.

For something like that to work I think the crux is actually more to
do with website design with well thought out systems for grouping (of
like experiments and hypothesis), flagging, comments and probably
something to show up if code is being left un-commented.

In other words, I think the only way it could work is by starting it
with a small group of people that could set up a kinda forum culture
of valid criticism and gradually expand it, rather than a wikipedia
like model of anyone can edit and change. It would also get around the
problem of people being scared of posting their code - I also think
that inevitably that with any such idea like this that essentially
what would happen (either directed or undirected) is that a standard
model for experiments would creep in. Actually accepting that this was
part of process rather than letting it occur as a consequence may be a
good, if not controversial idea.

I think the error propagation would naturally iron itself out if you
had a few hard core programming style people. They may even think it
as fun weekend work, so involvement of other interested people would
probably be beneficial.



C

[Jonathan Peirce]

Home hosting
-------------
Although it's certainly possible to host your own experiment code on
your institutional or personal web pages, or on github, that only ticks
one of the boxes for this concept; the one about replicability of a
single finding. People have to read the paper (and probably the fine
print) to find that the code is available online and then go find the
code. In Ariel's case, the reason that hardly anyone has viewed the code
is largely because they don't know it's available.
The idea of the centralised repository is that people can browse it
looking for things. Say I want to do a study on Inhibition of Return, I
might go to the library to find an existing study using something
similar. If what I do ends up being closely related, I (hopefully) cite
the original authors. Or, by browsing experiments that I can just
download and 'have a go', maybe I end up finding some new type of study
that I hadn't previously found useful. I guess the point is that there's
an educational aspect to this as well as a replicable science aspect.

People's fears
-------------
I'm sure we agree that it's a shame that people are embarassed to share
code, but they are, nonetheless. Partly just because they didn't write
the code with the aim of sharing it; they wrote it 'to get the job done'
rather than to be readable or elegant. Either way I think the very vast
majoirty of people don't want to share their code, and forcing them to
do so via journals would be very heavily resisted.

Jon