Startup costs for Dallinger, Empirica, Zooniverse, & Pushkin

[jkhart...@gmail.com]

What would you say is the time required to start using your platform, given some experience with programming but not necessarily with the language you use for your program?

Josh

[jkhart...@gmail.com]

For Pushkin, I think the answer is a moving target. We've been writing a command line interface (CLI) that lets you get a basic website and a very basic experiment set up in literally half an hour. We're working on experiment templates that are downloadable through the CLI which would really only require you to give a list of stimuli and set a few parameters. We've got templates for lexical decision, grammaticality judgment, and self-paced reading in alpha and should  be part of the official release within a few days. 

If you are familiar with jsPsych, you can basically write any jsPsych experiment you want. Learning the basics of jsPsych reportedly usually takes undergrads about a day or two. 

But there are a lot of features that aren't yet part of the CLI. So we have code for subject feedback like what Lauren Rutter described for testmybrain.org, but right now you'd have to integrate it yourself. Same for the forum and the user dashboard and some of the gameification we've used at gameswithwords.org. Also, the biggest thing is that the CLI doesn't cover deploying to a web server (like on AWS or Heroku), though I expect we'll have that done by early next week. 

For very fancy things ... Pushkin is open-source, so my hope is that advanced users will find it easier to build on Pushkin than write their own website from scratch. And that they will contribute code back to Pushkin so that it gets more powerful for everyone. 

[James Houghton]

I'm a strong python programmer, but hadn't used Javascript or done any web programming until I started using Empirica to design my experiment. It took me a week (of focussed effort) to get the first prototype of my experiment running and played by my lab-mates. (Then it took me most of a year to finish designing the experimental manipulation, and to finalize the parameters of the experiment, and preregister.)

Since then, we've been developing tutorials and materials to help folks get started quickly. If you have a simple experiment, you can follow the tutorials to get going in an afternoon.

In my experience, programming the experiment was easy compared with designing the science itself. And that's how it should be! I would suggest to anyone choosing a platform that they should choose whichever has features most suited to their research question, and then learn how to use it, rather than using whatever is the easiest to get started with.

[jwsu...@gmail.com]

Here's a cartoon of what the timeline of learning Dallinger looks like:

(first 15 minutes) Hey look, I found the Dallinger documentation and I'm starting to work through getting it set up on my system.

(next 4 hours) My god, Python packaging is annoying. Most of this is going swimmingly well, but there are a few things I needed to look up and why do I have 4 versions of Python installed in different places on my computer?

(next 15 minutes) Wow, I ran a few transmission-chain demos on my local system and this is great!

(next 15 minutes) Now I'm tweaking the configuration file and running variations on the demo.

(next 15 minutes) Now I'm running a transmission-chain experiment online and getting data.

(next 1 minute) I can replicate that experiment with no problem!

(next 1 hour) Let's see what other demos there are and play around with them.

(next 6 months) The capabilities of Dallinger have led me to think up new interesting experiment designs that nobody has ever run on Dallinger before. I have my own extremely specific set of requirements and cannot be flexible about any of them. I am learning the ins and outs of Dallinger in order to implement this new kind of experiment that may even involve creating new functionality in base Dallinger.

Our goal is for most people to be productive, happy users of the system without needing to get to the last step. In practice, most people do want to modify the demo experiment in some substantial way, and we've run 3-day workshops that are enough for someone with some basic programming experience to make meaningful changes. Making big changes (e.g., implementing a complex networked experiment with a new task and custom rules) will take some substantive knowledge of Python programming and more than a couple days using Dallinger. 

[jkhart...@gmail.com]

@Jordan that seems about right. So something we've been thinking about is doing hackathons to basically get/help people to write the code for functionality that they want. I just haven't had time to organize one, particularly with the pandemic going on. 

[a.sim...@gmail.com]

For the Zooniverse platform, my estimates are as follows:

- To create a small private prototype, if your data (image/video etc) is ready, you likely only need a day

- To go from a prototype to a beta test, I would guess you would need at least a month to identify an optimal task combination, create appropriate help text/tutorial, and set up an aggregation pipeline

- To go from beta test to a fully 'launched' project, you will need at least two weeks to a month (possibly much more depending on the results of the beta) to address any feedback from volunteers, process the data from the beta test, and schedule a launch day with the Zooniverse team

I should also note that programming experience isn't strictly necessary, but there is a python CLI for managing projects if that is preferable for a team.