reproducible research with Docker, Rstan, and Gitlab CI

[Jon Zelner]

Hi Folks - 

Just wanted to put a small something I've been working on out there to see what folks think about it. 

This is a stab at making a template for a fully reproducible soup-to-nuts Stan-based project, that starts with input data and ends with a PDF containing figures and estimates derived from the model output. This repository contains all the code needed to re-generate a simple Stan example (fitting a 2-component finite mixture model) with a single command:

https://gitlab.com/jzelner/reproducible-stan

The ideas represented here are a crystallization of things that have been going on in a number of other projects of mine that rely on sensitive data and so can't be opened up in a way that is fully reproducible to anyone. There are three big reproducibility ideas in here, some newer than others, none of which are my own but instead represent a synthesis of some very powerful tools and ideas:

1) make: Everything is orchestrated using a Makefile that updates outputs as a function of their dependencies on each other. In some ways, this is the most powerful idea and a very old one. 

2) Docker: One of the challenges of reproducing any project (and Stan ones in particular for non-initiates) is having the relevant software installed locally to both run the Stan model and to do the requisite munging of model output. Docker provides a way to create a portable environment that can run everywhere without forcing users to go through the relatively complicated install process. For example, the example presented here relies on: R, assorted R packages (readr, ggplot2, dplyr, gridExtra), RStan, pandoc, TexLive, etc. The provided Docker image makes it easy to just download the whole environment and re-generate the model results without a complicated local install.

3) Continuous integration (CI): CI tools are commonly used in software engineering to make sure that small changes to a project don't break the whole thing. This project is hosted on Gitlab to take advantage of their easy-to-use and free Docker-based continuous integration service. This way, every time a commit is made to the repository, the Makefile is re-run and the model output (MCMC samples, figures and PDF writeup) are uploaded to the repository and are available for download.

More information (and all the code and output) are available in the repository. It's very much a work-in-progress and is meant more as a prod for more thought and development than anything else, so any feedback would be greatly appreciated!

-Jon

[Bob Carpenter]

Has anyone tried running this on Windows?

How big are the docker images? How big would it be without
R, LaTeX, etc? I'm wondering how feasible it would be to
distribute a working Stan system this way and then communicate
with it via protocol buffers. Then we could have lightweight
command-line, R, and Python wrappers.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Andrew Gelman]

I'll wait on these updates before blogging it, then.
A

[Allen B. Riddell]

Docker support for Windows is new-ish.
https://docs.docker.com/engine/installation/windows/

One issue with docker is that it runs as a privileged user by default.
By contrast, you can use a C++ compiler (and Stan) without being a
privileged user.

A

[Jon Zelner]

To answer Bob's question, the current rstan image is quite large at about 5gb.   It's built on top of the hadleyverse image from Dirk Eddelbuettel's rocker project (https://github.com/rocker-org/rocker), which has a base R install plus all the Hadley Wickham packages as well as a working TeX installation. I think the TeX stuff accounts for a lot of the bloat. Then I added pandoc and cURL and some extra LaTeX fonts. So there's definitely room to pare it down, especially for a stand-alone Stan-machine, particularly if it just ran cmdstan. 

Allen's point about running as privileged by default (b/c it needs direct access to the linux kernel) is definitely a sticking point, particularly for working in any kind of HPC environment. 

One interesting use case for folks who don't want to run locally that this could help to solve could be to create a server image that runs Stan-as-a-service, potentially taking models and data via an HTTP request and returning a link to the results. It might also be possible to link it to dropbox for something a bit more seamless from the user end? If it was well-packaged, a lab group could download it to a local server with minimal config.

I was thinking it might be easiest for me to write up a 3-part series (Makefiles, Docker, and Gitlab CI) on my relatively lightly-trafficked blog, and then you guys can either link to or post a synopsis or carbon copy of those on the big blog?

-Jon

[Andrew Gelman]

Sounds good to me.

A

[Bob Carpenter]

Well, at least 5GB is an upper bound.

The only reason we haven't built servers for a Stan server
is that we have no way to pay for the hosting. It should be easy to
set up a server-based version of Stan that operates via HTTP
or some other form of socket. We wouldn't need Docker for that,
though that might be a dev ops friendly solution.

- Bob

[Jon Zelner]

Yeah - I think the plus side to doing it via Docker is just deployment and not worrying about dependencies in the same way.

Also, here's the first installment of blog posts on reproducibility/re-generation. This really only gets through making R scripts executable, using command-line arguments in a coherent way and coordinating it all with Makefiles. But these ended up being necessary prerequisites to getting to Docker, which will be next:

http://www.jonzelner.net/statistics/make/docker/reproducibility/2016/05/31/reproducibility-pt-1/

-Jon

[Andrew Gelman]

Let me repeat that I'd still love to have this as a demo feature, even if it requires a password so we are the only ones who can run it.

[Jon Zelner]

Just in case it's of interest, here's a post on how to get started with Docker for RStan:

http://www.jonzelner.net/docker/reproducibility/2016/06/03/docker/

-Jon

[Jon Zelner]

Hi Everyone - 

Just wanted to let you know that the series of posts is complete, and that the last two show what I think is the coolest part, specifically getting RStan up and running in a Docker container and then using that to easily share and reproduce results for free on GitLab. 

The series starts here:

http://www.jonzelner.net/statistics/make/docker/reproducibility/2016/05/31/reproducibility-pt-1/

And the post on making changes online and auto-updating results using GitLab's continuous integration service is here:

http://www.jonzelner.net/docker/gitlab/ci/reproducibility/2016/06/06/gitlab-ci/

Andrew - this last one may get at your interest in a Stan server, since it performs some of the same functionality, although via pushes to a git repository hosted on GitLab rather than via an HTTP request. But this approach has the benefit of being totally free and not requiring users to set up any kind of infrastructure (and will work just as well in a private repository as in the public one this project is built off of). 

This is very much a work-in-progress, so any feedback would be greatly appreciated! Working on getting comments up and running on my blog, so in the interim email (j...@jonzelner.net) or stan-users is the best way to get in touch if you're interested!

-Jon