stan-as-a-service?

[Daniel Lee]

Allen, you might want to move relevant parts of the other conversation to stan-dev. Or at least coordinate with Jon.

---------- Forwarded message ----------
From: Jon Zelner <jlz...@columbia.edu>
Date: Wed, Jul 1, 2015 at 9:27 AM
Subject: stan-as-a-service?
To: Daniel Lee <bea...@alum.mit.edu>, Rob Trangucci <robert.t...@gmail.com>, Bob Carpenter <ca...@alias-i.com>

Hi guys - 

I've been experimenting a bit with Docker and am thinking of putting together a stan-specific Docker image on top of the official base R one. Just wanted to check in with you guys to make sure this isn't something you've already done? 

Mostly I'm doing this to avoid bit-rot on R packages, etc in the time between sending a paper out and getting it back for revisions or to be able to revisit an old analysis. But I'm also toying with the idea of making a simple stan server that just takes stan code, input data and config options as JSON and sends back model output just to not kill my battery while working remotely.

-Jon
--

Jon Zelner
Health and Society Scholar
RWJF Health and Society Scholars Program
Columbia University
http://www.jonzelner.net

[Sebastian Weber]

stan-as-a-service: Including ODE problems???

[Krzysztof Sakrejda]

On Wednesday, July 1, 2015 at 9:37:05 AM UTC-4, Daniel Lee wrote:
> ---------- Forwarded message ----------
> From: Jon Zelner <jlz...@columbia.edu>
> Date: Wed, Jul 1, 2015 at 9:27 AM
> Subject: stan-as-a-service?
But I'm also toying with the idea of making a simple stan server that just takes stan code, input data and config options as JSON and sends back model output just to not kill my battery while working remotely.

I've been playing around with some JSON-based configuration for modeling scripts (https://github.com/sakrejda/context) loosely based on the discussions flying around about for Stan's configuration API and I would be interested in working on a server/JSON interface for Stan, but not before August.

Krzysztof

p.s.-there's also a stale pull request of mine that I need to update once I have a little time but I haven't forgotten about it. Likely late July.

[Bob Carpenter]

I think this would be a great idea --- Allen has been suggesting
we just set Stan up with a protocol buffer interface (though
JSON could work, too) --- presumably with a simple REST API
that'd make it easy to access from a variety of clients.

Would the data be included in the request format or attached
separately to the http request?

Have you considered protocol buffers instead of JSON?

- Bob

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[jzelner]

To me the advantage of JSON over protocol buffers is the ease of getting from a webpage or some other front-end API to the server w/JSON, plus you lose the overhead of the protocol buffer compiler.

My original thought was to do a request-reply type interface using RZMQ so that the requesting process (i.e. on my laptop) would block while waiting for the server reply. My primary motivation for doing this would be to preserve the order of a make-based buildchain that looks something like:

raw data -> data munging in R -> run stan model -> post-process stan -> 'compile' to figures & Latex.

-Jon

[Allen B. Riddell]

I think JSON might be problematic if someone has 2000 iterations of
10,000 parameters. But starting with JSON probably wouldn't hurt.

[Allen B. Riddell]

Hi Daniel, Jon,

People are free to experiment with this experimental Stan cloud service.
(It's on a very slow cloud VPS.)

http://45.55.246.82:8000/

Dockerfile is here:

https://github.com/ariddell/stan-cloud

Happy to help out with this stuff more once I get PyStan 2.7 done.

Best,

Allen

[jzelner]

Yeah - I think JSON would be better for the request end than the reply; agree that it would get very messy/inefficient to be throwing around large output files in JSON.

I think in an ideal world, the reply would be a URL to the traces stored somewhere, e.g. in S3 or just on the server. That way if you had some kind of distributed workflow, you could pass that reference around rather than having a ton of traffic up and down for large output.

One component of this kind of server implementation might also be an efficient database backend for post-processing. For example, one of the things I did like about working with PyMC was the ability to store the output of models in an HDF5 database.

-Jon

[Bob Carpenter]

The issue in coding output in something like JSON
is how to stream output. The output is a row for
each draw and a column for each value recorded per
draw (parameters, lp__, diagnostics).

1. If we don't stream, it's natural to represent the
data column-wise, perhaps even with structured values.
That makes extract very easy.

This also fits in with Jon's idea to just return a link.
I like this idea. We could conceivably automatically
generate graphical output this way and store it on the
server, too.

Julia is also using HDF5 --- I only heard about it last
week. Anyone think it's worth doing for Stan?

2. If we do stream, we probably want to start by sending
parameter names and dimensions, and then for each draw,
just return a vector. That'd be about the same cost as
for now with CSV. The client could then convert that
back to something with more structure or aggregate a
bunch of draws to implement extract.

We could stream on the back end (server side) to create
the file to return for (1). We can easily stream through
compression.

For (1), we could also compress; that's much harder with
(2) because the data's by column and we couldn't maintain
compression memory across draws without a lot of work (at least
I don't know packages that'll let you do that).

- Bob

[Michael Betancourt]

Regarding 2, go look at the writer refactor!

[Krzysztof Sakrejda]

On Thursday, July 2, 2015 at 12:49:08 PM UTC-4, Michael Betancourt wrote:
> Regarding 2, go look at the writer refactor!

Does that refactor have an informal milestone more specific than "future"?

[jzelner]

The only knock on HDF5 is that it can be a bit of a pain to get installed, at least the last time I tried. But once you have it up and going, it's great; pretty close to in-memory speed, much faster to load than from CSV. Plus if this was part of a Stan docker image, the issue of installation wouldn't be a big one.

Also, there's no reason something like this couldn't be compatible with the current CSV-based output, i.e. instead of writing a running model into the DB, just post-process it into it, since the benefit is mostly in access time rather than write time.

It would also make it easy to bundle the input data with the output (a perennial issue at least for me), since it has support for collections of data objects with heterogeneous links. A bit like a NoSQL kind of database but without all the unnecessary features and much, much better compression and access.

[Bob Carpenter]

> On Jul 2, 2015, at 12:49 PM, Michael Betancourt <betan...@gmail.com> wrote:
>
> Regarding 2, go look at the writer refactor!

Agreed --- it's abstracting the core of the one-draw-at-a-time
write.

[Bob Carpenter]

Write and read are always going to be horrible if
what's being written and read is an ASCII dump in decimal
of an intrinsically binary number. Binary's the only solution.

Usually, the write and read hit from this isn't the bottleneck
in MCMC, but it can come up in optimization, where the I/O eats up
the majority of the time. That's why Marcus has been urging us to
think about binary I/O pretty much as soon as he wrote an efficient
optimizer.

- Bob

[Jeffrey Arnold]

Regarding hdf5, for what it's worth, PyMC2 has a hdf5 backend (along with sqlite, flat files, ...). See https://github.com/pymc-devs/pymc/tree/master/pymc/database. But for some reason, they have either dropped, or not ported over this backend to PyMC3.

[Jeffrey Arnold]

Not as ambitious as the Stan-as-a service, but I put together a few Dockerfiles for rstan (built on top of rstudio) and cmdstan while I was trying to scale out my computing. Here's the repo on github: https://github.com/jrnold/docker-stan

-Jon

[Ben Goodrich]

On Tuesday, July 21, 2015 at 12:54:23 AM UTC-4, Jeffrey Arnold wrote:

Not as ambitious as the Stan-as-a service, but I put together a few Dockerfiles for rstan (built on top of rstudio) and cmdstan while I was trying to scale out my computing. Here's the repo on github: https://github.com/jrnold/docker-stan

This line in the Dockerfile

'rstan::set_cppo("fast")'        

does not do anything with rstan >= 2.7.0. You should instead execute the R code here

https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started#configuration

including at the end

cat("\nCC=clang", "CXX=clang++", 
    file = M, sep = "\n", append = TRUE)

I would also put

library(rstan)
rstan_options(auto_write = TRUE)
options(mc.cores = parallel::detectCores())

into ~/.Rprofile .

Ben

[Jonah]

I see that you also included loo. Cool! If you want you can have it install the loo package from CRAN now, but no big deal as the github version should work fine.

On Tuesday, July 21, 2015 at 12:54:23 AM UTC-4, Jeffrey Arnold wrote:

[Jeffrey Arnold]

Thanks, I adjusted the Dockerfiles to account for the changes in rstan 2.7.0. But I thought it better to use rstan::rstan_options rather than loading rstan in the Rprofile.

Oops, I forgot that loo is on CRAN. And of course I included loo; loo would not be the one package that I left out.