Stan model cache on the Net (speeding up "compile" time)

[Allen B. Riddell]

Hi,

It occurs to me that there aren't that many Stan models out there in
heavy use (esp. in the classroom). Ignoring variable name permutations
and ordering perhaps there are about 500 that account for ~90% of use?
Why not just compile them all in advance and serve them to users in
binary form, à la CRAN or PyPI? Imagine how happy users would be if all
example and textbook models "compiled" in 2 seconds.

Here are some numbers: a very simple model takes 47 seconds to compile
on my computer but with my very slow internet connection it takes ~10
seconds to download the saved model (20M) from a server in New Jersey.
I bet on fiber one could download the model in 2 seconds. And fiber is
only going to get cheaper. Both R and Python have a mechanism for
packaging up a model that's already in use when people save a model to
disk.

Best,

Allen

p.s. Yes, a model would have to be compiled for each architecture ---
PyPI already does this with its binary "wheels". And one would have to
cryptographically sign the models. (One could package the public key in
the source code.)

[Allen B. Riddell]

It occurs to me that the greatest benefit of this feature is that most
users would no longer need to have a C++ compiler available if they had
an internet connection available---again, provided the cache were
suitably large (>1,000 models) and Stan models were made invariant to
variable ordering and variable name permutations.

Best,

Allen

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bob Carpenter]

Progress on making precompiled models easy to deploy would
be great. I don't know who uses them --- for me, the benefit
of Stan is being able to customize a model.

I somehow doubt that 90% of our use is in existing models,
but we could probably pick up more users with prepackaged
models.

Are you still using g++ or do you have a really slow computer
or is there some Python roadblock? Compiling a model takes
around 15--20s in RStan for me using clang++ on a 3-year old Macbook
Pro with a 2.3GHz i7 CPU.

- Bob

[Bob Carpenter]

What are you imagining those 1000 models are?
And how would people find them?

We couldn't put this out in an official (CRAN) R
package because of storage and build-time constraints
for packages.

- Bob

[Allen B. Riddell]

On 03/27, Bob Carpenter wrote:
> Progress on making precompiled models easy to deploy would
> be great. I don't know who uses them --- for me, the benefit
> of Stan is being able to customize a model.
>
> I somehow doubt that 90% of our use is in existing models,
> but we could probably pick up more users with prepackaged
> models.
>
> Are you still using g++ or do you have a really slow computer
> or is there some Python roadblock? Compiling a model takes
> around 15--20s in RStan for me using clang++ on a 3-year old Macbook
> Pro with a 2.3GHz i7 CPU.
>

Old computer, i5, and g++

[Allen B. Riddell]

On 03/28, Bob Carpenter wrote:
> What are you imagining those 1000 models are?
> And how would people find them?
>

In the social sciences the range of models used isn't that broad. I
suppose I'm just assuming that a significant chunk of our users are
implementing models that already exist (e.g., the horseshoe or a model
from a well-known paper in their field). One could also scrape the web,
github, and stan-users and compile every valid Stan model ever seen
there. Storage and bandwidth are cheap and getting cheaper.

> We couldn't put this out in an official (CRAN) R
> package because of storage and build-time constraints
> for packages.

Right, we'd have to use our own infrastructure. I think this could be
cheap(ish). At its simplest it could just be a static directory serving
files where the filename is the SHA256 hash of the compiled .hpp for the
model.

I can prototype this in PyStan this summer I think. It would be easier
if we did the RStan/PyStan API refactor first though.

Best,

Allen

[Krzysztof Sakrejda]

For ecologists you could also make the transition to Stan easier just from having standard GLMM's, 'animal models', and mark-recapture models that are specified by model matrix (from R and Python). Making variance partitioning easy on the interpretation end would help too but I think it's a separate issue from the time/need to compile models. I think really standard models would get most of the benefit and also make it easier to trouble shoot for people on stan-users.

Krzysztof

[Michael Betancourt]

Progress on making precompiled models easy to deploy would
be great.  I don't know who uses them --- for me, the benefit
of Stan is being able to customize a model.

+\infty

When building a model with someone new to stats in almost

all cases the most excited part is when they get to adjust the

model to their science instead of an arbitrary model.  Making

it easy for people to not have to think about their model does

no one any good (unless people can use precompiled models

for only three months and then we cut them off or charge them,

in which case we retain the incentive of easy entry without the

cost of people falling into routine models).

[Allen B. Riddell]

I don't think we disagree at all! I imagine the fetching of the
precompiled model would be completely transparent to the user---the
model is just identified by the hash of the model's C++ code. The cache
could hold 100,000+ distinct models. And of course if it's an unseen
model, you'd use the compiler (or, someday, have a nice, fast machine
compile it on demand in the cloud and add it to the cache). A side
benefit would be that we would know (via traffic stats) what models were
being used most often.

Model compile time is the only issue being addressed here.

Best,

Allen

[Andrew Gelman]

Good point regarding the troubleshooting.

[Ben Goodrich]

On Friday, March 27, 2015 at 9:02:31 AM UTC-4, Bob Carpenter wrote:

What are you imagining those 1000 models are?
And how would people find them?

We couldn't put this out in an official (CRAN) R
package because of storage and build-time constraints
for packages.  

There aren't official CRAN constraints on build time and the storeage is pretty small if there are no samples. This is basically what rstanarm is intended to be. It wouldn't be 1000 models but it could easily be 10. I'm not sure what the Python equivalent would be. I would say for now, put models that we like in the example-models repo somewhere.

Ben

[Bob Carpenter]

> On Mar 28, 2015, at 3:18 AM, Allen B. Riddell <a...@ariddell.org> wrote:
>
> ...

> The cache
> could hold 100,000+ distinct models.

While we could afford the storage of 100K * 10MB (that's
still only 1TB), it would present a difficult organization
problem. I'm uncomfortable officially distributing models
that we don't vet.

One place to start would be to go through the BUGS models and
make sure they're all up to our recommended practices.
That'll get us up to about 75 or something like that. Then
repeat 1500 times :-)

- Bob

[Krzysztof Sakrejda]

On Friday, March 27, 2015 at 6:42:30 PM UTC-4, Bob Carpenter wrote:

> > On Mar 28, 2015, at 3:18 AM, Allen B. Riddell wrote:
> >
> > ...
>
> > The cache
> > could hold 100,000+ distinct models.
>
> While we could afford the storage of 100K * 10MB (that's
> still only 1TB), it would present a difficult organization
> problem. I'm uncomfortable officially distributing models
> that we don't vet.
>
> One place to start would be to go through the BUGS models and
> make sure they're all up to our recommended practices.
> That'll get us up to about 75 or something like that. Then
> repeat 1500 times :-)
>
> - Bob

I would be much more likely to contribute models I'm interested in rather than trying to get some arbitrary BUGS-related collection working. The barrier for me contributing standardized models (in addition to time) is that there's no clear standard to follow... and so far no easy framework for vetting that a pile of models works. For me that was part of the reason for getting the X*b thing working well since so many models can be expressed that way:

http://www.amazon.com/Richly-Parameterized-Linear-Models-Statistical/dp/143986683X/

Krzysztof

[Bob Carpenter]

> On Mar 28, 2015, at 10:36 AM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>
> On Friday, March 27, 2015 at 6:42:30 PM UTC-4, Bob Carpenter wrote:
>>> On Mar 28, 2015, at 3:18 AM, Allen B. Riddell wrote:
>>>
>>> ...
>>
>>> The cache
>>> could hold 100,000+ distinct models.
>>
>> While we could afford the storage of 100K * 10MB (that's
>> still only 1TB), it would present a difficult organization
>> problem. I'm uncomfortable officially distributing models
>> that we don't vet.
>>
>> One place to start would be to go through the BUGS models and
>> make sure they're all up to our recommended practices.
>> That'll get us up to about 75 or something like that. Then
>> repeat 1500 times :-)
>>
>> - Bob
>
> I would be much more likely to contribute models I'm interested in rather than trying to get some arbitrary BUGS-related collection working.

So would everyone, which is why our example models are in
such a poor state.

> The barrier for me contributing standardized models (in addition to time) is that there's no clear standard to follow...

There are instructions on what we're looking for. See the top-level
README here:

https://github.com/stan-dev/example-models

I agree that some good examples would help. Feel free to be
the first. Basically everyone has looked at the instructions and
said it's too much work.

> and so far no easy framework for vetting that a pile of models works.

We just want one of our developers to do it informally. Daniel
had some stuff automated before, but there were objections from Michael
on the statistical tests, some suggestions from Andrew and Michael on how
to fix it, but it's just languished on the to-do list. We really need to get this
in for regression (of the software engineering type) testing of Stan itself,
not just for example models.

One big issue with testing is the variability in wall timing and ESS/iteration
on different runs.

> For me that was part of the reason for getting the X*b thing working well since so many models can be expressed that way:
>
> http://www.amazon.com/Richly-Parameterized-Linear-Models-Statistical/dp/143986683X/

Understood.

- Bob

[Krzysztof Sakrejda]

On Friday, March 27, 2015 at 7:43:30 PM UTC-4, Bob Carpenter wrote:
> I agree that some good examples would help. Feel free to be
> the first. Basically everyone has looked at the instructions and
> said it's too much work.

You got me there. I'm only likely to do it for new models or ones that are new to me. I could see it working as part of a class or some sort of Stan hackathon (someday...)

> > and so far no easy framework for vetting that a pile of models works.
>
> We just want one of our developers to do it informally. Daniel
> had some stuff automated before, but there were objections from Michael
> on the statistical tests, some suggestions from Andrew and Michael on how
> to fix it, but it's just languished on the to-do list. We really need to get this
> in for regression (of the software engineering type) testing of Stan itself,
> not just for example models.

I had an automated way of stashing models (without history) by hash and doing metadata in sqlite for a while too, although I never developed any way to a) check for updates or b) schedule updated/new models to run.

> One big issue with testing is the variability in wall timing and ESS/iteration
> on different runs.

Agreed, I don't think there's an easy solution to this in terms of generating pass/fail types of tests but I could see defining metrics to calculate and visualize (posterior-predictive check statistics would work for parameters at least).

Krzysztof