Fwd: [stan-users] Piggypacking Stan's output format and tools

[Cole Monnahan]

[forwarded thread from stan-users]

Thanks Bob, inline comments below...

On Wednesday, August 31, 2016 at 3:20:22 PM UTC-7, Bob Carpenter wrote:

We have a developer's list, which is more relevant for this
kind of thing.  If you join, we can give you posting permissions.

You can even use our code (as long as you cite it) --- it's BSD
licensed.

I wasn't planning on copying any code -- just using the R packages as is. I will obviously make sure everything I do is legal/gracious.

 

Was there anything you needed in RStan that's not in ShinyStan?

I thought the stanfit processing was in RStan?

 

Either way, you're better off just reading the data in via
the much simpler Coda format (mcmc.list) and using the monitor()
function and for anything fancier, using ShinyStan, which will
also take simple Coda-formatted input.

The problem with this is I want to have the HMC specific diagnostics like step size, acceptance probabilities, tree depths, divergences, etc. Once you use shinystan to diagnose a NUTS chain it's hard to go back... 

We'll be changing the exact format of the CSV files and may even
move to JSON or protocol buffers---no guarantee on stability at
the CSV level in the future.

 

Fair enough. Maybe it makes more sense to write some R code to create my own stanfit objects from output then? Will that class be stable in the near future?

- Bob




> On Aug 31, 2016, at 11:50 PM, Cole Monnahan <mon...@uw.edu> wrote:
>
> This may be more relevant for the Stan development team, and if so please forward along or tell me where to resubmit.
>
> I'm interested in adding HMC functionality to an existing piece of software (migrating models to Stan is too time intensive for the moment). My goal is to try and mimic the basic functionality that Stan provides, specifically static HMC & NUTS. If I can get 75% of the functionality and speed that would be a huge step forward.
>
> I don't want to reinvent the wheel so my plan is to model the algorithms (inputs/outputs) to match Stan and then use all the nice features for diagnostics and inference already built in to Rstan.
>
> So I was thinking it would be a good idea to write output to .csv files like cmdstan does, and then coerce them into stanfit objects. Then I'd be able to use the methods for that class, such as monitor, pairs, and even shinystan. I imagine some of the slots wouldn't be relevant, such as @model and @stanmodel, but those could be populated with dummy values. It seems that if I went this approach I could just use read_stan_csv directly.
>
> My question is whether this is an advisable track to take? Or would there be a simpler way to approach this? If it's a reasonable approach, is there an easy way to see the format of the cmdstan csv files? It's not immediately apparent by digging through the cmdstan repo.
>
> Thanks,
> Cole
>
> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Krzysztof Sakrejda]

Depending on what you want to do you might be able to use more of Stan than you think. Could you describe your project a little better? There are so many solid Stan models now maybe what you want us already implemented.

[Cole Monnahan]

Hi Krzysztof,

I'm talking about a program called AD Model Builder which is used mainly for maximum likelihood estimation of models in the fisheries community. It's designed with more direct access than Stan (although I admit I haven't used cmdstan), so the user is basically writing C++ directly. This allows users to come up with very generic models (kind of like rstanarm). Think of it as Stan written in the 80s by a single person to solve specific fisheries issues. For instance, this manual describes a common model that has probably thousands of combinations of used options and is 10K lines of C++. It is used widely for resource management around the world due to its flexibility. This is not the kind of model that anyone would be recreating in Stan anytime soon.

So I figured it would be much easier to add NUTS to the source of ADMB so it is available for all existing models. ADMB coupled AD and static HMC a decade ago, so the skeleton is there to build in NUTS. I just don't want to get to a point where users can utilize the amazing post-processing/diagnostic tools in Stan. 

On Thu, Sep 1, 2016 at 11:40 AM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:

Depending on what you want to do you might be able to use more of Stan than you think.  Could you describe your project a little better? There are so many solid Stan models now maybe what you want us already implemented.

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.

[Bob Carpenter]

> On Sep 1, 2016, at 7:56 PM, Cole Monnahan <mon...@uw.edu> wrote:
>
> [forwarded thread from stan-users]
>
> Thanks Bob, inline comments below...
>
> On Wednesday, August 31, 2016 at 3:20:22 PM UTC-7, Bob Carpenter wrote:

> ...

> You can even use our code (as long as you cite it) --- it's BSD
> licensed.
>
> I wasn't planning on copying any code -- just using the R packages as is. I will obviously make sure everything I do is legal/gracious.

I meant from stan-dev/stan for doing the HMC. That's
all C++. That is, it might be easier to wrap your models
and use our HMC implementations.

We set the legal bit up on purpose to make it easy for
people to use however they want. We do appreciate citations,
though, in addition to the legal requirement to cite us
as copyright holders and licensers of the original code.

> Was there anything you needed in RStan that's not in ShinyStan?
>
> I thought the stanfit processing was in RStan?

I think ShinyStan does everything RStan does for posterior
analysis.

> ... Maybe it makes more sense to write some R code to create my own stanfit objects from output then? Will that class be stable in the near future?

Probably not. And the stanfit object holds Stan models
and config and other things you don't need.

Your best bet's to just use the standard Coda format, which
I'm sure we'll keep supporting for the sole reason that it
lets us do comparisons to other systems using our more
conservative R-hat and n_eff estimators.

- Bob

[Bob Carpenter]

> On Sep 1, 2016, at 9:15 PM, Cole Monnahan <mon...@uw.edu> wrote:
>
> Hi Krzysztof,
>
> I'm talking about a program called AD Model Builder which is used mainly for maximum likelihood estimation of models in the fisheries community.

Yup. Just went to your conference :-)

What I think would be most difficult in writing ADMB programs
is that the library of built-ins is relatively sparse compared
to Stan. And it doesn't have all the efficiency bells and
whistles for matrices and probability densities up to a proportion.

There's also TMB, which is more like a combination of
INLA and RStanArm.

> It's designed with more direct access than Stan (although I admit I haven't used cmdstan),

CmdStan is just an alternative high-level interface from
the command line. The Stan programs are the same. You can
take output from CmdStan and run it through RStan's or ShinyStan's
diagnostics and visualizations.

> so the user is basically writing C++ directly. This allows users to come up with very generic models (kind of like rstanarm).

You can write C++ directly for Stan, but you have to do
it with templates rather than the direct build-in of
ADMB types as you guys do, because all of our programs assume
generic templated model code and do the autodiff through
functionals applied to derived functors.

> Think of it as Stan written in the 80s by a single person to solve specific fisheries issues. For instance, this manual describes a common model that has probably thousands of combinations of used options and is 10K lines of C++. It is used widely for resource management around the world due to its flexibility. This is not the kind of model that anyone would be recreating in Stan anytime soon.

I'd be very nervous about using 10K lines of C++ nobody
understands. But it might translate to much less Stan code
because our abstractions are higher level.

> So I figured it would be much easier to add NUTS to the source of ADMB so it is available for all existing models. ADMB coupled AD and static HMC a decade ago, so the skeleton is there to build in NUTS.

Yes, you guys were a few years ahead of us with autodiff and HMC.
We pretty much worked from scratch, though, because we couldn't
find general enough autodiff implementations for what we wanted.

> I just don't want to get to a point where users can utilize the amazing post-processing/diagnostic tools in Stan.

You just need matrices of the draws. And the best tools for that
are in ShinyStan right now and that's where the active development is.

- Bob

[Cole Monnahan]

Bob, see inline below. -Cole

On Thu, Sep 1, 2016 at 3:36 PM, Bob Carpenter <ca...@alias-i.com> wrote:

> On Sep 1, 2016, at 9:15 PM, Cole Monnahan <mon...@uw.edu> wrote:
>
> Hi Krzysztof,
>
> I'm talking about a program called AD Model Builder which is used mainly for maximum likelihood estimation of models in the fisheries community.

Yup.  Just went to your conference :-)

Hah yes of course I remember you being there. Was just giving some background.

 

What I think would be most difficult in writing ADMB programs
is that the library of built-ins is relatively sparse compared
to Stan.  And it doesn't have all the efficiency bells and
whistles for matrices and probability densities up to a proportion.

Yeah I'm not trying to compete with Stan. I'm trying to get NUTS functional for some really important models built using entrenched and outdated software.

 

There's also TMB, which is more like a combination of
INLA and RStanArm.

> It's designed with more direct access than Stan (although I admit I haven't used cmdstan),

CmdStan is just an alternative high-level interface from
the command line.  The Stan programs are the same.  You can
take output from CmdStan and run it through RStan's or ShinyStan's
diagnostics and visualizations.

> so the user is basically writing C++ directly. This allows users to come up with very generic models (kind of like rstanarm).

You can write C++ directly for Stan, but you have to do
it with templates rather than the direct build-in of
ADMB types as you guys do, because all of our programs assume
generic templated model code and do the autodiff through
functionals applied to derived functors.

I don't think this is possible or worthwhile considering the state of the ADMB source code. It's way easier to get NUTS into it.

 

>  Think of it as Stan written in the 80s by a single person to solve specific fisheries issues. For instance, this manual describes a common model that has probably thousands of combinations of used options and is 10K lines of C++. It is used widely for resource management around the world due to its flexibility. This is not the kind of model that anyone would be recreating in Stan anytime soon.

I'd be very nervous about using 10K lines of C++ nobody
understands.  But it might translate to much less Stan code
because our abstractions are higher level.

Undoubtedly. We're probably at least a decade away from moving off ADMB just due to the inertia of the community. I'm trying to graduate in the next year, however, so I'm looking for a shorter path here. 

> So I figured it would be much easier to add NUTS to the source of ADMB so it is available for all existing models. ADMB coupled AD and static HMC a decade ago, so the skeleton is there to build in NUTS.

Yes, you guys were a few years ahead of us with autodiff and HMC.
We pretty much worked from scratch, though, because we couldn't
find general enough autodiff implementations for what we wanted.

> I just don't want to get to a point where users can utilize the amazing post-processing/diagnostic tools in Stan.

You just need matrices of the draws.  And the best tools for that
are in ShinyStan right now and that's where the active development is.

The issue I have with using coda's classes is that they don't have the NUTS  diagnostics like step size, tree depth, and divergent. Is that not true? That's key when using ShinyStan.

- Bob

[Bob Carpenter]

See inline.

> On Thu, Sep 1, 2016 at 3:36 PM, Bob Carpenter <ca...@alias-i.com> wrote:
>
> > On Sep 1, 2016, at 9:15 PM, Cole Monnahan <mon...@uw.edu> wrote:
> >
> > Hi Krzysztof,
> >
> > I'm talking about a program called AD Model Builder which is used mainly for maximum likelihood estimation of models in the fisheries community.
>
> Yup. Just went to your conference :-)
>
> Hah yes of course I remember you being there. Was just giving some background.

Me, too :-)

> What I think would be most difficult in writing ADMB programs
> is that the library of built-ins is relatively sparse compared
> to Stan. And it doesn't have all the efficiency bells and
> whistles for matrices and probability densities up to a proportion.
>
> Yeah I'm not trying to compete with Stan. I'm trying to get NUTS functional for some really important models built using entrenched and outdated software.

Understood. I'm just saying that might not be the best way
to spend the time to get these really important models functional
and may not lead to the most efficient resulting system. It
will keep everything in ADMB, though. And I understand that
many people don't want to change systems they use.

> There's also TMB, which is more like a combination of
> INLA and RStanArm.
>
> > It's designed with more direct access than Stan (although I admit I haven't used cmdstan),
>
> CmdStan is just an alternative high-level interface from
> the command line. The Stan programs are the same. You can
> take output from CmdStan and run it through RStan's or ShinyStan's
> diagnostics and visualizations.
>
> > so the user is basically writing C++ directly. This allows users to come up with very generic models (kind of like rstanarm).
>
> You can write C++ directly for Stan, but you have to do
> it with templates rather than the direct build-in of
> ADMB types as you guys do, because all of our programs assume
> generic templated model code and do the autodiff through
> functionals applied to derived functors.
>
> I don't think this is possible or worthwhile considering the state of the ADMB source code. It's way easier to get NUTS into it.

That's what I meant when I said you could use our code.
You can use our NUTS code directly rather than trying to code
it all. You'd just need to go down to the MCMC library level
rather than using higher-level interfaces. But even those aren't
really designed well for use with anything other than one of our
generic model classes.

> > Think of it as Stan written in the 80s by a single person to solve specific fisheries issues. For instance, this manual describes a common model that has probably thousands of combinations of used options and is 10K lines of C++. It is used widely for resource management around the world due to its flexibility. This is not the kind of model that anyone would be recreating in Stan anytime soon.
>
> I'd be very nervous about using 10K lines of C++ nobody
> understands. But it might translate to much less Stan code
> because our abstractions are higher level.
>
> Undoubtedly. We're probably at least a decade away from moving off ADMB just due to the inertia of the community. I'm trying to graduate in the next year, however, so I'm looking for a shorter path here.

You might be surprised how fast people can move if you find
something faster and more robust. And you don't need to get
everybody to move, just the important people. But obviously
up to you and you've decided already that it's impractical, so
I'll drop it.

...

> The issue I have with using coda's classes is that they don't have the NUTS diagnostics like step size, tree depth, and divergent. Is that not true? That's key when using ShinyStan.

I'm not sure what the story is with those. Maybe you could write
another more specific message to get the RStan' developers attention.

This stuff should all have generic input formats and not require
a stanfit object, which builds in all sorts of Stan specifics. Jonah's
been pretty swift to update ShinyStan and I think that's your best
path forward.

- Bob

[Cole Monnahan]

Hi Bob,

Thanks as always for your time and thoughts. I know this isn't Stan per se, but I do think my work will garner interest in the algorithms and Stan itself moving forward.

I'm not a C++ programmer so I can't say how easy it would be to interface ADMB and Stan. I'll ask around with the ADMB maintainers, but it's hard for me to imagine that it would be easy. In any case, the first step is to demonstrate how much better NUTS is than RWM for existing models.

Good point about ShinyStan. I'll look into that and discuss with Jonah.

Thanks,

Cole

- Bob

[Daniel Lee]

Although our model class / concept isn't full documented, you should be able to write an adapter to take existing ADMB code and make it look like what Stan's expecting. If you pull that off, you'll be able to run Stan's algorithms.

If you look at this branch on Stan: feature/issue-1751-service-methods (https://github.com/stan-dev/stan/tree/feature/issue-1751-service-methods)

if you wanted to run just our default NUTS algorithm, you can run this function:

https://github.com/stan-dev/stan/blob/feature/issue-1751-service-methods/src/stan/services/sample/hmc_nuts_diag_e_adapt.hpp

You'll need to create a Model that conforms to what the algorithms are expecting, a few things for shuttling input and output back and forth, and the arguments for calling the function.

Daniel

[Bob Carpenter]

I don't think that'll work. I'm pretty sure ADMB code builds
in their autodiff types rather than being templated.

- Bob

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Bob Carpenter]

Just be careful implementing NUTS. It's a *very* subtle
algorithm and has undergone many changes since the paper.
The adaptation is a critical component and Michael's changes in
Stan 2 were significant improvements. All the little details
that ensure detailed balance must be done right or you'll get
the wrong answers. So you'll want to be careful to
do extensive tests that you generate the right output distributions
with the right interval coverage.

What language are you going to write this in if not C++?

- Bob

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Michael Betancourt]

Right — in order to use the Stan sampler code you’d have to find
a way of hacking the gradient(model…) functor to expose ADMB’s
gradient calculation. Given the complexities of this, it would be
easier to collapse all of the NUTS code into a single file (or sets
of files), but then you’d have to understand all of our sampler
abstractions.

In other words, given the constraints that have been presented
I think that hand writing the latest NUTS implementation over
ADMB will be the only way to get anything done within a year.
It will probably be fragile and error prone, but given all of the
compromises that are being discussed that would seem to be
acceptable.