Deserialization and validation of sampler/optimizing parameters

[Allen B. Riddell]

Hi everyone,

Here's a proposal for a change to the PyStan/RStan/CmdStan interface
that would eliminate a lot of code duplication. It also happens to be an
easier change than any of the proposed Stan 3.0 API changes
(https://github.com/stan-dev/rstan/wiki/Interfaces-3.0-Spec). I'd
welcome any comments.

---

Sampler and optimizing parameters (e.g., adapt_gamma, stepsize_jitter,
...) are gathered and validated by RStan, PyStan, and Stan using nearly
identical code. It would be advantageous if checking to see if
parameters make sense occurred in one place in the Stan C++ library.
Interfaces (RStan/PyStan/etc) could provide a set of arguments to Stan,
Stan could validate the arguments and then return a coherent StanArgs
instance (or throw an informative exception). This change would require
modest changes to existing code.

RStan provides an example of what the StanArgs class might look like in:
rstan/rstan/inst/include/rstan/stan_args.hpp.

One challenge is that (de)serializing C++ classes directly is impossible
because C++ lacks reflection. On the other hand, there are some JSON libraries
out there, including one that's built from Boost, that would make the work of
constructing an instance easier. In this scenario, PyStan/RStan would submit a
prepared set of arguments for validation in JSON. CmdStan might even use this
interface directly since command-line arguments are already strings.

For some additional background, here's an example from PyStan and RStan of
duplicated code (stan_fit.hpp):

```
if (model.num_params_r() == 0 && algorithm != Fixed_param) {
throw std::runtime_error("Must use algorithm=\"Fixed_param\" for model that has no parameters.");
}
```

and CmdStan:

```
if (model.num_params_r() == 0 && algo->value() != "fixed_param") {
<< "Must use algorithm=fixed_param for model that has no parameters."
```

The main reason to consolidate validation of arguments is that keeping track of
default values (when the user doesn't provide a value for, say,
`stepsize_jitter`) in 3+ different places creates a situation where it's easy
for a developer to make a mistake and to have one interface fall out of sync
with the others.

Thanks,

Allen

p.s. Estimate of the lines of code dealing with validating sampler/optimizing arguments:

3% of lines of code in `rstan/rstan/R`
25% of lines of code in `rstan/rstan/inst/include/rstan`
6% of lines of code in `pystan/*.py`.
30% of lines of code in `pystan/*.hpp`.

[Bob Carpenter]

We absolutely need to do something along these lines to
get rid of all the duplicate code.

I believe Daniel and Michael are slowly working their way
toward that kind of design, though maybe with some code
object rather than something like JSON.

The big question in my mind is how to write the interfaces
on the language side so that the calls look idiomatic from
Python, R, etc.

- Bob

P.S. In case you need a JSON parser, Mitzi and I wrote one as
part of Stan that works with callbacks (like a SAX parser
for XML). Using the one in Boost looked like a huge pain
and I enjoy writing parsers. Nobody's ever plumbed it through,
but we were planning to allow it as an alternative data format
for initialization and data.

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Allen B. Riddell]

Bob,

I'd be happy to help out with this effort. Not knowing that much about
previous efforts to do this kind of thing in C++ projects, my initial
thought was to have a function like:

int stan::gm::marshal_stan_args(json_string, empty_stan_args)

with Stan throwing an exception if it doesn't work. (C++ exceptions are
easy to handle in Python.) Alternatively, à la stan::gm::compile

int stan::gm::marshal_stan_args(&std::cerr, json_string, empty_stan_args)

but I'd rather have explicit feedback to the interface about an error
than simply writing to std:cerr and returning -1. The main reason for
this is that some of these new web-based interactive environments (like
IPython notebook) do not really have std:cerr. Perhaps it could return
json with information about success or failure? Or populate another
struct like StanArgsError? (If only C++ could return multiple
arguments!)

Thanks,

Allen

[Allen B. Riddell]

Hi Bob,

Here's a C++11 JSON library that doesn't look too painful to work with:

- https://github.com/nlohmann/json

and I suppose you know about the Boost-derived JSON Spirit

- http://www.codeproject.com/Articles/20027/JSON-Spirit-A-C-JSON-Parser-Generator-Implemented

Best,

Allen

On 02/23, Bob Carpenter wrote:

[Michael Betancourt]

stan::services::argument_parser basically does this already
-- it accepts a string of flattened arguments, checks their
validity, and returns an informative error on failure. The idea
is that once command has been refactored any of the
interfaces can call it by provided call backs to handle things
like output and error messages. This assumes that we
use the hierarchical argument structure, which is very much
unsettled amongst the developers.

[Bob Carpenter]

We can't use C++11 yet.

In addition to what Allen sent (which is a parser using
Boost that's not part of Boost), there's also some stuff
within boost itself:

http://www.boost.org/doc/libs/1_55_0/doc/html/boost_propertytree/parsers.html

I have a very strong preference for SAX-like parsers
(use callbacks) over DOM-like parsers (build up an abstract
tree in memory). The problem with the DOM-like parsers is
that they duplicate memory. A second problem is that errors
are going to be runtime (I'm sort of surprised they didn't use
Boost's variants lib to eliminate that, though that would make it
harder to use). The parser using Boost that
Allen links to is DOM-like as is the one built into Boost's
property trees.

We also need something with a BSD license (blame me for this---I
really don't like GPL and want to keep Stan as BSD --- I can live
with PyStan being GPL for now, but I'd strongly prefer it to be
BSD so that it's not poisonous to inclusion in proprietary libs
or other BSD software [compatiblity only works one way]).

I also want something that's well tested and maintained going
forward, which none of the other libs I could find were.

Mitzi and I wrote a pretty vanilla JSON parser that is SAX-like,
involves no dependencies, has callbacks of the types we need, and
has been pretty well tested.

- Bob

[Bob Carpenter]

As Michael says, they're very close to that now.

I completely agree on exceptions over writing to std::cerr.
Nothing's going to take the latter approach.

If you need multiple returns, you have two options:

1. define a struct to hold them, or use pair<Type1,Type2>.
In C++, pair is so lightweight it doesn't introduce any overhead
at all in terms of memory.

2. pass in one or the other as a mutable reference to be set.

- Bob

[Allen B. Riddell]

Hi Michael,

Well, this is clearly the direction to go.

One question: how would stan::services work with PyStan/RStan when those
interfaces want to provide initialization data directly (e.g., without
referencing a file)? It was this situation that led me to think about
JSON in the first place.

I'm also not clear on the callback approach -- Does RStan use that
approach anywhere? Returning a pair or throwing an exception makes sense
to me in this case.

Thanks,

Allen

[Bob Carpenter]

JSON is also just a string. It's easy to pass
a string and then turn it into an input stream.
It's what RStan does when users pass in the Stan
program as a string.

- Bob