interfaces and command error-checking (Rstan/PyStan)

[Bob Carpenter]

I think one way to take the burden off the interfaces
checking consistency of commands is to break the monolithic
command-line call down into multiple little function calls.
In that situation, almost everything will be consistent.

R goes partway down this path, but not all the way, because
the stan() function still encompasses both NUTS and HMC.

For example, if the commands were

* stan_sample_nuts(...)
* stan_sample_hmc(...)
* stan_optimize_lbfgs(...)
* stan_optimize_bfgs(...)

(The names aren't important, so please don't get hung up here if
you think they should say "bayes" or "mcmc" or "(p)mle" or "map"
or "mode" or whatever.)

Then almost all of the built-in options will be legal. To work
this out in more detail for nuts, here are all the config options
(including model, which is implicit in CmdStan):

nuts(model (file | string | dynamic-object),
max_depth (int > 0),
metric (unit | diag |dense),
stepsize (double > 0),
stepsize_jitter (double in (0,1)),
num_samples (int >= 0),
num_warmup (int >= 0),
save_warmup (bool),
thin (int >= 1),
adapt(false | true
| dictionary(gamma (double in ?),
delta (double in (0,1)),
kappa (double in ?),
t0 (double in ?)),
data (var_dictionary),
init (double > 0 | var_dictionary),
seed (int in (0,2^32 - 1)),
output_file (file),
diagnostic_file (file),
refresh (int >= 0) );

That's it. Now if within Python or R, you have some simple
routines to validate ranges and validate the various values, you'd
be good to go without any complicated config construction or
callbacks. These routines could be used across the different
functions.

This doesn't specify defaults, but presumably that's also easy.

Adaptation is worth special attention, because it's
a big disjunction that's either boolean with defaults or
specifies overrides in terms of a nested argument.

The huge advantage is that the error messages will match what
the names are in RStan or PyStan.

I'm not sure if allowing the disjunctive types, like init
is a good idea in R or Python. If you have two different
arguments, then there are consistency requirements, so I'd
rather see that as a substructure, just as in CmdStan.

Then, to run, there needs to be a way to turn these
into a config C++ object that can actually be passed to
the Stan C++ to do the work, but that shouldn't be that
hard, either.

I'd like to compare how much work this would be to
marshaling a config object from a similar input in a dictionary,
converting to JSON, sending it to Stan C++ for validation,
then interpreting the results back in terms of the actual
argument names in PyStan or RStan.

This seems way more modular to me and doesn't require
any kind of config within Stan C++ --- it all becomes
a C++ structure that just runs and then CmdStan would get
all the config stuff that's currently in Stan C++.

- Bob

[Michael Betancourt]

Surprise! I fundamentally disagree.

The problem with splitting up the API into a bunch of small functions
is that we have no way to enforcing a consistent user experience
which will force users to learn each interface separately. We’re
already seeing this in RStan which is rapidly drifting from CmdStan
(and hence JuliaStan and MatlabStan) and taking PyStan with it.
It’s already a pain in the ass to do these workshops where I have to
relearn all of the idiosyncrasies of each interface in addition to all
of the usual work.

Then there’s the problem of actually implementing an interface,
which would now require the implementer to come up with a user
experience and piece together all of the modular components.

I am very much in favor of having a monolithic API route that
provides at least a marginal enforcement of a unified user
experience. Interfaces will always be able to screw it up with
enough work but the path of least resistance will be a similar
user experience everywhere.

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bob Carpenter]

> On Mar 18, 2015, at 8:45 PM, Michael Betancourt <betan...@gmail.com> wrote:
>
> Surprise! I fundamentally disagree.

Surprise, surprise, no surprise (Dr. Seuss-isms just
for Andrew).

> The problem with splitting up the API into a bunch of small functions
> is that we have no way to enforcing a consistent user experience

This is the key to the disagreement, and I disagree very
strongly.

I think the experience from R should be R-like, from
Python should be Pythonic (can't believe I just said that),
and from the command line, should be shellish.

The only way to make everything literally the same would be to have
the R user experience look like this:

> stan_make("foo.stan");

> foo("sample algorithm=hmc metric=unit_e engine=nuts max_depth=15 data file=model.data.R init=modelinit.R random seed=5 output file=samples.csv");

> stan_print("samples.csv")

No way of doing extract, no way of calling log prob directly,
no way of passing inits in as a function, no way of passing data
in as list, no plotting, etc. Basically nothing's that not
built into CmdStan.

I guess we could use list notation to make it a bit more R-like,
without straying too far from the command line you set up:

> foo(list("sample", algorithm=list("hmc",metric="unit_e", engine=list("nuts", max_depth=15), data=list(file="model.data.R"), init="model.init.R",random=list(seed="5")...);

Is this what you are envisioning? If not, would you mind elaborating
a bit on what you think the R user experience should be?

> which will force users to learn each interface separately.

I don't think many users are going to learn multiple Stan
interfaces, so I think that's a moot point other than for us,
as you note in the next paragraph.

> We’re
> already seeing this in RStan which is rapidly drifting from CmdStan
> (and hence JuliaStan and MatlabStan) and taking PyStan with it.
> It’s already a pain in the ass to do these workshops where I have to
> relearn all of the idiosyncrasies of each interface in addition to all
> of the usual work.

I feel your pain there. Why not just do the workshops using
CmdStan only? I'm doing one next week exclusively in RStan.
Then I believe Andrew, Daniel, and I will be doing a three-day
course using just RStan.

> Then there’s the problem of actually implementing an interface,

This is a separate issue.

To clarify, I'm *not* still advocating a modular approach on
the R side where we expose samplers, etc. You and Daniel
convinced me that would be too hard to maintain.

What I'm suggesting is having the interface provide an idiomatic
way of piecing together a C++ structure (including
writers, iostreams, etc.) that would be passed to Stan C++
and further validated on that side to make sure nothing untoward
happens.

Then no holds barred once the data gets back on the interface
side.

You'll never be able to agree with Andrew as to what the
default print should look like. You'll be stuck on scientific
notation before even getting going.

> which would now require the implementer to come up with a user
> experience and piece together all of the modular components.

Back to the first disagreement. I think that's a good thing!
Otherwise, we wind up with every interface looking like the
command line and being heavily limited in what it offers.

> I am very much in favor of having a monolithic API route that
> provides at least a marginal enforcement of a unified user
> experience. Interfaces will always be able to screw it up with
> enough work but the path of least resistance will be a similar
> user experience everywhere.

Could you please sketch out what you think the R user experience
would look like so as to match the one in CmdStan? Or Python's?

- Bob

[Daniel Lee]

How Bob laid it out is how I envisioned it. I would also like there to be a wrapper for all the small functions so it can be called from the top, like MIchael's plan, but it should be implemented through small functions.

[Allen B. Riddell]

I think there's a great deal everyone agrees on. Splitting up the
interface into smaller functions sounds good to me, but perhaps that can
be tabled until Stan 3.0 and the proposed reorganization of
RStan/PyStan.

As for the general question of how to provide parameters, I think
there's probably some middle ground we can reach. Those using Stan know
they are getting into a DSL, so we have *some* freedom to depart from
Pythonic and R styles. We already all agree, for example, that the
parameters should have the same names (i.e., "adapt_t0" and not
"adapt.t0"). I think we can make other small concessions here in order
to have an experience that is as consistent as possible. I want
switching between R and Python to be as painless as possible.)

I think there are two important things that the Stan C++ services
library should provide:

- Validation of config (e.g., is adapt_int_buffer positive)
- One source of truth about default values.

While promoting code reuse is the proximate goal, just having
RStan/PyStan and CmdStan on the same page about something as modest as
config validation would probably do more in the short term to move us
towards a consistent user experience than anything else I can think of.

Best,

Allen

[Allen B. Riddell]

On 03/18, Bob Carpenter wrote:

> Adaptation is worth special attention, because it's
> a big disjunction that's either boolean with defaults or
> specifies overrides in terms of a nested argument.
>
> The huge advantage is that the error messages will match what
> the names are in RStan or PyStan.
>
> I'm not sure if allowing the disjunctive types, like init
> is a good idea in R or Python. If you have two different
> arguments, then there are consistency requirements, so I'd
> rather see that as a substructure, just as in CmdStan.
>
> Then, to run, there needs to be a way to turn these
> into a config C++ object that can actually be passed to
> the Stan C++ to do the work, but that shouldn't be that
> hard, either.
>
> I'd like to compare how much work this would be to
> marshaling a config object from a similar input in a dictionary,
> converting to JSON, sending it to Stan C++ for validation,
> then interpreting the results back in terms of the actual
> argument names in PyStan or RStan.
>

This sounds good to me (serializing/deserializing a C++ object). I'd
still like to consider Protocol Buffers here instead of JSON here
because PB is/are fast (I think we could even put initial values and
data in there) and there's the explicit schema that everyone can read
(which JSON would lack). Moreover, you can specify some consistency
requirements in Protocol Buffers explicitly (see Union Types,
https://developers.google.com/protocol-buffers/docs/techniques#union).
Clearly Python and R will need to know something about what a valid
config looks like -- but I'd prefer that they need not have to know too
much.

As for passing large amounts of data via Protocol Buffers, there's this
magical-sounding "ZeroCopy" functionality provided by PB that sounds
pretty good
(https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.io.zero_copy_stream#ZeroCopyInputStream).

Best,

-a

[Krzysztof Sakrejda]

On Wednesday, March 18, 2015 at 6:41:25 AM UTC-4, Bob Carpenter wrote:
> I think the experience from R should be R-like, from
> Python should be Pythonic (can't believe I just said that),
> and from the command line, should be shellish.

The only way to have an R-like experience would be to have multiple naming conventions in the spirit of 'read.table' which blesses R with a combination of col.names/nrows/stringsAsFactors in the arguments. Maybe optimize.bfgs and optimize.lbfgs versus sampleHMC and sampleNUTS. In addition you would need at last two perpetually-coexisting ways of defining functions in Stan, preferably three. Also all error messages would be some variant of "Error in FUN(X[[1L]], ...) : subscript out of bounds". Nothing would ever be deprecated. Please save us from the R-like experience.

Krzysztof

[Michael Betancourt]

Let me be more particular about what I mean:

The C++ code needs a few things from each interface

in order to be executed — a configuration, from where

certain info will be obtained (data, initialization, etc), 

and from where certain info will be written (info, errors,

output, diagnostics, etc).  

The interfaces can implement these however they want,

so for example CmdStan can grab inputs from files

where as RStan and PyStan can grab them from the

local environment (making the experience appropriate

to their respective environments).  Same with the outputs.

The implementations vary but the basic interaction is

the same.

Where our disagreements lie is with the configuration.

Firstly, I want every interface to run the same optimization,

sampling, and diagnostic routines so that a user can

switch interfaces and have the same functionality —

I don’t see how this is guaranteed without enforcing

a monolithic function.  

Secondly, I want _every_  configuration option available

from each interface so it’s just as easy to, say, tweak the 

step size in RStan as it is in CmdStan (the “control” 

interface makes me sad every time I have to get in there).

Again, how the configuration is constructed can vary

by interface (string, nested dict, list, etc) but I feel

strongly that the configuration itself is uniform.

If we just give the interfaces a mess of routes and let

them do what they want I don’t see how we can guarantee

any uniformity of experience across the interfaces.

[Daniel Lee]

On Wed, Mar 18, 2015 at 9:57 AM, Michael Betancourt
<betan...@gmail.com> wrote:
> Let me be more particular about what I mean:
>
> The C++ code needs a few things from each interface
> in order to be executed — a configuration, from where
> certain info will be obtained (data, initialization, etc),
> and from where certain info will be written (info, errors,
> output, diagnostics, etc).

This is the part that isn't true. If the C++ code defines an interface
for how data is used by the C++, then the interfaces don't need to
pass configuration for data, initialization, etc, info, errors,
output, diagnostics, etc. They just need to pass the implementation to
access data, initialization, etc and the implementation to write info,
errors, output, diagnostics, etc.

[Daniel Lee]

On Wed, Mar 18, 2015 at 9:57 AM, Michael Betancourt
<betan...@gmail.com> wrote:

> Where our disagreements lie is with the configuration.
> Firstly, I want every interface to run the same optimization,
> sampling, and diagnostic routines so that a user can
> switch interfaces and have the same functionality —
> I don’t see how this is guaranteed without enforcing
> a monolithic function.

The first part we agree on. The second part, enforcing a monolithic
function, is where I disagree. The point was to have the interfaces
implement a few interfaces and that gets passed into a function.

I'm ok with the idea of having a config that's parsed, but I think we
need to separate out the configuration for the algorithms vs the
configuration that's needed to handle implementation of these things
that need to be passed in.

> Secondly, I want _every_ configuration option available
> from each interface so it’s just as easy to, say, tweak the
> step size in RStan as it is in CmdStan (the “control”
> interface makes me sad every time I have to get in there).
> Again, how the configuration is constructed can vary
> by interface (string, nested dict, list, etc) but I feel
> strongly that the configuration itself is uniform.

Sure, but it's even more uniform if there was a small function that
exposed doubles instead, right?

> If we just give the interfaces a mess of routes and let
> them do what they want I don’t see how we can guarantee
> any uniformity of experience across the interfaces.

I'm not suggesting that. I'm suggesting that there are lots of smaller
functions and there's a single large function that RStan, PyStan, and
CmdStan share. But, I can already imagine CmdStan wanting to build an
executable that only includes NUTS or only includes optimization.

[Michael Betancourt]

I'm ok with the idea of having a config that's parsed, but I think we
need to separate out the configuration for the algorithms vs the
configuration that's needed to handle implementation of these things
that need to be passed in.

Agreed, sorry if that wasn’t clear earlier.  I was defining

config as the config of the algorithms, not the callbacks

the interfaces have to define.

Sure, but it's even more uniform if there was a small function that
exposed doubles instead, right?

Example?

I'm not suggesting that. I'm suggesting that there are lots of smaller
functions and there's a single large function that RStan, PyStan, and
CmdStan share. But, I can already imagine CmdStan wanting to build an
executable that only includes NUTS or only includes optimization.

I agree that command should be split up into smaller functions

for easier testing (that part is already happening — which should

make it easier to test each function and also make it easier to 

create tests that require running the algorithms, such as the

statistical validation tests) but I still think that the single large

function should be the only official “route” exposed to the

interfaces.

In other words, the Stan C++ API would expose all the little

functions but if you want to be an official Stan interface then

you have to use the big interface function that wraps all of

them up into a singular user experience.

[Bob Carpenter]

Krzysztof:

I completely agree that hte names of arguments should be
consistent, even if "_" isn't as idiomatic in R as "." as
a separator.

I do not want to build a typical R function, just one that
uses data types that are familiar from R, such as lists,
etc., and that returns output in the form usable in R, such
as a reference object.

Allen and Michael:

I'm not sure how to do that in a transparent way without
having the glue:

R command
-> C++ config object
-> Stan C++ validator
-> Stan C++ optimization, sampling, or validation
-> C++ result object

Michael:

Do you object to all the additional functions inside R,
like extract() and what not? Or do you object to the
what Ben's calling "andrewfy", which was formerly referred
to by Andrew as "Sebastian's function"? That is, something
that provides make-like behavior inside the stan() call?
Or are you insisting on a separate make() then run call?
I'm OK with the latter, but RStan's going down the former
path.

What we really need is a spec from you as to what RStan
and PyStan interfaces would look like. I lack the imagination
to envsion what you're talking about other than what I put
in the previous message. Is the lack of a counter-proposal
implicit acknowledgment that what I wrote is what you'd like
to see in RStan?

What kind of drift are you talking about specifically?

More below.

> On Mar 19, 2015, at 1:18 AM, Michael Betancourt <betan...@gmail.com> wrote:
>
>> I'm ok with the idea of having a config that's parsed, but I think we
>> need to separate out the configuration for the algorithms vs the
>> configuration that's needed to handle implementation of these things
>> that need to be passed in.
>
> Agreed, sorry if that wasn’t clear earlier. I was defining
> config as the config of the algorithms, not the callbacks
> the interfaces have to define.
>
>> Sure, but it's even more uniform if there was a small function that
>> exposed doubles instead, right?
>
> Example?

I think the main issue here is whether the config is

* a string wrapping up the command-line interface,
* a structured protocol buffer
- with data
- without data
* a C++ object

I've been arguing for the C++ object approach, but Allen thinks
it's complicated to do that from PyStan (for reasons I don't
undersetand enough about Python to understand).

Whatever we do, CmdStan is going to make a call to make to
build the model (either wrapped up in another function, or
expicitly called as make as the CmdStan manual suggests now).
Whatever else we do, RStan and PyStan are going to need an
R and Python function to call respectively.

Is it really that hard to validate input from the interfaces?
There's almost no consistency issues from a lower-level interface
standpoint (that is, once you specify

method=sample algorithm=hmc engine=nuts

The one thing that would be nice for the C++ structure or
protocol buffers that Allen brought up is consistent
unions (aka variant data types). There's a nice
implementation in Boost that uses only static types, but it's a
bit of a pain to use because every function has to be defined
via callbacks. And there's the compile-time hit because of all
the templates.

>> I'm not suggesting that. I'm suggesting that there are lots of smaller
>> functions and there's a single large function that RStan, PyStan, and
>> CmdStan share. But, I can already imagine CmdStan wanting to build an
>> executable that only includes NUTS or only includes optimization.
>
> I agree that command should be split up into smaller functions
> for easier testing (that part is already happening — which should
> make it easier to test each function and also make it easier to
> create tests that require running the algorithms, such as the
> statistical validation tests) but I still think that the single large
> function should be the only official “route” exposed to the
> interfaces.

What I really dislike about this is the fact that it has
a polymorphic return. You get one set of output from optimize
and a different one from validate and a third one from
optimize.

I dislike even more the R idiom of just defining a massive
structure and partially populating it on return. Especially
the way it's used in something like glm to be mutable without
enforcing consistency.

> In other words, the Stan C++ API would expose all the little
> functions but if you want to be an official Stan interface then
> you have to use the big interface function that wraps all of
> them up into a singular user experience.

If we do this (and I don't like it), then I agree that
we'd want to have the interfaces pass in a giant string.

Michael --- you still haven't said what you want that function
to look like from the RStan or PyStan side. Until we have
an example of that, it's hard to talk about whether it's acceptable
to R/Python users or not.

- Bob

[Krzysztof Sakrejda]

On Wednesday, March 18, 2015 at 7:50:25 PM UTC-4, Bob Carpenter wrote:
> Krzysztof:
>
> I completely agree that hte names of arguments should be
> consistent, even if "_" isn't as idiomatic in R as "." as
> a separator.
>
> I do not want to build a typical R function, just one that
> uses data types that are familiar from R, such as lists,
> etc., and that returns output in the form usable in R, such
> as a reference object.

Sorry about polluting the discussion with that, I realize it really didn't fit here. Much of the stan interface is very intuitive and it all avoids the worst of R's idiosyncrasies. I do realize you guys are trying to be serious here.

Krzysztof

[Michael Betancourt]

> I'm not sure how to do that in a transparent way without
> having the glue:
>
> R command
> -> C++ config object
> -> Stan C++ validator
> -> Stan C++ optimization, sampling, or validation
> -> C++ result object

Wouldn’t the second and third steps be wrapped together?
As in you construct a config object, pass it into the Stan
call, and then get back your results.

> Do you object to all the additional functions inside R,
> like extract() and what not? Or do you object to the
> what Ben's calling "andrewfy", which was formerly referred
> to by Andrew as "Sebastian's function"? That is, something
> that provides make-like behavior inside the stan() call?
> Or are you insisting on a separate make() then run call?
> I'm OK with the latter, but RStan's going down the former
> path.
>
> What we really need is a spec from you as to what RStan
> and PyStan interfaces would look like. I lack the imagination
> to envsion what you're talking about other than what I put
> in the previous message. Is the lack of a counter-proposal
> implicit acknowledgment that what I wrote is what you'd like
> to see in RStan?

If we wanted to be _really_ consistent amongst the interfaces
then the UX would be something like

1) Preprocessing
2) Compile Stan object/executable
3) Configure algorithm arguments and callbacks
4) Call Stan object/executable with configuration, returning results in fit object
5) Postprocessing

(1) and (5) are left to the interfaces so we don’t have to worry about those.
Each interface can add whatever functionality it wants here, such as print
and fit, although it would be great to unity these as much as possible.

(2) and (4) is a bit tricky because right now RStan and PyStan try to wrap
these together which necessitates tricks like Sebastian’s function. Personally
I’d much rather have the user create an explicit object and then call a run method
of that object, so if you want to rerun without recompiling you just call the
method again instead of creating a new object (this also parallels the CmdStan
procedure).

(4) is also completed in RStan and PyStan because the results aren’t
uniformly accessible. Instead you have to access the samples
and the sampler parameters separately in incredibly annoying ways,
but this is just a disagreement about exactly what extract should return.

My big complaint is in (3) which is where some of the biggest disparities
are (i.e. that awkward control list). The config can be implemented by
each interface separately but the user should have access to the
arguments in roughly the same way — for example, if we maintain the
current hierarchical structure then there should be some way of
specifying the config hierarchically. The flattened single arguments of
RStan and PyStan upsets me.

> What kind of drift are you talking about specifically?

In terms of time-dependence I’m specifically referring to all of these
functions that try to wrap RStan functionality into a UX that’s moving
further and further away from how the other interfaces are implemented.

[Ben Goodrich]

On Thursday, March 19, 2015 at 5:26:09 AM UTC-4, Michael Betancourt wrote:

(2) and (4) is a bit tricky because right now RStan and PyStan try to wrap
these together which necessitates tricks like Sebastian’s function.  Personally
I’d much rather have the user create an explicit object and then call a run method
of that object, so if you want to rerun without recompiling you just call the
method again instead of creating a new object (this also parallels the CmdStan
procedure).

The stan() function does (2), (3), and (4) together but it is possible to do them separately. Or technically, you can do (2) separately but (3) and (4) are lumped together. In any event, if necessary, we could change that. We could even use more Pythonic calling semantics

model <- stan_model("foo.stan")
model$config(adapt_delta = 0.9)
model$sampling()
model$summary()

The only difficulty is that such idioms are relatively new to the R language and people (especially Andrew) are not used to them. It isn't a technical problem. I do mostly agree with Bob that the interfaces should behave more like the external software, although in this case it would be more a case of RStan behaving more like R will a few years in the future.
 

My big complaint is in (3) which is where some of the biggest disparities
are (i.e. that awkward control list).  The config can be implemented by
each interface separately but the user should have access to the
arguments in roughly the same way — for example, if we maintain the
current hierarchical structure then there should be some way of
specifying the config hierarchically.  The flattened single arguments of
RStan and PyStan upsets me.

That is not inherent either. Lots of R functions take a flat control list but we could specify that in a hierarchical fashion with either a list of lists or a JSON thing. Almost no one would even notice if we changed it completely.

RStan could adapt to almost anything and I think PyStan could as well. We just need to come to an agreement on monolithic config vs. modular and JSON vs. Protocol Buffers and things like that.

Ben

[Michael Betancourt]

Yeah — I don’t think there are any technical limitations 

to any of this.   We just need to agree to and define

a spec so that we’re all on the same page.

[Allen B. Riddell]

I think this is mostly right. If we agree on a spec one could start
trying out some prototypes. It might be worth hashing this out over some
lower latency medium.