Stan 2.0 command-line structure

[Bob Carpenter]

I hadn't quite realized the extent to which Michael has been
innovating on the command-line front in the feature/arg_refactor
branch. I still haven't quite put my head around all the dependencies
entailed by the help or the doc.

I think it would be helpful to have more examples with more
config of command-line calls in the manual.

I like the way the help works. Calling

./my-model help

prints out a list of subcommands (and any global options)
and then

./my-model sample help

does what I expect in printing out the options appropriate
for sample. It would help to have more detail here --- I was
hoping for enough to let me configure the command, not just
a list of keys without values. This forces the user to go back
to help-all and then try to navigate that maze.

Ideally, we'd have the same behavior from

./my-model nuts help

Note that I wrote this without "sample". And that "help" comes
last. The idea is that you invoke the help option of the
subcommand nuts. But I don't see how to make this fly.

In general, is there a way to make subcommands imply their super-commands?
I'd rather not have to write

./my-model sample nuts ...

But this doesn't work either, as apparently nuts needs some kind
of key. So I dug through the doc and played with a sample model
and realized I needed to do this:

./my-model sample algorithm=hmc engine=nuts

It's not so bad. But why no "--" before keys? Is this kosher with
the command-lines of the world? I've just followed what I've seen in
other software like a good monkey (of the "monkey see, monkey do" species).

I don't have a better suggestion than "algorithm" (hmc) vs. "engine" (nuts).
It's a bit arbitrary because NUTS is a kind of HMC algorithm, which
in turn is a kind of Metropolis algorithm, which in turn is a kind of MCMC
algorithm. So it's algorithms (or engines if we want to go steam punk)
all the way down.

This brings up the issue that the commands and options are not in
a strict hierarchy. It seems like the random options, for instance,
get used across subcommands. Then there's mass matrix adaptation for HMC,
which is very much like covariance adaptation for Metropolis, but in other
ways it's very different.

I'm still cogitating on this, but it may be
that there's no better way in general to do it than Michael
came up with. You can think about it as being all key=value, with
lots of consistency checking implied by the "hierarchy".

- Bob

[Michael Betancourt]

and then

  ./my-model sample help

does what I expect in printing out the options appropriate
for sample.  It would help to have more detail here --- I was
hoping for enough to let me configure the command, not just
a list of keys without values.  This forces the user to go back
to help-all and then try to navigate that maze.

We can change that to a more verbose output of the possible subcommands/values easily. The idea was to keep the help output sparse for easy viewing.

 

Ideally, we'd have the same behavior from

  ./my-model nuts help

Note that I wrote this without "sample".  And that "help" comes
last.  The idea is that you invoke the help option of the
subcommand nuts.  But I don't see how to make this fly.

Can't be done in the current framework since we allow subcommands to be resused (for example, hmc, nesterov, and bfgs all use arg_stepsize to configure their stepsize) so the mapping from subcommand to command isn't one-to-one.  We could have help completely ignore the hierarchy and give the arguments print methods of their own, but then we wouldn't be able to print info about related arguments since that requires the hierarchy.

 

In general, is there a way to make subcommands imply their super-commands?
I'd rather not have to write

  ./my-model sample nuts ...

But this doesn't work either, as apparently nuts needs some kind
of key.  So I dug through the doc and played with a sample model
and realized I needed to do this:

./my-model sample algorithm=hmc engine=nuts

Right, there's no argument called "nuts" -- it's just a value.  We could add "shortcuts" that would expand "nuts" into the proper command, if necessary.

 

It's not so bad.  But why no "--" before keys?  Is this kosher with
the command-lines of the world?  I've just followed what I've seen in
other software like a good monkey (of the "monkey see, monkey do" species).

It's cleaner.  I think Dan made an executive decision at some point.

 

I don't have a better suggestion than "algorithm" (hmc) vs. "engine" (nuts).
It's a bit arbitrary because NUTS is a kind of HMC algorithm, which
in turn is a kind of Metropolis algorithm, which in turn is a kind of MCMC
algorithm.  So it's algorithms (or engines if we want to go steam punk)
all the way down.

Yeah, the hierarchy is in no way unique but this classification provided the least reuse of arguments.  Also, we really should rename "metropolis" to "rw_metropolis" (Random-Walk Metropolis) before we implement it.

 

This brings up the issue that the commands and options are not in
a strict hierarchy.  It seems like the random options, for instance,
get used across subcommands.   Then there's mass matrix adaptation for HMC,
which is very much like covariance adaptation for Metropolis, but in other
ways it's very different.

The design allows argument implementations to be resued, but then they become unique in the hierarchy; this avoids code duplication while also respecting the hierarchy.  For example, we can have different adaptation options for RWM and HMC but not have to rewrite the arguments that they share.

 

I'm still cogitating on this, but it may be
that there's no better way in general to do it than Michael
came up with.  You can think about it as being all key=value, with
lots of consistency checking implied by the "hierarchy".

It's slightly more general.  There are three kinds of arguments:

1) valued arguments

2) unvalued arguments

Valued arguments, which break down into lists arguments and singleton arguments, follow the key=value format.  

Unvalued arguments have no value.  These break down into categorical arguments (for example, adapt which just leads into all of the adaptation options) and flags that specify a choice (these are used like enums, for example in the possible metrics for HMC).

 

[Ben Goodrich]

On Wednesday, August 7, 2013 11:04:17 AM UTC-4, Bob Carpenter wrote:

It's not so bad.  But why no "--" before keys?  Is this kosher with
the command-lines of the world?  I've just followed what I've seen in
other software like a good monkey (of the "monkey see, monkey do" species).

The de facto way of doing things is with "short" and "long" dashes. The one dash version means that you just type a space and then the value after like

tail -n 5 TO-DO.txt

and the two dash version uses an equal sign to do the equivalent thing

tail --lines=5 TO-DO.txt

Usually if something is not proceeded by any dashes, it is considered data, as in TO-DO.txt.

Since command-line Stan is disproportionally used by people who do things via the command-line, I think we should stick with the conventions.

Ben

[Bob Carpenter]

On 8/7/13 12:17 PM, Ben Goodrich wrote:
> On Wednesday, August 7, 2013 11:04:17 AM UTC-4, Bob Carpenter wrote:
>
> It's not so bad. But why no "--" before keys? Is this kosher with
> the command-lines of the world? I've just followed what I've seen in
> other software like a good monkey (of the "monkey see, monkey do" species).
>
>
> The de facto way of doing things is with "short" and "long" dashes. The one dash version means that you just type a
> space and then the value after like
>
> |
> tail -n 5 TO-DO.txt
> |
>
> and the two dash version uses an equal sign to do the equivalent thing
>
> |
> tail --lines=5 TO-DO.txt
> |
>
> Usually if something is not proceeded by any dashes, it is considered data, as in TO-DO.txt.

Thanks. Mitzi just explained this pattern to me when I asked
her about command-line conventions! I had never noticed
before that single dashes had no equal signs and the double
dashes did.

Mitzi also said that the single dash often
had an abbreviated name, so it might be

--lines=5

vs.

-l 5

> Since command-line Stan is disproportionally used by people who do things via the command-line,

Linux integration == more command-line users

Python, MATLAB integration == fewer command-line users

Machine learning "marketing" == more command-line users

:-)

> I think we should stick
> with the conventions.

I agree. This doesn't seem like the place to innovate.

I prefer the verbose names with -- and =, which not
coincidentally matches the way it was done in 1.0.

We don't have the equivalent of data in the Stan command, or I
should say we have several kinds of data, plain old data, initialization
data, and covariance or mass matrix data.

I'm going to work on adding more examples to the command doc organized
in a more task-oriented way. I'd rather expand upon and explain
what Michael has now for doc and help-all than trash it (I find it
very elegant now that I understand what's going on).

- Bob

[Michael Betancourt]

My only concern with the dashes is what to do for the categorical arguments. For example, this is fine

./model sample --algorithm=hmc

but what about the "adapt" argument:

./model sample --algorithm=hmc adapt --engage=true

or

./model sample --algorithm=hmc --adapt --engage=true

adapt isn't a key/value pair so doesn't fit into the usual use-case, but giving it a different prefix breaks the symmetry amongst arguments.

Technically adding the prefix is easy -- honestly it's probably easiest to just strip away the prefix and use the current code. Requiring a prefix is slightly more involved but still straightforward.

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Bob Carpenter]

Thanks. I can translate this for the manual.

Are the

On 8/7/13 11:22 AM, Michael Betancourt wrote:
> and then
>
> ./my-model sample help
>
> does what I expect in printing out the options appropriate
> for sample. It would help to have more detail here --- I was
> hoping for enough to let me configure the command, not just
> a list of keys without values. This forces the user to go back
> to help-all and then try to navigate that maze.
>
>
> We can change that to a more verbose output of the possible subcommands/values easily. The idea was to keep the help
> output sparse for easy viewing.

I think we want it to be as sparse as possible while remaining
complete. And barring that, to at least indicate where more information
can be found. The top-level does this, but I find the subcommand
a bit too sparse.

> Ideally, we'd have the same behavior from
>
> ./my-model nuts help
>
> Note that I wrote this without "sample". And that "help" comes
> last. The idea is that you invoke the help option of the
> subcommand nuts. But I don't see how to make this fly.
>
>
> Can't be done in the current framework since we allow subcommands to be resused (for example, hmc, nesterov, and bfgs
> all use arg_stepsize to configure their stepsize) so the mapping from subcommand to command isn't one-to-one. We could
> have help completely ignore the hierarchy and give the arguments print methods of their own, but then we wouldn't be
> able to print info about related arguments since that requires the hierarchy.

OK.

Eventually, it might be nice to have the inverse relations.
That is,

stepsize -> hmc | nesterov | bfgs

so that if someone enters stepsize and not an appropriate subcommand,
we can be more informative.

I thought you decided to use time_int instead of stepsize for HMC?

> In general, is there a way to make subcommands imply their super-commands?
> I'd rather not have to write
>
> ./my-model sample nuts ...
>
> But this doesn't work either, as apparently nuts needs some kind
> of key. So I dug through the doc and played with a sample model
> and realized I needed to do this:
>
> ./my-model sample algorithm=hmc engine=nuts
>
>
> Right, there's no argument called "nuts" -- it's just a value. We could add "shortcuts" that would expand "nuts" into
> the proper command, if necessary.

Now that I see what's going on I understand.

> It's not so bad. But why no "--" before keys? Is this kosher with
> the command-lines of the world? I've just followed what I've seen in
> other software like a good monkey (of the "monkey see, monkey do" species).
>
>
> It's cleaner. I think Dan made an executive decision at some point.

See my reply and Ben's --- I think we should reinsert the "--".

> I don't have a better suggestion than "algorithm" (hmc) vs. "engine" (nuts).
> It's a bit arbitrary because NUTS is a kind of HMC algorithm, which
> in turn is a kind of Metropolis algorithm, which in turn is a kind of MCMC
> algorithm. So it's algorithms (or engines if we want to go steam punk)
> all the way down.
>
>
> Yeah, the hierarchy is in no way unique but this classification provided the least reuse of arguments. Also, we really
> should rename "metropolis" to "rw_metropolis" (Random-Walk Metropolis) before we implement it.

OK by me, but I think "Metropolis" without an qualifiers usually
implies "random walk Metropolis". The same way HMC without qualifiers
usually implies what you distinguished as EMHMC.

> This brings up the issue that the commands and options are not in
> a strict hierarchy. It seems like the random options, for instance,
> get used across subcommands. Then there's mass matrix adaptation for HMC,
> which is very much like covariance adaptation for Metropolis, but in other
> ways it's very different.
>
>
> The design allows argument implementations to be resued, but then they become unique in the hierarchy; this avoids code
> duplication while also respecting the hierarchy. For example, we can have different adaptation options for RWM and HMC
> but not have to rewrite the arguments that they share.

Right --- it's very cool now that I see what's going on.

> I'm still cogitating on this, but it may be
> that there's no better way in general to do it than Michael
> came up with. You can think about it as being all key=value, with
> lots of consistency checking implied by the "hierarchy".
>
>
> It's slightly more general. There are three kinds of arguments:
>
> 1) valued arguments
> 2) unvalued arguments
>
> Valued arguments, which break down into lists arguments and singleton arguments, follow the key=value format.
>
> Unvalued arguments have no value. These break down into categorical arguments (for example, adapt which just leads into
> all of the adaptation options) and flags that specify a choice (these are used like enums, for example in the possible
> metrics for HMC).

Do we really need to say "adapt" or "random"? It introduces
order constraints and seems very clunky. I'd rather write
adapt_foo for all of its features if we had to and perhaps reuse
the code at a slightly higher level than the names of the args.

- Bob

[Bob Carpenter]

On 8/7/13 7:00 PM, Michael Betancourt wrote:
> My only concern with the dashes is what to do for the categorical arguments. For example, this is fine
>
> ./model sample --algorithm=hmc
>
> but what about the "adapt" argument:
>
> ./model sample --algorithm=hmc adapt --engage=true
>
> or
>
> ./model sample --algorithm=hmc --adapt --engage=true
>
> adapt isn't a key/value pair so doesn't fit into the usual use-case, but giving it a different prefix breaks the symmetry amongst arguments.
>
> Technically adding the prefix is easy -- honestly it's probably easiest to just strip away the prefix and use the current code. Requiring a prefix is slightly more involved but still straightforward.

I'm not keen on multiple ways of doing things. And we'd only
want to strip out "--" before a key.

Is there a way to just get rid of adapt altogether? Like
using --adapt-engage=true? I don't like the fact that it's
order dependent.

Sorry to be nitpicking at this late date --- I should've
looked at this more closely during the design. I do think
this kind of user-facing design should be made as stable as
possible, and I don't want to do it one way for 2.0 and then
change it again right away.

- Bob

[Bob Carpenter]

On 8/7/13 7:03 PM, Bob Carpenter wrote:
> Thanks. I can translate this for the manual.
>
> Are the

Please ignore my dropping off midsentence --- I changed to inlining
the comments and never went back to clean up.

- Bob

[Michael Betancourt]

> Eventually, it might be nice to have the inverse relations.
> That is,
>
> stepsize -> hmc | nesterov | bfgs
>
> so that if someone enters stepsize and not an appropriate subcommand,
> we can be more informative.

What we could do is search through the hierarchy for all occurrences and show the possible paths.

> I thought you decided to use time_int instead of stepsize for HMC?

int_time and stepsize (it's the number of leapfrog steps that's not particularly meaningful)

> OK by me, but I think "Metropolis" without an qualifiers usually
> implies "random walk Metropolis". The same way HMC without qualifiers
> usually implies what you distinguished as EMHMC.

Right but then you get pedantic people arguing that any implementation of HMC is Metropolis in the end. As we're drafting review papers on the theory behind HMC I'm starting to get really careful about notation as to leave no ambiguity at all. Might as well try to do it right now.

> Do we really need to say "adapt" or "random"? It introduces
> order constraints and seems very clunky. I'd rather write
> adapt_foo for all of its features if we had to and perhaps reuse
> the code at a slightly higher level than the names of the args.

Personally I like the ability to have categorical arguments like this. For one, it avoids having to rewrite the category name over and over again (imagine the user frustration of having to type adapt_mu=5 adapt_eta=8 adapt_t0=2…). We could add braces adapt={ mu=5 eta=8 t0=2 } but then we'd have to deal with closure checking. Or perhaps just add a trailing colon? adapt: mu=5 eta=8 t0=2?

[Michael Betancourt]

> I'm not keen on multiple ways of doing things. And we'd only
> want to strip out "--" before a key.
>
> Is there a way to just get rid of adapt altogether? Like
> using --adapt-engage=true? I don't like the fact that it's
> order dependent.

See the other email I just sent out.

> Sorry to be nitpicking at this late date --- I should've
> looked at this more closely during the design. I do think
> this kind of user-facing design should be made as stable as
> possible, and I don't want to do it one way for 2.0 and then
> change it again right away.

I'm cool with it -- I definitely want to get it right.

Have we considered trying to get some power users to try the new format?

[Michael Betancourt]

On second thought, I think adding prefixes is a really bad idea. The problem is that categorical arguments can also be values, so if you want to have

adapt_mu=5 adapt_eta=8 adapt_t0=2…

you'd also have to have

--algorithm=hmc_metric=unit_e --algorithm=hmc_stepsize=0.1…

which is not fun to type in. It'd also break the recursive encapsulation of the argument objects/methods.

Braces or a suffix'd colon would still be possible,

adapt={mu=5 eta=8 t0=2...}
--algorithm=hmc={metric=unit_e stepsize=0.1…}

Although that would require some nontrivial parsing work.

[Bob Carpenter]

We can certainly beta test before a bigger release.

The problem with power users for testing is that they'll
adapt to anything. Something like this also takes time
to adjust to before really evaluating long term. If you'd
have asked me to evaluate git, I'd say it took a month or
so before I was convinced in practice that it was a good
idea.

And never forget the old Hank Ford quote, "If I'd have asked
my customers what they wanted, they'd have said faster horses".

- Bob

[Bob Carpenter]

On 8/7/13 7:24 PM, Michael Betancourt wrote:
>> Eventually, it might be nice to have the inverse relations.
>> That is,
>>
>> stepsize -> hmc | nesterov | bfgs
>>
>> so that if someone enters stepsize and not an appropriate subcommand,
>> we can be more informative.
>
> What we could do is search through the hierarchy for all occurrences and show the possible paths.
>
>> I thought you decided to use time_int instead of stepsize for HMC?
>
> int_time and stepsize (it's the number of leapfrog steps that's not particularly meaningful)
>
>> OK by me, but I think "Metropolis" without an qualifiers usually
>> implies "random walk Metropolis". The same way HMC without qualifiers
>> usually implies what you distinguished as EMHMC.
>
> Right but then you get pedantic people arguing that any implementation of HMC is Metropolis in the end. As we're drafting review papers on the theory behind HMC I'm starting to get really careful about notation as to leave no ambiguity at all. Might as well try to do it right now.

Just about anything is Metropolis in the end, so
maybe we just drop the "Metropolis" suffix because
it's redundant? :-)

Seriously, though, language is like this and you get a choice along
the line between verbose and explicit and terse and implicit.

I'm OK with rw_metropolis, but don't think it's necessary in this
context. Just saying rw_metropolis isn't enough for people to know
exactly what we're doing with adaptation.

>> Do we really need to say "adapt" or "random"? It introduces
>> order constraints and seems very clunky. I'd rather write
>> adapt_foo for all of its features if we had to and perhaps reuse
>> the code at a slightly higher level than the names of the args.
>
> Personally I like the ability to have categorical arguments like this. For one, it avoids having to rewrite the category name over and over

> again (imagine the user frustration of having to type adapt_mu=5 adapt_eta=8 adapt_t0=2�).

> We could add braces adapt={ mu=5 eta=8 t0=2 } but then we'd have to deal with
> closure checking. Or perhaps just add a trailing colon? adapt: mu=5 eta=8 t0=2?

I'm not saying it's a bad idea. Just that it's unusual,
and hence provides a bit of a learning curve. So
and I'm not sure this is where we want to innovate.

I prefer the trailing colon to nothing as it distinguishes the
categorical features from the flags and indicates their scope.
I don't like the brackets.

What we probably really need is some kind of config file.
I took a quick poll. I hate config files, Mitzi loves them :-)

I personally find this:

--adapt_mu=5 --adapt_eta=8 --adapt_t0=2

easier to understand than this:

adapt: mu=5 eta=8 t0=2

What do others think here? I could be convinced to innovate
on the command line. Someone has to do it some time.

I'm strongly opposed to having multiple ways of doing things.
The problem is that it bloats the doc size and also splits
all the help threads and examples.

- Bob

[Michael Betancourt]

Haha. I was thinking more testing for nasty bugs rather than focus group.

[Michael Betancourt]

> I'm OK with rw_metropolis, but don't think it's necessary in this
> context. Just saying rw_metropolis isn't enough for people to know
> exactly what we're doing with adaptation.

Ah, but rw_metropolis isn't supposed to be a complete specification. We'd have a "proposal_covar" option that would mirror HMC's metric options.

> What we probably really need is some kind of config file.
> I took a quick poll. I hate config files, Mitzi loves them :-)

The hierarchical output is easily converted back into an argument string -- I already do it for the tests. So we basically have config files already.

> I personally find this:
>
> --adapt_mu=5 --adapt_eta=8 --adapt_t0=2
>
> easier to understand than this:
>
> adapt: mu=5 eta=8 t0=2
>
> What do others think here? I could be convinced to innovate
> on the command line. Someone has to do it some time.

The only problem with

--adapt_mu=5 --adapt_eta=8 --adapt_t0=2

is that is also forces really awkward strings like

--algorithm=hmc_metric=unit_e

Raw categoricals like

adapt mu=5 eta=8 t0=2

are the easiest option.

[Bob Carpenter]

On 8/8/13 4:34 PM, Michael Betancourt wrote:
...

> The only problem with
>
> --adapt_mu=5 --adapt_eta=8 --adapt_t0=2
>
> is that is also forces really awkward strings like
>
> --algorithm=hmc_metric=unit_e

That would be have to be compressed. Why not just have
hmc_metric=unit_e? Why have --algorithm? This is what I'm
missing conceptually in the new scheme.

> Raw categoricals like
>
> adapt mu=5 eta=8 t0=2
>
> are the easiest option.

That's the current status, right?

- Bob

[Daniel Lee]

Looks like I missed a whole bunch of conversations while I had my head deep in code. I'm going to try to pick things out of past conversations and hopefully the message comes out coherent enough.

On Aug 7, 2013, at 7:38 PM, Michael Betancourt wrote:
> Braces or a suffix'd colon would still be possible,
>
> adapt={mu=5 eta=8 t0=2...}
> --algorithm=hmc={metric=unit_e stepsize=0.1…}
>
> Although that would require some nontrivial parsing work.

A couple points here:
- Braces are a bad idea, I think. I think the shell will do stuff with braces before it enters our program. I think users might be able to get it to us if they quoted or escaped it somehow, but I think it's really not the best option for us.
- Regarding colon: it will be nontrivial parsing work. I would rather not introduce more complexity in our wrapper than necessary. I would prefer to not maintain complex logic / code here.

On Aug 7, 2013, at 11:22 AM, Michael Betancourt wrote:
> It's not so bad. But why no "--" before keys? Is this kosher with
> the command-lines of the world? I've just followed what I've seen in
> other software like a good monkey (of the "monkey see, monkey do" species).
>
> It's cleaner. I think Dan made an executive decision at some point.

On Aug 7, 2013, at 6:45 PM, Bob Carpenter wrote:
>
> > I think we should stick
>> with the conventions.
>
> I agree. This doesn't seem like the place to innovate.
>
> I prefer the verbose names with -- and =, which not
> coincidentally matches the way it was done in 1.0.

Yes, I think I made that call a while ago, but I was thinking along the lines of git subcommands. I didn't think it through. We have a lot more key, value pairs that we deal with.

And yes, I don't want to innovate here. Unfortunately, I don't think there's a good standard. I know it's a little late, but if we're trying to do something more standard, what about using the Boost Program_options library? http://www.boost.org/doc/libs/1_54_0/doc/html/program_options.html

Unfortunately, it's NOT header only. But if it makes sense to use it, I'll figure it out unless someone else volunteers.

On Aug 8, 2013, at 4:15 PM, Bob Carpenter wrote:
> On 8/7/13 7:24 PM, Michael Betancourt wrote:
>>> Eventually, it might be nice to have the inverse relations.
>>> That is,
>>>
>>> stepsize -> hmc | nesterov | bfgs
>>>
>>> so that if someone enters stepsize and not an appropriate subcommand,
>>> we can be more informative.
>>
>> What we could do is search through the hierarchy for all occurrences and show the possible paths.
>>

> ...

>>> Do we really need to say "adapt" or "random"? It introduces
>>> order constraints and seems very clunky. I'd rather write
>>> adapt_foo for all of its features if we had to and perhaps reuse
>>> the code at a slightly higher level than the names of the args.
>>
>> Personally I like the ability to have categorical arguments like this. For one, it avoids having to rewrite the category name over and over

> > again (imagine the user frustration of having to type adapt_mu=5 adapt_eta=8 adapt_t0=2…).

> > We could add braces adapt={ mu=5 eta=8 t0=2 } but then we'd have to deal with
> > closure checking. Or perhaps just add a trailing colon? adapt: mu=5 eta=8 t0=2?
>
> I'm not saying it's a bad idea. Just that it's unusual,
> and hence provides a bit of a learning curve. So
> and I'm not sure this is where we want to innovate.

> ...

> What do others think here? I could be convinced to innovate
> on the command line. Someone has to do it some time.

I know, there was a lot of discussion here. What I would like is consistency. And hopefully we have enough foresight to provide consistency in the command line interface for at least a few releases. I think we're on the right path.

One thing I did want to mention is that the combinatorics haven't overwhelmed us yet. I still think with the number of options we have for v2.0, we can still have a "flat" subcommand structure even though it acts and feels hierarchical. Or vice versa. For printing help, we can even do what git does and have each subcommand have a char* array which we just pipe to output.

I'll get to the point: we should focus on designing the interaction on the command line.

With that said, I can roll with anything that's consistent, whether that's standard or innovative. Mike's done enough for me to be ok with the options, but if anyone has strong opinions, I'll gladly defer.

On Aug 8, 2013, at 4:15 PM, Bob Carpenter wrote:
> What we probably really need is some kind of config file.
> I took a quick poll. I hate config files, Mitzi loves them :-)

I think it makes sense to have config files eventually.

On Aug 8, 2013, at 7:57 PM, Bob Carpenter wrote:

> On 8/8/13 4:34 PM, Michael Betancourt wrote:
> ...
>> The only problem with
>>
>> --adapt_mu=5 --adapt_eta=8 --adapt_t0=2
>>
>> is that is also forces really awkward strings like
>>
>> --algorithm=hmc_metric=unit_e
>
> That would be have to be compressed. Why not just have
> hmc_metric=unit_e? Why have --algorithm? This is what I'm
> missing conceptually in the new scheme.

I'm with Bob here.

>> Raw categoricals like
>>
>> adapt mu=5 eta=8 t0=2
>>
>> are the easiest option.
>
> That's the current status, right?

I think so. I think "adapt" is the subcommand and "mu=5 eta=8 t0=2" are arguments. If I'm to understand correctly, the standard way to do this is to type:
"adapt --mu=5 --eta=8 --t0=2"
or is it:
"adapt --mu 5 --eta 8 --t0 2"
?

And should something like this be standard too?
"adapt -m=5 -e=8 -t=2"
That would be the single dash abbreviated form. Spaces wouldn't work here.





Daniel

[Michael Betancourt]

> And should something like this be standard too?
> "adapt -m=5 -e=8 -t=2"
> That would be the single dash abbreviated form. Spaces wouldn't work here.

The expected standard would be
"adapt -m 5 -e 8 -t 2"
but, as you note, trying to parse the spaces would be a pain in the ass so I'd rather avoid this option and stick to just the "--key=value" format.

>> That would be have to be compressed. Why not just have
>> hmc_metric=unit_e? Why have --algorithm? This is what I'm
>> missing conceptually in the new scheme.

Okay, let's step back a bit.

A hierarchical structure allows context-specific variables (just like a hierarchical statistical model), which allows us to have many, many possible configurations without forcing the user to worry about specifying irrelevant arguments. We can, and have in the current implementation, even use the hierarchical structure to prevent the specification of irrelevant arguments -- the code just errs out.

To implement a hierarchy we need to elements: specification of exchangeable levels and specification of child nodes when the options are mutually exclusive. The latter is accomplished with list arguments that admit one and only one value, while the former is accomplished with categorical arguments. In particular, to have the full power of a hierarchy we need to allow the value of list arguments to themselves be categorical arguments, so that we choose amongst different possible levels.

As an example, consider choosing a sampling algorithm.

1) We specify that we want to sample instead of optimize or VB or test.
--method=sample

2) Given that we're sampling, we now have a bunch of new options from which to choose. We specify that we want to use HMC instead of other sampling algorithms
--algorithm=hmc

3) Again, HMC defines new options that are unique to this exact path through the hierarchy. We specify that we want to use a unit metric
--metric=unit_e

4) And let's use NUTS
--engine=nuts

5) Oh, snap. NUTS has context-specificy options, too. Let's open up max_treedepth a bit,
--max_treedepth=12

The full argument string is then

--method=sample --algorithm=hmc --metric=unit_e --engine=nuts --max_treedepth=12

Note that this is a very flexible method. If we wanted to specify the step size we could do it before or after going down the "engine" subtree.

--method=sample --algorithm=hmc --metric=unit_e --stepsize=0.1 --engine=nuts --max_treedepth=12

or

--method=sample --algorithm=hmc --metric=unit_e --engine=nuts --max_treedepth=12 --stepsize=0.1

Now why can't use a formatting like Bob suggested? The problem is that it's basically just flattening the hierarchy (which is also something Dan suggested). Personally, I hate this idea because it makes argument specification a huge pain for the user. For example, the above specification becomes

sample_algorithm=hmc_metric=unit_e sample_algorithm=hmc_stepsize=0.1 sample_algorithm=engine=nuts --max_treedepth=12

There's far too much typing required of the user, which is not only frustrating but also asking for typos. And we've only specified a few arguments here!

While it would be great to have a better UI (imagine context-dependent pop-up lists! What we could do with a web-interface and some javascript…) we're limited to the CLI and we're going to have to make some compromise. Personally, I believe that users will pick up the current implementation quickly enough and full path specification of each argument is typo- and fatigue-inducing overkill.

But I'm also biased. :-)

[Michael Betancourt]

I followed up with Bob's idea about more helpful comments and implemented configuration suggestions.  For example,

betan: ./example sample algorithm=hmc engine=nuts int_time=5

int_time=5 is either mistyped or misplaced.

Perhaps you meant one of the following valid configurations?

  method=sample algorithm=hmc engine=static int_time=<double>

Failed to parse arguments, terminating Stan

[Bob Carpenter]

On 8/9/13 4:35 AM, Michael Betancourt wrote:
>> And should something like this be standard too?
>> "adapt -m=5 -e=8 -t=2"
>> That would be the single dash abbreviated form. Spaces wouldn't work here.
>
> The expected standard would be
> "adapt -m 5 -e 8 -t 2"
> but, as you note, trying to parse the spaces would be a pain in the ass so I'd rather avoid this option and stick to just the "--key=value" format.
>
>>> That would be have to be compressed. Why not just have
>>> hmc_metric=unit_e? Why have --algorithm? This is what I'm
>>> missing conceptually in the new scheme.
>
> Okay, let's step back a bit.
>
> A hierarchical structure allows context-specific variables (just like a hierarchical statistical model), which allows us to have many, many possible configurations without forcing the user to worry about specifying irrelevant arguments. We can, and have in the current implementation, even use the hierarchical structure to prevent the specification of irrelevant arguments -- the code just errs out.

I understand --- but this is what I think is problematic if I'm
understanding properly. Isn't there a "random" element that
shows up and then defines what the following key-value pairs mean?

> To implement a hierarchy we need to elements: specification of exchangeable levels and specification of child nodes when the options are mutually exclusive. The latter is accomplished with list arguments that admit one and only one value, while the former is accomplished with categorical arguments. In particular, to have the full power of a hierarchy we need to allow the value of list arguments to themselves be categorical arguments, so that we choose amongst different possible levels.

But it doesn't have to look like that to the user, as your example
below shows.

> As an example, consider choosing a sampling algorithm.
>
> 1) We specify that we want to sample instead of optimize or VB or test.
> --method=sample
>
> 2) Given that we're sampling, we now have a bunch of new options from which to choose. We specify that we want to use HMC instead of other sampling algorithms
> --algorithm=hmc
>
> 3) Again, HMC defines new options that are unique to this exact path through the hierarchy. We specify that we want to use a unit metric
> --metric=unit_e
>
> 4) And let's use NUTS
> --engine=nuts
>
> 5) Oh, snap. NUTS has context-specificy options, too. Let's open up max_treedepth a bit,
> --max_treedepth=12
>
> The full argument string is then
>
> --method=sample --algorithm=hmc --metric=unit_e --engine=nuts --max_treedepth=12

This is totally OK by me. The hierarchy is an implementation detail.

Is this order independent, so that I can put the arguments in any order?

I also don't see any bare arguments like "random" here that change
the meaning of the following parameters.

> Note that this is a very flexible method. If we wanted to specify the step size we could do it before or after going down the "engine" subtree.
>
> --method=sample --algorithm=hmc --metric=unit_e --stepsize=0.1 --engine=nuts --max_treedepth=12
>
> or
>
> --method=sample --algorithm=hmc --metric=unit_e --engine=nuts --max_treedepth=12 --stepsize=0.1
>
> Now why can't use a formatting like Bob suggested? The problem is that it's basically just flattening the hierarchy (which is also something Dan suggested). Personally, I hate this idea because it makes argument specification a huge pain for the user. For example, the above specification becomes
>
> sample_algorithm=hmc_metric=unit_e sample_algorithm=hmc_stepsize=0.1 sample_algorithm=engine=nuts --max_treedepth=12

I was only suggesting this in the case that something like "stepsize" was ambiguous
between, say, a warmup optimizer and a post-warmup sampler step size, that you use more
specific names.

There are two kinds of pain to users. One is trying to figure
out some new interface, the other is typing long strings. Often
the long forms make the figuring out easier and sometimes they
make it harder.

Also, the double equals for algorithm/metric is not standard,
so I certainly wouldn't want to introduce compound arguments.

> There's far too much typing required of the user, which is not only frustrating but also asking for typos. And we've only specified a few arguments here!
>

> While it would be great to have a better UI (imagine context-dependent pop-up lists! What we could do with a web-interface and some javascript�) we're limited to the CLI and we're going to have to make some compromise. Personally, I believe that users will pick up the current implementation quickly enough and full path specification of each argument is typo- and fatigue-inducing overkill.

I prefer things I can script and read pretty easily. But I
know opinions vary.

> But I'm also biased. :-)

So what's the deal with things like "random"?

What I really think we need is more argument lists like the above
in the doc as well as examples of what's not allowed.

- Bob

[Michael Betancourt]

> I understand --- but this is what I think is problematic if I'm
> understanding properly. Isn't there a "random" element that
> shows up and then defines what the following key-value pairs mean?

Yes. "random" is a categorical argument that defines a new level and key/value pairs within that level. "hmc", for example, is also a categorical argument that defines a new level and respective key/value pairs. The difference is that "hmc" is itself defined as a value of a higher argument whereas "random" is not a value. In other words, "hmc" is one of a mutually-exclusive set whereas "random" is not mutally-exclusive with any other arguments.

> This is totally OK by me. The hierarchy is an implementation detail.
>
> Is this order independent, so that I can put the arguments in any order?

Yup, the arguments within a level are permutation invariant.

> I also don't see any bare arguments like "random" here that change
> the meaning of the following parameters.

"sample", "hmc", and "nuts" are all categorical variables that define new levels (contexts, meaning, etc). As noted above, the difference is that they exist as values. "random" is not derived from any other arguments and instead exists on its own.

>
> Also, the double equals for algorithm/metric is not standard,
> so I certainly wouldn't want to introduce compound arguments.

Exactly. My point is that if we want categorical arguments that are values to make sense then we have to have bare arguments.

> So what's the deal with things like "random"?

See above. "random" and "adapt" are not values for other arguments so they appear "bare".

> What I really think we need is more argument lists like the above
> in the doc as well as examples of what's not allowed.

Sure. Some graphical representations of configurations (with not-included branches of the tree greyed out) would be nice, it just depends on how much detail we'd want to put in.

[Michael Betancourt]

I pushed updated doc with diagrams trying to show what the categorical arguments are doing. Is the expanded explanation any cleaner? We can add a following section with various examples.

>> While it would be great to have a better UI (imagine context-dependent pop-up lists! What we could do with a web-interface and some javascript…) we're limited to the CLI and we're going to have to make some compromise. Personally, I believe that users will pick up the current implementation quickly enough and full path specification of each argument is typo- and fatigue-inducing overkill.

>
> I prefer things I can script and read pretty easily. But I
> know opinions vary.
>
>> But I'm also biased. :-)
>
> So what's the deal with things like "random"?
>
> What I really think we need is more argument lists like the above
> in the doc as well as examples of what's not allowed.
>
> - Bob
>

[Bob Carpenter]

On 8/11/13 6:33 PM, Michael Betancourt wrote:
> I pushed updated doc with diagrams trying to show what the categorical arguments are doing. Is the expanded explanation any cleaner? We can add a following section with various examples.

Cool. I'm going to try to add the examples now.

And then maybe we can throw it to our users to test.

If the unusual command-line structure turns out to be too confusing,
we can always change it later. It's independent of the lower-level
refactoring you've done on the MCMC package and of the language.

- Bob