Argument Refactor

[Michael Betancourt]

I just pushed a recursive argument refactoring and I'd like to gauge everyone's opinions. Checkout arg_refactor, build any model (in this branch the nuts_command method just parses arguments and then returns, so the actual model doesn't matter), and run with no arguments to see the options. Then you can play around with specifying (both good and bad) values. Let me know what you think.

-Mike

[Daniel Lee]

I took a quick look at it. It's a good start, but it's not there yet.

My suggestion is to not include the changes for the next release. Michael, I'll help out with the arguments from teh command line.


Daniel

On Thu, Apr 25, 2013 at 6:12 AM, Michael Betancourt <betan...@gmail.com> wrote:

I just pushed a recursive argument refactoring and I'd like to gauge everyone's opinions.  Checkout arg_refactor, build any model (in this branch the nuts_command method just parses arguments and then returns, so the actual model doesn't matter), and run with no arguments to see the options.  Then you can play around with specifying (both good and bad) values.  Let me know what you think.

-Mike

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Michael Betancourt]

Yeah, I haven't touched it in a month.  But it works, and just needs to incorporate all of the changes to command

Unless you thought there were missing features, Dan?

[Daniel Lee]

It's hard to parse the text as is. And in a few minutes of using it, I saw some behavior I wasn't comfortable with. Additional, unused arguments don't stop the process. I think it should.

I'm thinking about something a little more elaborate, much like what git's doing. I'm going to take a look at git's source to see if we can do something similiar.

Daniel

[Bob Carpenter]

On 6/6/13 11:58 AM, Daniel Lee wrote:
> It's hard to parse the text as is. And in a few minutes of using it, I saw some behavior I wasn't comfortable with.
> Additional, unused arguments don't stop the process. I think it should.

Good idea. I don't think they do now.

> I'm thinking about something a little more elaborate, much like what git's doing. I'm going to take a look at git's
> source to see if we can do something similiar.

We should also just figure out what the right behavior is.
Can't be that hard to implement!


Unless we can roll out a release 1.4 that's backward compatible, I'd
like to get all the changes in for 2.0.

It's not nice to users to change the command-line structure twice.

- Bob

[Michael Betancourt]

Unused arguments definitely cause the command to fail in the parsing step.  If a parent argument has mutually exclusive child arguments A and B, for example, and you set options for both A and B it will fail.  I'm not sure to what you are referring.

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.

[Daniel Lee]

I'll need to look through the history of my commands to find it. I think I put in a "help" at the end and it just ran. 

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Daniel Lee]

I was wrong about the "help" argument. But having arguments that don't match anything allows the program to run. Like adding "he" or anything else misspecified.

There's also the problem of the src/models/basic_estimators/normal_censored.stan not running properly. All of the samples are nan. I think I need to merge develop into this branch. I'll try to get that going soon.

For those that haven't looked, here's the help option now:

> ./models/basic_estimators/normal_censored help
help
  Display valid arguments

id
  Unique process identifier
  Valid values: id > 0
  Defaults to 0

data
  Input data file
  Valid values: All
  Defaults to ""

init
  Initialization method:"0" initializes to zero,"random" initializes randomly,anything else identifies a file
  Valid values: All
  Defaults to "random"

test_gradient
  Check model gradient against finite differences

random
  Random number configuration
  Valid subarguments: seed

  seed
    Random number generator seed
    Valid values: seed > 0, if negative seed is generated from time
    Defaults to -1

output
  File Output Options
  Valid subarguments: sample append_sample diagnostic append_diagnostic refresh save_warmup

  sample
    Output file for sample information
    Valid values: All
    Defaults to samples.csv

  append_sample
    Append sample output to existing file?
    Valid values: [0, 1]
    Defaults to 0

  diagnostic
    Output file for diagnostic information
    Valid values: All
    Defaults to ""

  append_diagnostic
    Append diagnostic output to existing file?
    Valid values: [0, 1]
    Defaults to 0

  refresh
    Number of interations between screen updates
    Valid values: 0 < refresh
    Defaults to 100

  save_warmup
    Stream warmup samples to output?
    Valid values: [0, 1]
    Defaults to 0

method
  Analysis method
  Valid values: sample optimize
  Defaults to sample

  sample
    Bayesian inference with Markov Chain Monte Carlo
    Valid subarguments: iter warmup save_warmup thin adapt algorithm

    iter
      Total number of iterations
      Valid values: 0 < iter
      Defaults to 2000

    warmup
      Number of sampling iterations reserved for warmup
      Valid values: 0 < warmup
      Defaults to 1000

    save_warmup
      Stream warmup samples to output?
      Valid values: [0, 1]
      Defaults to 0

    thin
      Period between saved samples
      Valid values: 0 < thin
      Defaults to 1

    adapt
      Warmup Adaptation
      Valid subarguments: engaged gamma delta kappa t0

      engaged
        Adaptation engaged?
        Valid values: [0, 1]
        Defaults to 0

      gamma
        Adaptation regularization scale
        Valid values: 0 < gamma
        Defaults to 0.05

      delta
        Adaptation target acceptance statistic
        Valid values: 0 < delta < 1
        Defaults to 0.65

      kappa
        Adaptation relaxation exponent
        Valid values: 0 < kappa
        Defaults to 0.75

      t0
        Adaptation iteration offset
        Valid values: 0 < t0
        Defaults to 10

    algorithm
      Sampling algorithm
      Valid values: hmc metro
      Defaults to hmc

      hmc
        Hamiltonian Monte Carlo
        Valid subarguments: engine metric stepsize stepsize_jitter

        engine
          Engine for Hamiltonian Monte Carlo
          Valid values: static nuts
          Defaults to static

          static
            Static integration time
            Valid subarguments: int_time stepsize

            int_time
              Total integration time for Hamiltonian evolution
              Valid values: 0 < int_time
              Defaults to 2 * pi

            stepsize
              Step size for discrete evolution
              Valid values: 0 < stepsize
              Defaults to 1

          nuts
            The No-U-Turn Sampler
            Valid subarguments: max_depth

            max_depth
              Maximum tree depth
              Valid values: 0 < max_depth
              Defaults to 10

        metric
          Geometry of base manifold
          Valid values: unit_e diag_e dense_e
          Defaults to unit_e

          unit_e
            Euclidean manifold with unit metric

          diag_e
            Euclidean manifold with diag metric

          dense_e
            Euclidean manifold with dense metric

        stepsize
          Step size for discrete evolution
          Valid values: 0 < stepsize
          Defaults to 1

        stepsize_jitter
          Uniformly random jitter of the stepsize, in percent
          Valid values: 0 < stepsize_jitter < 1
          Defaults to 0

      metro
        Random Walk Metropolis Monte Carlo
        Valid subarguments:

  optimize
    Point estimation
    Valid subarguments: algorithm iter save_iterations

    algorithm
      Optimization algorithm
      Valid values: nesterov bfgs newton
      Defaults to nesterov

      nesterov
        Nesterov's accelerated gradient method
        Valid subarguments: stepsize

        stepsize
          Step size for discrete evolution
          Valid values: 0 < stepsize
          Defaults to 1

      bfgs
        BFGS with linesearch
        Valid subarguments:

      newton
        Newton's method
        Valid subarguments:

    iter
      Total number of iterations
      Valid values: 0 < iter
      Defaults to 2000

    save_iterations
      Stream optimization progress to output?
      Valid values: [0, 1]
      Defaults to 0

I'm thinking about simplifying the standard help output to look more like:

> ./models/basic_estimators/normal_censored help
usage: model [--version]
             [id=<number>]
             [init={0|random|<path>}]
             [data=<path>]
             <command> [<args>]

The most common commands are:
   sample       Samples the posterior parameters of the model
   optimize     Optimize the posterior parameters of the model
   test         Tests the model for numerical issues

See 'model help <command>' for detailed information on a specific command

And I'm thinking the detailed information can go into the subcommand help like:

> ./models/basic_estimators/normal_censored help sample

    Bayesian inference with Markov Chain Monte Carlo
    Valid subarguments: iter warmup save_warmup thin adapt algorithm

    iter
      Total number of iterations
      Valid values: 0 < iter
      Defaults to 2000

    warmup
      Number of sampling iterations reserved for warmup
      Valid values: 0 < warmup
      Defaults to 1000

    save_warmup
      Stream warmup samples to output?
      Valid values: [0, 1]
      Defaults to 0

    thin
      Period between saved samples
      Valid values: 0 < thin
      Defaults to 1

    adapt
      Warmup Adaptation
      Valid subarguments: engaged gamma delta kappa t0

      engaged
        Adaptation engaged?
        Valid values: [0, 1]
        Defaults to 0

      gamma
        Adaptation regularization scale
        Valid values: 0 < gamma
        Defaults to 0.05

      delta
        Adaptation target acceptance statistic
        Valid values: 0 < delta < 1
        Defaults to 0.65

      kappa
        Adaptation relaxation exponent
        Valid values: 0 < kappa
        Defaults to 0.75

      t0
        Adaptation iteration offset
        Valid values: 0 < t0
        Defaults to 10

    algorithm
      Sampling algorithm
      Valid values: hmc metro
      Defaults to hmc

      hmc
        Hamiltonian Monte Carlo
        Valid subarguments: engine metric stepsize stepsize_jitter

        engine
          Engine for Hamiltonian Monte Carlo
          Valid values: static nuts
          Defaults to static

          static
            Static integration time
            Valid subarguments: int_time stepsize

            int_time
              Total integration time for Hamiltonian evolution
              Valid values: 0 < int_time
              Defaults to 2 * pi

            stepsize
              Step size for discrete evolution
              Valid values: 0 < stepsize
              Defaults to 1

          nuts
            The No-U-Turn Sampler
            Valid subarguments: max_depth

            max_depth
              Maximum tree depth
              Valid values: 0 < max_depth
              Defaults to 10

        metric
          Geometry of base manifold
          Valid values: unit_e diag_e dense_e
          Defaults to unit_e

          unit_e
            Euclidean manifold with unit metric

          diag_e
            Euclidean manifold with diag metric

          dense_e
            Euclidean manifold with dense metric

        stepsize
          Step size for discrete evolution
          Valid values: 0 < stepsize
          Defaults to 1

        stepsize_jitter
          Uniformly random jitter of the stepsize, in percent
          Valid values: 0 < stepsize_jitter < 1
          Defaults to 0

      metro
        Random Walk Metropolis Monte Carlo
        Valid subarguments:

I think I can restructure that to make it a little easier to figure out... like pulling up something like:

algorithm={hmc|metropolis}

Am I thinking about this right? I'm trying to think how I'd want it to look as a user. Regarding splitting out test, I thought we'd have different tests down the line, like gradient now, Hessian later, etc.

[Michael Betancourt]

Hmm, can you give a specific example where unmatched arguments don't cause a failure? That might just be a leaking edge case.

The branch is waaaaaaaaaaay behind develop, and I can take care of merging. There's a bunch of nontrivial stuff to do with some of the arguments anyways.

I'm also happy to tweak the I/O around -- just let me know what you'd prefer to see. Remember, this was just my first attempt with no one else's input.

>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Daniel Lee]

On Fri, Jun 7, 2013 at 3:57 PM, Michael Betancourt <betan...@gmail.com> wrote:

Hmm, can you give a specific example where unmatched arguments don't cause a failure?  That might just be a leaking edge case.

Sure. I don't think it's an edge case:
> ./models/basic_estimators/normal_censored foo
foo is either mistyped or misplaced.

Exception: variable does not exist; processing stage=data initialization; variable name=U; base type=double
Diagnostic information:
Dynamic exception type: std::runtime_error
std::exception::what: variable does not exist; processing stage=data initialization; variable name=U; base type=double

It still tries to run, even though it picks up that "foo is either mistyped or misplaced."

 

The branch is waaaaaaaaaaay behind develop, and I can take care of merging.  There's a bunch of nontrivial stuff to do with some of the arguments anyways.

Yeah, I know. I was going to spend some time merging, if you don't have the time.

 

I'm also happy to tweak the I/O around -- just let me know what you'd prefer to see.  Remember, this was just my first attempt with no one else's input.

Yeah. I'm not going to leave you out to dry -- I'm going to help implement. If we are going to change our command structure, I think we should do it the best we can so we don't have to replace it multiple times. I'm trying to think of we should do something like:

Default:

./stan_model sample

Fully specified

./stan_model sample=nuts

or

./stan_model sample algorithm=nuts

where maybe we're defining the algorithm. I don't think we should just leave it hanging like:

./model sample nuts

Or should we?

[Michael Betancourt]

> On Fri, Jun 7, 2013 at 3:57 PM, Michael Betancourt <betan...@gmail.com> wrote:
> Hmm, can you give a specific example where unmatched arguments don't cause a failure? That might just be a leaking edge case.
>
> Sure. I don't think it's an edge case:
> > ./models/basic_estimators/normal_censored foo
> foo is either mistyped or misplaced.
>
> Exception: variable does not exist; processing stage=data initialization; variable name=U; base type=double
> Diagnostic information:
> Dynamic exception type: std::runtime_error
> std::exception::what: variable does not exist; processing stage=data initialization; variable name=U; base type=double
>
> It still tries to run, even though it picks up that "foo is either mistyped or misplaced."

Yeah, that's just bad exception handling. Should be easy to fix.

> The branch is waaaaaaaaaaay behind develop, and I can take care of merging. There's a bunch of nontrivial stuff to do with some of the arguments anyways.
>
> Yeah, I know. I was going to spend some time merging, if you don't have the time.

Give me the weekend? Arguments are next on my to-do list and I can get it into a working state faster than anyone else.

>
> I'm also happy to tweak the I/O around -- just let me know what you'd prefer to see. Remember, this was just my first attempt with no one else's input.
>
> Yeah. I'm not going to leave you out to dry -- I'm going to help implement. If we are going to change our command structure, I think we should do it the best we can so we don't have to replace it multiple times. I'm trying to think of we should do something like:
>
> Default:
> ./stan_model sample
>
> Fully specified
> ./stan_model sample=nuts
> or
> ./stan_model sample algorithm=nuts
>
> where maybe we're defining the algorithm. I don't think we should just leave it hanging like:
> ./model sample nuts
>
> Or should we?

Wait, does ./model sample nuts work? It shouldn't, and if it does it's an error on my part. nuts isn't a valid option for sample, only for algorithm.

[Daniel Lee]

No. Just wondering if it should. Those aren't actually valid.

[Michael Betancourt]

I tried to keep the syntax rigid to avoid any possible errors.

[Marcus Brubaker]

When we discussed this a while ago one idea was to have nuts, bfgs, nesterov, etc be separate commands and then sample and optimize would alias to whatever we wanted the default to be.  This would make some things easier since, for instance, bfgs and nesterov will take a completely different set of parameters.  Or rather, they would if it was possible.  BFGS doesn't have much in the way of user visible options right now, but I would like to change that if it wouldn't pollute the overall options space.

I like the var=val convention as I think it's clearer.

I think it also makes sense to have some global options as well.  E.G., data and init, although maybe it's just best to reimplement them for each subcommand in case we need to do specialization like an init for an ensemble sampler.

Cheers,

Marcus

[Bob Carpenter]

On 6/8/13 12:07 PM, Marcus Brubaker wrote:
> When we discussed this a while ago one idea was to have nuts, bfgs, nesterov, etc be separate commands and then sample
> and optimize would alias to whatever we wanted the default to be.

I think this makes sense. We can always add a default
"sample" and "optimize" command later.

> This would make some things easier since, for
> instance, bfgs and nesterov will take a completely different set of parameters.

Right.

One implication of this is that the default sampler or optimizer
shouldn't have any options, because they'd change if the
defaults changed.

Thinking about this, I'm wondering if defaults will be more
trouble than they're worth.

> Or rather, they would if it was
> possible. BFGS doesn't have much in the way of user visible options right now, but I would like to change that if it
> wouldn't pollute the overall options space.
>
> I like the var=val convention as I think it's clearer.

I agree.

> I think it also makes sense to have some global options as well. E.G., data and init, although maybe it's just best to
> reimplement them for each subcommand in case we need to do specialization like an init for an ensemble sampler.

I think they can share code, at least, if
they have similar functionality. For doc,
we can have the options be hierarchical, but
each command should just dump out its own complete
doc.

./model nuts --help

would dump out help options just for NUTS. But then I
suppose that could refer to things it inherits from general
HMC, which in turn inherits options from Metropolis,
which inherits methods from MCMC, ...

- Bob



- Bob

[Michael Betancourt]

Are you guys looking at the argument refactor code and finding problems or just talking about desired behavior? Unless I'm mistaken the code behaves almost exactly like you've described below.

[Michael Betancourt]

I've pushed the merge to origin, so arg_refactor should have all of the necessary functionality. Could everyone interested in the refactor pull and give it a try? Running ./model_name help will give you a list of all options. Do try your best to break it -- it should be pretty robust but you never know.

It seems like the one big outstanding feature request is having context-specific help to only print out the argument tree from a given node. Is that correct?

Dan, let me know what format you'd prefer and I can clean things up.

On Jun 9, 2013, at 4:02 AM, Bob Carpenter <ca...@alias-i.com> wrote:

>
>

[Marcus Brubaker]

Sorry, I haven't actually looked at the code in that branch or played with it yet.  I've been swamped the last week or so (and will likely continue to be for the next two or three...)

[Michael Betancourt]

Dan,

How about `model help` just prints the valid arguments and prompts the user to call help on the individual argument? In any case, should the printing of the valid arguments include the verbose information (but without printing any help on child arguments) or just the name of the command?

Any preference on the behavior of the subcommand? In particular, between:

a)

Calling `model <argument> help` just prints information on the immediate child arguments but stops there. Users would have to dig through the hierarchy one layer at time to go deeper.

If so, same question as above -- print verbose information or just a list of valid child arguments?

b)

Calling `model <argument> help` prints information about the child arguments and their child arguments recursively.

c)

All arguments have a help and help-all option. help prints the immediate child arguments, help-all prints all child arguments recursively.

-Mike

>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

>> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Michael Betancourt]

I just pushed some updates to admit much more flexible help options. Give it a try starting with 'model help' and go from there.

Thoughts?