flag for warmup?

[Ben Goodrich]

Can we introduce a flag that is readable from within a Stan program indicating whether we are in the warmup phase?

In my case, I have a function that is called from the generated quantities block that does rejection sampling, but for sufficiently bad parameter draws, it can hang for an arbitrarily long amount of time. Maybe only rstan executes the generated quantities in the warmup phase? Anyway, I would rather return not_a_number() in the warmup phase and only risk hanging in the post-warmup phase when the draws are better.

Ben

[Andrew Gelman]

This goes along with being able to access chain number and iteration number, which could be useful for all sorts of algorithmic ideas.  I guess people might also want to access step number, but for NUTS that’s not so clear.

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonah]

Yeah this would be nice. I ran into this issue recently. Is there a reason to run generated quantities at all before any draws are retained?

[Ben Goodrich]

On Monday, August 3, 2015 at 11:57:28 PM UTC-4, Jonah wrote:

Yeah this would be nice. I ran into this issue recently. Is there a reason to run generated quantities at all before any draws are retained?

Technically, you can do traceplots and stuff with the warmup included, but that is not much of a reason.

Worse actually, the generated quantities block seems to get executed while optimizing.

[Bob Carpenter]

I think that's the main reason --- so that each iteration produces
the same output.

If we want to start introducing flags like this, what do we
do for optimization, diagnostics, and standalone evaluations.

If you look at the write_array() method of the generated C++
class, you'll see flags for include transformed parameters and
include_gqs. If you set include_tparams = false, then they won't
be written. If you set include_gqs = false, they won't get
computed. At least I'm pretty sure that's what happens. I now
need to go figure out how cmdstan has changed with library includes
so that I can build stanc again.

- Bob

[Ben Goodrich]

On Tuesday, August 4, 2015 at 12:15:33 AM UTC-4, Bob Carpenter wrote:

If we want to start introducing flags like this, what do we
do for optimization, diagnostics, and standalone evaluations.

For optimization, I can't see a reason to execute the generated quantities block until it has reached the optimum. The path the things in the parameters block took to the optimum isn't even very interesting but might be useful for debugging.

I think we still don't have any diagnostics for optimization, but for MCMC, they should be stored every iteration.

I'm not sure what standalone evaluations are. If you mean of the log-posterior function, then again I'm not seeing what the generated quantities block would need to be executed.

In general, I think having a lot of flags be readable in generated quantities would be a good thing. Having them be in scope for other blocks is more dangerous.

Ben

[Ben Goodrich]

On Tuesday, August 4, 2015 at 12:27:09 AM UTC-4, Ben Goodrich wrote:

On Tuesday, August 4, 2015 at 12:15:33 AM UTC-4, Bob Carpenter wrote:

If we want to start introducing flags like this, what do we
do for optimization, diagnostics, and standalone evaluations.

For optimization, I can't see a reason to execute the generated quantities block until it has reached the optimum. The path the things in the parameters block took to the optimum isn't even very interesting but might be useful for debugging.

I think we still don't have any diagnostics for optimization, but for MCMC, they should be stored every iteration.

And for variational inference, no generated quantities during the ELBO optimization phase but definitely do generated quantities in the phase where it transforms multivariate normal draws.

Ben

[Bob Carpenter]

Yes, I meant the log probability function evals.

I still don't see why we need a flag in the code itself.
Couldn't we just control whether the block was run
from the outside?

- Bob

[Ben Goodrich]

On Tuesday, August 4, 2015 at 12:51:32 AM UTC-4, Bob Carpenter wrote:

Yes, I meant the log probability function evals.

I still don't see why we need a flag in the code itself.
Couldn't we just control whether the block was run
from the outside?

Yes, that is the simplest thing to do, but I think it would need to be a configuration argument in the MCMC case.  Andrew is hoping for chain_id and iteration flags to be in scope in all the blocks.

Ben
 

> On Aug 4, 2015, at 12:31 AM, Ben Goodrich <goodri...@gmail.com> wrote:
>
> On Tuesday, August 4, 2015 at 12:27:09 AM UTC-4, Ben Goodrich wrote:
> On Tuesday, August 4, 2015 at 12:15:33 AM UTC-4, Bob Carpenter wrote:
> If we want to start introducing flags like this, what do we
> do for optimization, diagnostics, and standalone evaluations.
>
> For optimization, I can't see a reason to execute the generated quantities block until it has reached the optimum. The path the things in the parameters block took to the optimum isn't even very interesting but might be useful for debugging.
>
> I think we still don't have any diagnostics for optimization, but for MCMC, they should be stored every iteration.
>
> And for variational inference, no generated quantities during the ELBO optimization phase but definitely do generated quantities in the phase where it transforms multivariate normal draws.
>
> Ben
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.

[Bob Carpenter]

> On Aug 4, 2015, at 1:26 AM, Ben Goodrich <goodri...@gmail.com> wrote:
>
> On Tuesday, August 4, 2015 at 12:51:32 AM UTC-4, Bob Carpenter wrote:
> Yes, I meant the log probability function evals.
>
> I still don't see why we need a flag in the code itself.
> Couldn't we just control whether the block was run
> from the outside?
>
> Yes, that is the simplest thing to do, but I think it would need to be a configuration argument in the MCMC case. Andrew is hoping for chain_id and iteration flags to be in scope in all the blocks.

I remember Andrew saying that, but can't recall why. There's
an issue for that feature request:

https://github.com/stan-dev/stan/issues/1166

Is printing the only reason? If so, we could probably just
control it from the outside by passing in a a NULL print stream
for all but the specified iterations. If people need fine-grained
control, we'd need to pass it in.

Seems like less of a priority to me.

- Bob

[Andrew Gelman]

I wanted chain number and iteration number for various algorithms that do different things in different iterations, for example adaptation, or data subsampling, or stochastic algorithms that change the target at each step. It comes up from time to time and I just assumed that chain number and iteration number are out there, so why not be able to access them.
A

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Ben Goodrich]

On Tuesday, August 4, 2015 at 1:36:33 AM UTC-4, Bob Carpenter wrote:

I remember Andrew saying that, but can't recall why.

Besides printing, he says he wants to change the behaviour from iteration to iteration or chain to chain.

Ben

[Bob Carpenter]

Maybe we can take this up in a future meeting. It wouldn't
be that hard assuming we come up with some defaults for things
that are NOT going to run MCMC. Would it be iteration number for
VI and MLE, too? And I guess you want VI to run multiple chains
so I suppose you'll want MLE to run multiple chains too. Then
we just set it to what for diagnosis? We let the diagnosis program
pass it in.

All these things are relatively easy, but they have lots of edge
cases and have to be passed through all the interfaces.

- Bob