cleaning up Stan output

[Andrew Gelman]

Currently Stan outputs an update every 10th of the run. That’s 40 outputs in a typical 4-chain run. 40 lines of output makes it impossible to read the console.

Here’s my proposal:

1. Just spit out two messages: one after 10% of warmup iterations giving estimated total time, another after 10% of the sampling iterations with an updated estimated time, since they can be faster than warmup.

2. Just print one pair of messages on the console, no matter how many chains are running. Simplest would be to just wait until all 4 (or however many) chains have sent out their 10% messages, then print the latest of these onto the console.

3. Also we should have a no-print option. We could have multiple verbose settings, but I’d like a default that is something like my suggestion above, also a no-print option, also maybe an option that prints out more, that I dunno.

A

[Bob Carpenter]

I like our current output. I wouldn't mind having an Andrew-mode
that does (1) for cases that only take a few seconds to run. Otherwise,
I want intermediate "pacifier" output to let me know it's still going.
When you're running something that takes 24 hours, getting one
message after 3 hours and another after 13 hours isn't so helpful,
especially when the time estimates will be super noisy.

Doing (2) presents technical challenges with interprocess
communication that are going to be a pain, if not impossible.

(3) we have, though I believe it may still produce some output.
This is something we need to fix, but I don't even know if there's
an issue for it.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

[Andrew Gelman]

I’d still like to do #2, at least in rstan, but let’s wait on that for now! Once we do the ragged R-hat thing we can do the run-for-fixed-time version anyway.

Regarding #1, perhaps we can have a default which does the printout after 10% and then decides what to do. If the first 10% took less than X minutes, it does my strategy. If it took more than X minutes, it does Bob’s strategy. We could for example set X=15. (Bob’s note below suggests that X should be something more than 1/60 and something less than 180.)

A

[Michael Betancourt]

Adaptation is not highly non linear so the 10% extrapolation will be a terrible estimate.

[Andrew Gelman]

Yes, indeed, but (a) I only want something rough (seconds vs minutes vs hours); (b) adaptation typically starts slow so it’s typically a conservative estimate, I think; (c) that’s why I suggest the later estimate after 60%; (d) I’m sick of having my console fill up with output every time I fit any Stan model.
A

[Michael Betancourt]

The models you’re fitting are too easy… we expanded my malaria model
earlier this week and now it’s taking over an hour to run so I appreciate every little
update of progress. :-o

[Andrew Gelman]

OK, you can set X=6 in your Stan calls, then.

[Ben Goodrich]

On Thursday, March 24, 2016 at 6:09:47 PM UTC-4, Andrew Gelman wrote:

I’d still like to do #2, at least in rstan, but let’s wait on that for now!

The output is currently annoying, but it is no more annoying in rstan than any other interface. I don't want to write an R-specific output generator, so we need to do it for Stan as a whole. I would prefer updates at specified intervals of time rather than by specified intervals of iterations.

Ben

[Andrew Gelman]

Ben:

I agree.  On my agenda is to work out ragged R-hat and then we can work with Bob on the Stan-for-fixed-time thing.  It will also help once Stan can take mass matrix (and any other tuning parameters, if there are any) as input, because this will make it easier for people to start with short runs and then restart.

A

[Bob Carpenter]

How hard the ragged R-hat will be will depend on how heavily
our interfaces assume that MCMC output is rectangular.

Having updates specified by time might make everyone happy.

- Bob

[Jonah Gabry]

For rstan, shinystan, and rstanarm it will be a big pain in the ass to move to ragged structures, in large part because of the implications for all the post-processing functions that assume equal posterior sample sizes. It's definitely doable though.

Jonah

[Bob Carpenter]

Are these calling the Stan C++ versions or implementing
things like ESS and R-hat themselves?

- Bob

> On Mar 24, 2016, at 9:56 PM, Jonah Gabry <jga...@gmail.com> wrote:
>
> For rstan, shinystan, and rstanarm it will be a big pain in the ass to move to ragged structures, in large part because of the implications for all the post-processing functions that assume equal posterior sample sizes. It's definitely doable though.
>
> Jonah
>

[Jonah Sol Gabry]

They're all doing what rstan does, which for most things means computing them in R I believe (but need to double check on a few).

And then there are the plotting functions and posterior analysis functions that also often assume equal lengths for the chains.

You received this message because you are subscribed to a topic in the Google Groups "stan development mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-dev/YHY_VrigUpM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-dev+u...@googlegroups.com.

[Jonah Sol Gabry]

Anything involving matrices or arrays would have to be converted to lists or a database format (fancy lists). Both the storage type and the operations inside the functions would need to be changed unfortunately.

[Ben Goodrich]

On Thursday, March 24, 2016 at 11:50:44 PM UTC-4, Jonah Gabry wrote:

Anything involving matrices or arrays would have to be converted to lists or a database format (fancy lists). Both the storage type and the operations inside the functions would need to be changed unfortunately.

We would probably pad with NAs the chains that ran shorter than the longest chain.

[Jonah Sol Gabry]

Yeah, I like that as a short term solution, as it would only really require making sure all the functions were NA compatible. It could be nice to have a database storage structure at some point.

--