output progress
Krzysztof Sakrejda
In general there are four "writers":
Info writer: key=value for per-run stuff, configuration and other single values (e.g.-adapted step size) always output complete configuration, is a bug otherwise, should be usable for reproducing a run, is a bug otherwise
Progress writer: key-value with log level for progress messages, respects log4j log levels
Output writer: csv-style output file, estimates and related diagnostic quantities
Diagnostic writer: csv-style diagnostic file, internal versions of parameters and related diagnostic quantities, potentially respects log4j log levels
If anyone has comments at this stage let me know.
Krzysztof
Bob Carpenter
very helpful.
As to goal, I don't think there needs to be a consolidated
output format for MCMC, MLE, and VB (or diagnostics) in the
sense that each produces output mutually readable by the other.
What I do think we want is a way to round trip all of
the following in a modular reader/writer fashion:
1) single parameter values
2) sequences of parameter values, perhaps named
(e.g., MCMC init value vs. draws; VB init vs. draws vs.
best fit)
3) vectors and matrices (if we don't want to make this
just an instance of parameter values)
4) key-value pairs for config information (I see that you're
structuring keys rather than allowing values to have
structure --- just pointing that out, I can see arguments
either way)
5) specific relevalnt types, like time
It would be possible to put all this together into
some kind of structure if we can stream it all out in
a reasonable way.
I think that's what's being targeted in the "BIG DESIGN DECISION"
section, but I'm not sure.
There are some nitpicky little details here and there. The timing
estimate, for example, isn't really part of the config for HMC.
Nor are the adapted step size or mass matrix, though you'd need to
save those if you wanted to restart. So what we need is a step size
(or integration time) and mass matrix; you already have the former
but we need to generalize to the latter in Stan 3.
We don't need to build all of these on top of a base writer.
So I'd be more inclined to have progress writer (or other loggers)
have methods .warn(), .error(), .info(), etc than have
it use key-values with keys defined. For one things, keys are
very slow. We also need an .if_warn(), ..., and so forth, so we
can check what level the logger's at so we can avoid constructing
messages that will get swallowed.
I like the "sample writer" abstraction --- that's what I was talking
about above. Then we'd replace the current CSV reader thing with
something that modularly actually used a proper CSV reader (that is, one
that only reads CSV files and ignores comment lines, not one that
requires specific comment content).
- Bob
- Bob
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Krzysztof Sakrejda
> Thanks for the inventory of what's going on now. That's
> very helpful.
You and Daniel wrote the current inventory! :) K
Krzysztof Sakrejda
> very helpful.
>
> As to goal, I don't think there needs to be a consolidated
> output format for MCMC, MLE, and VB (or diagnostics) in the
> sense that each produces output mutually readable by the other.
I agree that it's not a big deal but after I spent a few minutes
with it I think the config(key:value)/messages(key:value)/output(csv)/diagnostics(csv) division
is pretty straightforward...
> What I do think we want is a way to round trip all of
> the following in a modular reader/writer fashion:
>
> 1) single parameter values
>
> 2) sequences of parameter values, perhaps named
> (e.g., MCMC init value vs. draws; VB init vs. draws vs.
> best fit)
Could you explain this a little more? I'm not clear on the
naming. I was thinking that the interfaces could get from messages
or config where init/warmup turns into samples.
>
> 3) vectors and matrices (if we don't want to make this
> just an instance of parameter values)
Honestly in the config/messages I was thinking of just writing
n_dim, n_rows, n_columns, n_..., ..., value1, value2, .... and
providing a function for the interfaces to use to read it back....
> 4) key-value pairs for config information (I see that you're
> structuring keys rather than allowing values to have
> structure --- just pointing that out, I can see arguments
> either way)
Yeah, I went for structuring keys since it keeps the C++
end simpler.
>
> 5) specific relevalnt types, like time
>
> It would be possible to put all this together into
> some kind of structure if we can stream it all out in
> a reasonable way.
>
> I think that's what's being targeted in the "BIG DESIGN DECISION"
> section, but I'm not sure.
I think Daniel wrote "BIG DESIGN DECISION"... I actually can' ttell for sure
what it meant...
>
> There are some nitpicky little details here and there. The timing
> estimate, for example, isn't really part of the config for HMC.
> Nor are the adapted step size or mass matrix, though you'd need to
> save those if you wanted to restart.
Yeah, 'config' is a mis-nomer and you are right that it's really
aggregating info needed ofr reproducibility.
> So what we need is a step size
> (or integration time) and mass matrix; you already have the former
> but we need to generalize to the latter in Stan 3.
> We don't need to build all of these on top of a base writer.
> So I'd be more inclined to have progress writer (or other loggers)
I was thinking that step 1 was just getting things structured and
formats agreed on and then the insides could shift around so
for a first round I was going to ignore the levels and use the
current writer framework... is that a bad idea?
> have methods .warn(), .error(), .info(), etc than have
> it use key-values with keys defined. For one things, keys are
> very slow. We also need an .if_warn(), ..., and so forth, so we
> can check what level the logger's at so we can avoid constructing
> messages that will get swallowed.
>
> I like the "sample writer" abstraction --- that's what I was talking
> about above. Then we'd replace the current CSV reader thing with
> something that modularly actually used a proper CSV reader (that is, one
> that only reads CSV files and ignores comment lines, not one that
> requires specific comment content).
Yep, on the same page here.
Krzysztof
Bob Carpenter
> On Jun 16, 2016, at 4:37 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>
> On Thursday, June 16, 2016 at 12:53:47 PM UTC-4, Bob Carpenter wrote:
>> Thanks for the inventory of what's going on now. That's
>> very helpful.
>>
>> As to goal, I don't think there needs to be a consolidated
>> output format for MCMC, MLE, and VB (or diagnostics) in the
>> sense that each produces output mutually readable by the other.
>
> I agree that it's not a big deal but after I spent a few minutes
> with it I think the config(key:value)/messages(key:value)/output(csv)/diagnostics(csv) division
> is pretty straightforward...
file outputs or object outputs in R/Python are
mutually readable.
>> What I do think we want is a way to round trip all of
>> the following in a modular reader/writer fashion:
>>
>> 1) single parameter values
>>
>> 2) sequences of parameter values, perhaps named
>> (e.g., MCMC init value vs. draws; VB init vs. draws vs.
>> best fit)
>
> Could you explain this a little more? I'm not clear on the
> naming. I was thinking that the interfaces could get from messages
> or config where init/warmup turns into samples.
mappings of parameters to values. But there are multiple
draws, so we probably don't want to repeat the naming
>>
>> 3) vectors and matrices (if we don't want to make this
>> just an instance of parameter values)
>
> Honestly in the config/messages I was thinking of just writing
> n_dim, n_rows, n_columns, n_..., ..., value1, value2, .... and
> providing a function for the interfaces to use to read it back....
easy to read and write.
a bad idea underneath, but I want to think about each of
the different interfaces in terms of what the natural callbacks
are.
- bob
Krzysztof Sakrejda
> > On Jun 16, 2016, at 4:37 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
> >
> > On Thursday, June 16, 2016 at 12:53:47 PM UTC-4, Bob Carpenter wrote:
> >> Thanks for the inventory of what's going on now. That's
> >> very helpful.
> >>
> >> As to goal, I don't think there needs to be a consolidated
> >> output format for MCMC, MLE, and VB (or diagnostics) in the
> >> sense that each produces output mutually readable by the other.
> >
> > I agree that it's not a big deal but after I spent a few minutes
> > with it I think the config(key:value)/messages(key:value)/output(csv)/diagnostics(csv) division
> > is pretty straightforward...
>
> That's lower level. The question's whether a set of
> file outputs or object outputs in R/Python are
> mutually readable.
Is there any reason to do this so that R/Python/CmdStan interfaces can't read each
others' output?
>
> >> What I do think we want is a way to round trip all of
> >> the following in a modular reader/writer fashion:
> >>
> >> 1) single parameter values
> >>
> >> 2) sequences of parameter values, perhaps named
> >> (e.g., MCMC init value vs. draws; VB init vs. draws vs.
> >> best fit)
> >
> > Could you explain this a little more? I'm not clear on the
> > naming. I was thinking that the interfaces could get from messages
> > or config where init/warmup turns into samples.
>
> I meant that init and draws are the same thing --- just
> mappings of parameters to values. But there are multiple
> draws, so we probably don't want to repeat the naming
Agreed, we just need to put the info somewhere that lets you
calculate which rows are inits, which are warmup, which are samples/estimates
I think we already had one go-around on this with Michael that came down
to a good interface:
1) https://github.com/stan-dev/stan/wiki/Logging-Spec
2) https://groups.google.com/forum/#!searchin/stan-dev/Betancourt%7Csort:date/stan-dev/YJvyzVTK_YQ/fzNhwbg4BgAJ
I'm guessing what you want is a spec that outlines the signatures for the calls
to the loggers? I can do that on the wiki and go from there.
Krzysztof
>
> - bob
Michael Betancourt
I think we already had one go-around on this with Michael that came down
to a good interface:
1) https://github.com/stan-dev/stan/wiki/Logging-Spec
2) https://groups.google.com/forum/#!searchin/stan-dev/Betancourt%7Csort:date/stan-dev/YJvyzVTK_YQ/fzNhwbg4BgAJ
I'm guessing what you want is a spec that outlines the signatures for the calls
to the loggers? I can do that on the wiki and go from there.
Michael Betancourt
Bob Carpenter
> On Jun 21, 2016, at 9:22 AM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>
> On Tuesday, June 21, 2016 at 1:24:11 AM UTC-4, Bob Carpenter wrote:
>>> On Jun 16, 2016, at 4:37 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>>>
>>> On Thursday, June 16, 2016 at 12:53:47 PM UTC-4, Bob Carpenter wrote:
>>>> Thanks for the inventory of what's going on now. That's
>>>> very helpful.
>>>>
>>>> As to goal, I don't think there needs to be a consolidated
>>>> output format for MCMC, MLE, and VB (or diagnostics) in the
>>>> sense that each produces output mutually readable by the other.
>>>
>>> I agree that it's not a big deal but after I spent a few minutes
>>> with it I think the config(key:value)/messages(key:value)/output(csv)/diagnostics(csv) division
>>> is pretty straightforward...
>>
>> That's lower level. The question's whether a set of
>> file outputs or object outputs in R/Python are
>> mutually readable.
>
> Is there any reason to do this so that R/Python/CmdStan interfaces can't read each
> others' output?
patterns are a bit lower level and in and of themselves
don't guarantee any consistency in output. They're just
callbacks.
So I think we need to spec out what actually gets written.
That can be done indirectly by specifying what the various
writers write.
> ...
> to a good interface:
>
> 1) https://github.com/stan-dev/stan/wiki/Logging-Spec
> 2) https://groups.google.com/forum/#!searchin/stan-dev/Betancourt%7Csort:date/stan-dev/YJvyzVTK_YQ/fzNhwbg4BgAJ
>
> I'm guessing what you want is a spec that outlines the signatures for the calls
> to the loggers? I can do that on the wiki and go from there.
that's again something that's going to be machine digestable.
- Bob
Bob Carpenter
wrap around loggers to produce new loggers. Some of them
turn off output, some tee it to multiple sources. So if you
had logger1 and logger2, you might have tee(logger1, logger2)
as a new logger implementing the same logging interface, or
filter_warn(logger1) if you don't go below warnings.
But then that's all usually wrapped up with file config for input
and output sources that are put together with reflection.
I don't think we need to get this crazy---just having the
right interface for logging is the way to start. And maybe an
output format.
- Bob
Michael Betancourt
In particular, do the interfaces want a bunch of parallel
sources or one source that gets filtered by the logger
itself.
Bob Carpenter
I was thinking the logger is an object instance
and there's just one, but you can send messages
to it at various log levels and then it can be
filtered to only print messages above a certain
severity. Basically, just another kind of writer.
The trick in all the frameworks is making the logger
available programatically without having to pass it
around everywhere. But I think we can just pass ours
around like the writers. Otherwise, it's going to need
to be global like the memory for the autodiff.
- Bob
Michael Betancourt
should that be implemented? Does the base
code pass in strings which are filtered by the
logger and _then_ passed to whatever sources
the interfaces provide? Or do the interfaces
provide a source for each level, and the logger
just passes strings to the appropriate source?
And do we want to pass strings or char arrays
or do the interfaces provide ostreams to which
various objects are dumped?
Bob Carpenter
trickier. I'd start with just this interface:
class logger {
// set level (all messages at level or above printed)
void set_debug();
void set_warn();
void set_error();
// query what level is set
bool is_debug() const;
bool is_warn() const;
bool is_error() const;
// log a message
void debug(const string& msg);
void warn(const string& msg);
void error(const string& msg);
}
and maybe some additional levels.
If you want to get fancy, then it'd be something like this:
ostream& debug();
ostream& warn();
ostream& error();
That'd be a much cleaner interface in C++, but then you need
to have no-op streams if the level's above the logger requested.
You want the "is_debug()" or something like it to avoid
constructing big strings that are then just thrown away,
especially at lower levels of granularity like debug.
- Bob
Michael Betancourt
so that’s not an issue. The question is whether or
not the interfaces can actually produce a stream
easily (i.e. if it’s much easier for them to accept
strings).
Bob Carpenter
just collect output into a buffer, so it's easy to produce
output streams. With an output stream, it's easy to feed
it strings. Just a matter of whether it's worth the extra
work.
- Bob