output format
Krzysztof Sakrejda
https://github.com/stan-dev/stan/wiki/Output-format
I'm ok with doing exploration around suggested libraries but (as you'll see from the JSONlines entry) if I can't think of positives for an option, I won't try very hard. I think we can have this discussion now and settle on a overall strategy soon. I'm on board for implementing something (reasonable in scope) and, as suggested by Daniel, I'm hoping Bob can keep the design to something practical.
It would be great to make this a short-term project so if you have opinions pile on now... (here or at the Wiki page if it's more concrete). Having a good set format and tools to make it compatible with multiple interfaces would make the whole Stan-for-everything strategy a lot easier on interface maintainers.
Krzysztof
Michael Betancourt
which means only two output formats (sequences of key/value pairs and CSV files.
Altogether we could have as many output files as
config: key/value pairs
warmup_samples: csv
warmup_diagnostics: csv
samples: csv
diagnostics: csv
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Krzysztof Sakrejda
.CSV format.
Michael Betancourt
information about the state of the Markov chain.
On Mar 17, 2016, at 5:54 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
> Where does stuff like the mass matrix go? That won't fit in a diagnostics
> .CSV format.
>
Krzysztof Sakrejda
Bob Carpenter
hard to write good C++ type-safe parsers that produce
objects from JSON as opposed to those that produce
parsers that accept callbacks. See the JSON parser
in stan/io.
When people say CSV here, are we really talking about
something with the flexibility of an R data frame? That
is, bunch of columns, each with a data type, then a bunch
of rows? Are the types just going to be integer, real, bool,
and string?
- Bob
> On Mar 17, 2016, at 2:47 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>
> So that adds a sixth metadata file that's also key/value. It probably doesn't make sense to jam that into config. The CSV files should be easy to concatenate across chains without losing info so I'm going to suggest pre-pending each row with the chain ID.
>
Michael Betancourt
just storing samples here.
Bob Carpenter
values? Or chain IDs, which sound like they'll be integers or
strings?
- Bob
Michael Betancourt
having separate chains in separate files works perfectly
well. Everything else gets treated as reals when computing
expectations and standard errors and other such posterior
estimates.
Krzysztof Sakrejda
> JSON wold be reasonable for key-value pairs. It's
> hard to write good C++ type-safe parsers that produce
> objects from JSON as opposed to those that produce
> parsers that accept callbacks. See the JSON parser
> in stan/io.
We use callbacks everywhere else so they should be fine here too,
but I looked at the JSON parser and did not find any examples of
what the callback "Handler" is supposed to be. Is there an example
somewhere?
>
> When people say CSV here, are we really talking about
> something with the flexibility of an R data frame? That
> is, bunch of columns, each with a data type, then a bunch
> of rows? Are the types just going to be integer, real, bool,
> and string?
I was assuming integer/double but without much type enforcement,
sort of the way we currently write some sampler parameters (e.g.,
tree_depth) as doubles even though they are really integers.
Bob Carpenter
> On Mar 17, 2016, at 4:19 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>
> On Thursday, March 17, 2016 at 3:35:49 PM UTC-4, Bob Carpenter wrote:
>> JSON wold be reasonable for key-value pairs. It's
>> hard to write good C++ type-safe parsers that produce
>> objects from JSON as opposed to those that produce
>> parsers that accept callbacks. See the JSON parser
>> in stan/io.
>
> We use callbacks everywhere else so they should be fine here too,
> but I looked at the JSON parser and did not find any examples of
> what the callback "Handler" is supposed to be. Is there an example
> somewhere?
with SAX parsing of XML? It's the same pattern. There should
be some simple unit tests, but not a real application. The handler
would do thing like look along certain paths for numbers and
sum them. Or stuff them in a database or whatever.
>> When people say CSV here, are we really talking about
>> something with the flexibility of an R data frame? That
>> is, bunch of columns, each with a data type, then a bunch
>> of rows? Are the types just going to be integer, real, bool,
>> and string?
>
> I was assuming integer/double but without much type enforcement,
> sort of the way we currently write some sampler parameters (e.g.,
> tree_depth) as doubles even though they are really integers.
vice-versa. It's a disaster for both speed and error handling.
R just works the way R works, and it seemed to much of people
to ask them to explicitly make 1 an integer because R just
straight-up assumes it's a double if not otherwise informed.
Our own data parsing does the opposite.
- Bob
Krzysztof Sakrejda
> I don’t think we need chain ID in the CSV itself — right now
> having separate chains in separate files works perfectly
> well.
1) strip comments, including post-header comments
2) track chain id outside the file, maybe in the filename or somewhere else,
3) write it into each .csv file before/during a merge, potentially with an
iteration number.
This gets in the way of my workflow and I don't think it's an
unusual thing to want to concatenate matching .csv files. If there's
an actual downside to including a chain id (integer) could you
clarify what you're thinking of?
Michael Betancourt
Krzysztof Sakrejda
> bin/print handles it seamlessly by taking in multiple files and
> storing them as a vector of matrices of state values x iterations.
Krzysztof Sakrejda
> > On Mar 17, 2016, at 4:19 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
> >
> > On Thursday, March 17, 2016 at 3:35:49 PM UTC-4, Bob Carpenter wrote:
> >> JSON wold be reasonable for key-value pairs. It's
> >> hard to write good C++ type-safe parsers that produce
> >> objects from JSON as opposed to those that produce
> >> parsers that accept callbacks. See the JSON parser
> >> in stan/io.
> >
> > We use callbacks everywhere else so they should be fine here too,
> > but I looked at the JSON parser and did not find any examples of
> > what the callback "Handler" is supposed to be. Is there an example
> > somewhere?
>
> Depends on what you want to do with it. Are you familiar
> with SAX parsing of XML? It's the same pattern. There should
> be some simple unit tests, but not a real application. The handler
> would do thing like look along certain paths for numbers and
> sum them. Or stuff them in a database or whatever.
Just looked up SAX parsing and found the abstract base class so it makes sense!
> >> When people say CSV here, are we really talking about
> >> something with the flexibility of an R data frame? That
> >> is, bunch of columns, each with a data type, then a bunch
> >> of rows? Are the types just going to be integer, real, bool,
> >> and string?
> >
> > I was assuming integer/double but without much type enforcement,
> > sort of the way we currently write some sampler parameters (e.g.,
> > tree_depth) as doubles even though they are really integers.
>
> We don't do that in the C++. We promote ints to doubles, but *not*
> vice-versa. It's a disaster for both speed and error handling.
right, we write integers and doubles as doubles. Reading is left to the
interfaces but I'm not sure that we can really say "we don't do that"
when the interfaces have to read in our "int-as-double-as-text" and,
for a column called "n_leapfrog", they have to convert text to int. We
don't have a binary format so I don't think you can blame that on the
interfaces.
>
> R just works the way R works, and it seemed to much of people
> to ask them to explicitly make 1 an integer because R just
> straight-up assumes it's a double if not otherwise informed.
> Our own data parsing does the opposite.
Yeah... R... I'll leave that right there...
So far from the discussion with you and Michael it seems like multiple .csv files with
JSON for the key/value files should be a good overall text format. I'll summarise this
for the Wiki and see what else pops up.
Krzysztof
Krzysztof Sakrejda
> On Thursday, March 17, 2016 at 4:51:04 PM UTC-4, Michael Betancourt wrote:
> > bin/print handles it seamlessly by taking in multiple files and
> > storing them as a vector of matrices of state values x iterations.
>
> It also fails for large models---at least it did last time I checked.
going to produce text .csv files so that it's easy to use text tools with these files
we should make it easy to use text tools, broadly conceived, rather than
Stan-specific tools we have to maintain.
K
Bob Carpenter
command line itself, which is a string. So we pretty much have to
deal with converting strings. If we did this all in C++, all the types
would stay coherent. For R, if you write 1, R's the one that decided
that should be the same as writing c(1.0); you can coerce it to do something
else, but it's no longer idiomatic R (and it still trips people up because
we insist arrays are arrays, but if you pass in just c(1.0), it won't be
the right type because of the double-duty for scalars and 1D arrays.
>> R just works the way R works, and it seemed to much of people
>> to ask them to explicitly make 1 an integer because R just
>> straight-up assumes it's a double if not otherwise informed.
>> Our own data parsing does the opposite.
>
> Yeah... R... I'll leave that right there...
>
> So far from the discussion with you and Michael it seems like multiple .csv files with
> JSON for the key/value files should be a good overall text format. I'll summarise this
> for the Wiki and see what else pops up.
with a format that includes the chain ID. I don't think it's
that much overhead to save, but it would mean a more complicated
CSV reader.
- Bob
Krzysztof Sakrejda
Even if we just stuck to a semi-standardized binary format the types could
stay coherent. We're not talking about serializing arbitrary C++ classes.
> For R, if you write 1, R's the one that decided
> that should be the same as writing c(1.0); you can coerce it to do something
> else, but it's no longer idiomatic R (and it still trips people up because
> we insist arrays are arrays, but if you pass in just c(1.0), it won't be
> the right type because of the double-duty for scalars and 1D arrays.
>
> >> R just works the way R works, and it seemed to much of people
> >> to ask them to explicitly make 1 an integer because R just
> >> straight-up assumes it's a double if not otherwise informed.
> >> Our own data parsing does the opposite.
> >
> > Yeah... R... I'll leave that right there...
> >
> > So far from the discussion with you and Michael it seems like multiple .csv files with
> > JSON for the key/value files should be a good overall text format. I'll summarise this
> > for the Wiki and see what else pops up.
>
> We can try to accomodate other interfaces other than CmdStan
> with a format that includes the chain ID. I don't think it's
> that much overhead to save, but it would mean a more complicated
> CSV reader.
I'm not sure why the cvs reader would be more complicated. I guess it's special treatment for each value of each row but that seems pretty minor. Am I missing something here?
>
> - Bob
Bob Carpenter
> On Mar 17, 2016, at 5:41 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>
> On Thursday, March 17, 2016 at 5:14:51 PM UTC-4, Bob Carpenter wrote:
>> We can try to accomodate other interfaces other than CmdStan
>> with a format that includes the chain ID. I don't think it's
>> that much overhead to save, but it would mean a more complicated
>> CSV reader.
>
> I'm not sure why the cvs reader would be more complicated. I guess it's special treatment for each value of each row but that seems pretty minor. Am I missing something here?
or inferring column types (remember how good Excel is at this?), and
our tests multiply tremendously. With just real values, it's trivial.
- Bob
Krzysztof Sakrejda
Sigh, I thought one thing and wrote another. I was thinking special treatment for each _first_ value of each row (so key has a known position, no inference necessary). I'm agree 110% with staying clear of type inference. If we're doing anything more complicated with types we should just do binary where you can be clear about it.
Krzysztof
>
> - Bob