persistent data format

[Ross Boylan]

I notice the recent Schmefactor thread mentions plans to switch to JSON
for reading/saving data. It might be worth considering HDF5, which I
think would require less space since it doesn't save numbers as text.

I'm relatively ignorant of both formats, though I just used HDF5 to get
some data from Julia to R.

Ross Boylan

[Bob Carpenter]

Not switch, add!

We could add HDF5, too, but I'm not sure it's a good idea.

First, is there a "5" in the name because it's a moving target?
The nice thing about human-readable data like JSON is that
it's always easy to munge into some other format later. And
JSON and R dump formats have been stable for quite a while now.

The examples I saw on the HDF site save numbers as text,
as in this one:

http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/hdf5-examples/1_8/C/H5D/testfiles/h5ex_d_alloc.ddl

It looks way too complex for us to implement parsers for, so
it'd require a dependency, which we're not too keen on.

The HDF Group distributes code under BSD, so at least
that's compatible.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Ross Boylan]

On Tue, 2014-07-29 at 18:09 -0400, Bob Carpenter wrote:
> Not switch, add!
>
> We could add HDF5, too, but I'm not sure it's a good idea.
>
> First, is there a "5" in the name because it's a moving target?
> The nice thing about human-readable data like JSON is that
> it's always easy to munge into some other format later. And
> JSON and R dump formats have been stable for quite a while now.

Wikipedia says HDF5 has been in use since 2002, and HDF4 was the first
version (!). They are both currently supported, but are incompatible.

>
> The examples I saw on the HDF site save numbers as text,
> as in this one:
>
> http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/hdf5-examples/1_8/C/H5D/testfiles/h5ex_d_alloc.ddl
>
> It looks way too complex for us to implement parsers for, so
> it'd require a dependency, which we're not too keen on.

I think that's the result of one of the commands to provide a human
friendly summary; maybe it's an alternate file format. I'm pretty sure
HDF is a binary format; see
http://www.ncsa.uiuc.edu/NARA/XML_and_Binary.pdf which lists it as a
binary format, or the various types associated with the API for writing
data. The hdf files I wrote were binary.

One oddity, considering it's supposed to be a complete description,
including metadata, is that I read a lot of cautions that writing a
matrix to disk from a C like language and reading it from a Fortran like
language would switch the rows and columns (column vs row-major
storage).

Another is that, on a quick look, it appears the data type include the
endianness of the data
(http://www.hdfgroup.org/HDF5/doc/RM/PredefDTypes.html), which sounds as
if it's on the client code to notice and deal with endianness changes.
I could be wrong; the library may take care of it automatically.

In any event, you'd want to use the existing hdf library to use the
format (oddly, I couldn't even find the on-disk format described
anywhere in my quick look at the hdf site--it was all about the
conceptual model, the API, and tools). That would be another
dependency. I would expect the JSON implementation on stan to use a
JSON library...

Ross

[Ross Boylan]

On Tue, 2014-07-29 at 17:01 -0700, Ross Boylan wrote:
> On Tue, 2014-07-29 at 18:09 -0400, Bob Carpenter wrote:
> > Not switch, add!
> >
> > We could add HDF5, too, but I'm not sure it's a good idea.
> >
> > First, is there a "5" in the name because it's a moving target?
> > The nice thing about human-readable data like JSON is that
> > it's always easy to munge into some other format later. And
> > JSON and R dump formats have been stable for quite a while now.
> Wikipedia says HDF5 has been in use since 2002, and HDF4 was the first
> version (!). They are both currently supported, but are incompatible.

Also, the HDF people are very concerned that hdf files remain useable
into the distant future:
http://www.hdfgroup.org/pubs/papers/HDFandpreservation_NIST_2010_paper_Folk.pdf

[Jeffrey Arnold]

Although I've never used it, but I've heard good things about it. It has pretty good cross-language support, although much better in Python than in R (the rhdf5 package is on Bioconductor, not CRAN).

Also, PyMC uses hdf5 as one of its data backends (http://pymc-devs.github.io/pymc/database.html).  

[Bob Carpenter]

Mitzi and I wrote a JSON library for Stan. None of the existing
ones had everything we were looking for:

* documentation
* BSD-compatible license
* SAX-like streaming behavior
* full implementation of the spec for UTF-8 input

I also don't think any of them handled real and integer
data generally enough for what we want to do, but that's
a minor nitpick we could've coded around.

JSON's a very easy standard to code if you're used to writing
ad hoc parsers :-) I think the first parser I wrote only took
a couple hours, then Mitzi finished off the character handling,
testing, and subsequent bug fixes. I think it maybe took her
another day or two total (she did the hard work --- laying down
something with the right structure that doesn't fully work is
easy!).

- Bob

[Bob Carpenter]

I'm at least glad to hear it's been stable since 2002.

To enable binary data to work cross-platform, a format needs to
specify endian-ness, and mismatched platforms need to swap
bytes. That's how UTF-16 and UTF-32 encodings of Unicode
are designed, for example.

Transforming strings to numbers is very expensive. My
thinking for the original design was that reading in data was
unlikely to be a bottleneck for MCMC. It is turning out
to be a bottleneck for optimization, though, which is much
much faster than MCMC.

Not getting the row-major versus column-major working through
all interfaces seems like a bug in the interface, or maybe I'm
not understanding some underlying principles. In any case, we
probably don't need to worry about Fortran. But then R and
C++ have different defaults (C++ arrays are row major, R's are
column major). Mixing these up causes no end of trouble, as when
BUGS adopted R's dump format, then put the arrays in opposite
majority (row major vs. column major).

- Bob

[Ross Boylan]

Short version: it looks as if HDF5 handles all the things we were
worried about its not handling.

On Tue, 2014-07-29 at 23:55 -0400, Bob Carpenter wrote:
> I'm at least glad to hear it's been stable since 2002.

The original hdf, aka hd4, was from 1988. However, it's explicitly
designed to permit new datatypes, either user or system in the future.
E.g., if future hardware has 512 byte floats that could be added.

>
> To enable binary data to work cross-platform, a format needs to
> specify endian-ness, and mismatched platforms need to swap
> bytes. That's how UTF-16 and UTF-32 encodings of Unicode
> are designed, for example.

Looking at the tutorials, esp
http://www.hdfgroup.org/HDF5/Tutor/rdwt.html which says
"The data in memory can have a datatype different from that of the file
and can also be of a different size (i.e., the data in memory is a
subset of the dataset elements, or vice versa)."
it seems things "just work":
1) client code doesn't need to swap bytes or check data type on disk;
2) client code can use platform native data types, which are stored with
full information about their endianness.
So if you read data on a platform with a different format from the one
it is written, the hdf library will know that from the info on disk and
adjust accordingly.

See also http://www.hdfgroup.org/HDF5/Tutor/datatypes.html

>
> Transforming strings to numbers is very expensive. My
> thinking for the original design was that reading in data was
> unlikely to be a bottleneck for MCMC. It is turning out
> to be a bottleneck for optimization, though, which is much
> much faster than MCMC.

My distaste for storing large amounts of numeric data as text is
probably as much aesthetic as practical, though the sheer bulk of
putting numbers as text does seem a practical problem.

I wouldn't think an optimizer would go anywhere near the textual
representation of numbers.

>
> Not getting the row-major versus column-major working through
> all interfaces seems like a bug in the interface, or maybe I'm
> not understanding some underlying principles.

According to 7.3.2.5 in
http://www.hdfgroup.org/HDF5/doc/UG/12_Dataspaces.html
the on disk format is C-style, and calls from Fortran will silently
switch the order going out and reverse it going in. But the dump tool
will show the C format and so things will look strange if you wrote from
Fortran.

They do note that Fortran and C have different limits on how many
dimensions they can handle, and the HDF5 spec currently limits it to the
higher of these (32, for C).

> In any case, we
> probably don't need to worry about Fortran. But then R and
> C++ have different defaults (C++ arrays are row major, R's are
> column major). Mixing these up causes no end of trouble, as when
> BUGS adopted R's dump format, then put the arrays in opposite
> majority (row major vs. column major).

Even though R is implemented in C, it seems to follow Fortran
conventions for arrays. Julia has the same ordering as R. I suppose
that means they could still get mixed up, since the hdf library would
think you were using C but the data have Fortran ordering.
>
> - Bob

From http://www.hdfgroup.org/HDF5/Tutor/crtfile.html
What is an HDF5 file?

An HDF5 file is a binary file containing scientific data and supporting
metadata.

That same page explains that the textual representation of data you saw
is what you get from h5dump (e.g., Fig 4.2).

Ross

[Bob Carpenter]

If you write out 14 decimal digits plus exponent for maximum precision,
that's around 20 bytes, or about 2.5 times as many as using binary.
If the numbers have fewer digits (like 1.23 or something), then it
takes half the space to represent as ASCII.

The real problem is that the conversion is super slow.

The problem with binary formats is that they're hard to inspect
to make sure they're correct.

But as I said, if someone wants to implement an HD5 interface, it
looks like it can be made to work cross-platform and their libraries
are compatible.

Did you do any speed tests? I have to believe their libraries are written
efficiently enough to make it worthwhile given how costly the string
conversions are.

R is column-major (first index major) in its matrices (arrays).
Stan is column-major for matrices (Eigen::Matrix default), but last-index
major for arrays (nested std::vector). So we already have to do
some gymnastics reading from the R dump format (first index major
throughout) or JSON (last index major throughout).

We could also use something like this for output rather than CSV formats.
Again, it's just a matter of writing appropriate wrappers.

- Bob

[Ross Boylan]

On Wed, 2014-07-30 at 12:37 -0400, Bob Carpenter wrote:
> Did you do any speed tests?

No. Well, I used it (writing from julia and reading from R) and there
were no noticeable delays, but I had very little data. HDF seems
pretty focused on performance (e.g., they have special access modules
designed for clustered file systems), and even if they weren't it's hard
to imagine they'd be slower than writing/reading text. They also
support random access, which potentially saves having to read the file
from the beginning to get a particular piece of info.

Ross