[rstan] feature request: dealing with compressed csv outputs in read_stan_csv

[Sebastian Weber]

Hi!

The outputs from cmd-line stan can become quite large for complex models. Hence the desire to compress these files with bzip2 (which saves 70% of the file size). But the

read_comments

call in read_stan_csv is a problem here. If I pass a list of *.csv.bz2 files to read_stan_csv, the r-function read.csv will not complain at all to read in the compressed csv file, but the read_comments function which is internal to rstan does not like the compressed file. So if the read_comments file could deal with compressed file streams that would solve the problem I think and then allow me to process compressed csv files which is a lot faster as 70% less of data needs to be transfer over our network (stuff is stored on NAS).

Thanks for considering this feature, which would help to work many of us to work more efficiently.

Best regards,

Sebastian

[Bob Carpenter]

That makes a lot of sense. I created an issue for CmdStan
to create compressed outputs from sampling and for bin/print
to read them:

https://github.com/stan-dev/stan/issues/519

and one for RStan to deal with reading:

https://github.com/stan-dev/rstan/issues/38

Looks like Boost has a bzip2 lib, which I linked in issue 519.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Marcus Brubaker]

Compression of some form is a good option to add, but in general our file sizes are going to be overly large so long as we're outputting text.  A double precision floating point number takes around 20 bytes to represent in a CSV (counting spaces, commas, decimals, signs, etc).  If we just dumped a binary format it would only take 8 bytes and be loads faster to dump and read.  Plus, compression could be added on top of that.

Cheers,

Marcus

To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "stan users mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.

[Tamas Papp]

But binary files are opaque, subject to bit rot, whereas CSV has been
and will be around for a long time. Efficient ways to compress exist (eg
bz2 is quite good for CSV, but even gz is great), and have
well-maintained open-source compression and decompression utilities.

I usually compress my CSV files not because of space requirements, but
because of quicker load times. Usually the bottleneck is disk read
speed, not the CPU.

Best,

Tamas

>>> email to stan-users+...@googlegroups.com.

>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "stan users mailing list" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to stan-users+...@googlegroups.com.

[Marcus Brubaker]

Not suggesting getting rid of the CSV file option.  In an (my) ideal world, Stan would:

- provide I/O with binary files as an option (along side CSV).  As a rule they would be much faster and smaller, but could also store extra stuff that is awkward to put into a CSV file (e.g., adaptation state and parameters to allow for restarting runs)

- provide conversion utilities from binary to CSV or other ASCII formats which would allow users to, for instance, extract specific parameters, do sub-sampling, etc

Cheers,

Marcus

[Brian Lau]

Binary outputs as an option would be great. Their opaqueness could be reduced by using a standard data format like HDF, which also allows for compression.

-b

[Sean O'Riordain]

If a binary format was to taken up and I am not really convinced, I would definitely be in favour of a standard format like HDF.  However, I would prefer csv compressed with either gz / bz2, as these can easily be processed using standard tools outside for pre/post processing.  It should be easier to implement too as it is just a layer on top of what is there instead of a completely new format.

Kind regards,

Sean

Kind regards,

Sean O'Riordain

seor...@tcd.ie

[Sebastian Weber]

A binary format is definitely a very sensible option too, but I guess it is a lot more effort than simply compressing the streams by filtering them through some compression adaptors from boost.

HDF is a very good choice, also sqlite is great (anything standard), but both options require some coding effort I guess.

And yes, I also use to compress stuff because it is quicker to load and since a couple of R-versions one even does not need to bother around with uncompressing them when using these files in R.

I am glad that so many people agree on this.

Best,
Sebastian

Sebastian Weber

[Michael Betancourt]

Realistically the core team probably won’t be able to get to making a choice of encoding
or compression scheme anytime soon, let alone implement the interface, but this sounds
like a nice relatively-isolated project for anyone who wanted to contribute any code.

Wink, wink. ;-)