buffering output [was [stan-users] RStan multiple parallel instances ...]
Bob Carpenter
for the output that doesn't involve files? Network file
systems can have high latency, which would really benefit from
buffering if the output isn't needed online.
I'm thinking we could build that functionality into specialized
writer filters (basically writer classes that hold references to
other writers).
- Bob
> Begin forwarded message:
>
> From: Seth Flaxman <fla...@gmail.com>
> Subject: [stan-users] Re: RStan multiple parallel instances run in the same directory gives unserialize error
> Date: March 28, 2016 at 11:31:25 AM EDT
> To: Ben Goodrich <goodri...@gmail.com>
> Cc: Stan users mailing list <stan-...@googlegroups.com>
> Reply-To: stan-...@googlegroups.com
>
> cool, I'll give that a try.
>
> with regards to local vs network storage, is there a way to change the
> path for the rda file so I'm not writing that to an NFS? and is there
> any file system swapping I should be worried about with the draws?
> (i.e. I'm trying to reduce how much I hit the network drive.)
>
> On Mon, Mar 28, 2016 at 2:19 PM, Ben Goodrich <goodri...@gmail.com> wrote:
>> On Monday, March 28, 2016 at 8:30:21 AM UTC-4, Seth Flaxman wrote:
>>>
>>> I start up a couple instances in parallel in the same directory using
>>> Rscript:
>>>
>>> $ Rscript myscript.r param1 param2 &
>>> $ Rscript myscript.r param1 param3 &
>>>
>>> And sometimes I see the following error from one of the programs after
>>> some (but usually not all) of the chains have completed:
>>
>>
>> I think it will work if you use R rather than Rscript. Or maybe Rscript
>> --default-packages=methods ... We have had trouble with Rscript.
>>
>> Ben
>>
>
> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
Sebastian Weber
We are using Stan in a cluster with network attached storage. And YES, buffering stuff sounds great if this make IO faster in this setting. Moreover, straight binary output or at least compression would be amazing.
Sebastian
On Thursday, March 31, 2016 at 2:12:18 AM UTC+2, Bob Carpenter wrote:
> Should we be thinking about some kind of buffering mechanism
> for the output that doesn't involve files? Network file
> systems can have high latency, which would really benefit from
> buffering if the output isn't needed online.
>
> I'm thinking we could build that functionality into specialized
> writer filters (basically writer classes that hold references to
> other writers).
>
> - Bob
>
> > Begin forwarded message:
> >
> >
Bob Carpenter
buffer support. The new writer framework should make buffering very
easy to implement.
- Bob
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
Krzysztof Sakrejda
Bob Carpenter
more than one output format --- the whole point of the writer
interface is to make it easy to adapt to different output
formats.
- Bob
> On Mar 31, 2016, at 6:32 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>
> I wrote and tested a protocol buffers binary writer that's a drop in replacement for the file writer in cmdstan. The main issue with going further down that road is that we don't have an agreed on output format and I don't want to push it forward and then have to rewrite everything when we agree on a format.
>
Krzysztof Sakrejda
> Is there a proposal for an output format?
It does seem like it's about time for one.
> We can have
> more than one output format --- the whole point of the writer
> interface is to make it easy to adapt to different output
> formats.
Sure, and there will be good reason to have some different formats.
I'd like to see some consensus on a binary format that the interfaces are
on board with. It makes for less of a maintenance burden, it makes it
easier for the CmdStan people to talk to the PyStan people.
Krzysztof Sakrejda
> more than one output format --- the whole point of the writer
> interface is to make it easy to adapt to different output
> formats.
Speaking of how easy it is to make output formats: there's a (tested!) postgres writer
callback (that relies on c++11... and spawns an extra thread to stay out of Stan's way) at
https://github.com/sakrejda/stan/tree/feature/psql-writer-threaded
All it takes to initialize it is a string (the standard postgres URI string) so I'm hoping early
next week to jam it into a CmdStan fork. Between c++11, std::threads, and postgres
(libpqxx/libpq) there's no hope of having this in any of the current interfaces but I think
I can keep up a CmdStan fork.
Krzysztof
Bob Carpenter
The C++11 is the real killer for compatiblity, but that's
less of an issue with CmdStan than anything else. Especially
if compiling the relevant translation unit is made optional.
I assume the C++11 is for the thread support. Is postgres slow
enough that it needs a dedicated thread? Can we still run four
processes for four chains in parallel? If so, then we'll need
8 cores to run 4 chains if that postgres thread is really necessary.
If the postgres thread is underutilitized, it might make sense to
just remove the C++11 and threading dependency. If it's all latency
but the throughput is OK (often the case with network I/O), then it
might make sense to buffer.
- Bob
Krzysztof Sakrejda
> Cool!
>
> The C++11 is the real killer for compatiblity, but that's
> less of an issue with CmdStan than anything else. Especially
> if compiling the relevant translation unit is made optional.
>
> I assume the C++11 is for the thread support.
Yeah, once I made the decision to go that route other things probably
snuck in but threading is the main thing.
> Is postgres slow
> enough that it needs a dedicated thread? Can we still run four
> processes for four chains in parallel? If so, then we'll need
> 8 cores to run 4 chains if that postgres thread is really necessary.
> If the postgres thread is underutilitized, it might make sense to
> just remove the C++11 and threading dependency. If it's all latency
> but the throughput is OK (often the case with network I/O), then it
> might make sense to buffer.
It's the throughput. All my (informal) timing was on a dual-core laptop
(four "cores") with a solid state disk and the database on the same machine.
I wanted to avoid diving into the postgres C library and the libpqxx C++ library
has this nice layer that will do transactions wrapped in another layer that checks
on connection failures and re-tries anything that fails. There's probably
room for improvement but writing code that's both fast AND resilient to
network problems is going to be hard.
I tried some informal timings (100 iterations of 1,000 parameters and 100 iterations of
100,000 parameters) and it looks like the on-the-fly munging the writer does
along with the postgres writes takes about as much CPU time as running Stan
so it's a deal breaker (for me) if they need to share a thread but the current
design is that the writer piles work onto a queue and has a thread (or a few) that
pick up work and write it. It'll make sense to do real timing tests once I have
this thing in the context of CmdStan and some real models.
Krzysztof
Bob Carpenter
I'd think you'd want to wrap a buffered commit and make the
whole thing transactional in order to report back, but wouldn't
want to commit as you go. Or is the whole reason you're doing this
to allow saving intermediate state if something goes wrong?
- Bob
Krzysztof Sakrejda
> Yikes. Is it a very easy Stan model with lots of parameters?
Once I have it in CmdStan it'll be simpler to see how the two parts work together. I think a model drawing N iterations of K N(0,1) variables per
iteration would be a good test.
> I'd think you'd want to wrap a buffered commit and make the
> whole thing transactional in order to report back, but wouldn't
> want to commit as you go. Or is the whole reason you're doing this
> to allow saving intermediate state if something goes wrong?
I'd like to do this more or less as streaming because I want to see
the output as it's produced. At the moment one iteration (however
many parameters) is one transaction. It's actually really fast
for the main thread because the parameters are just copied
copied (as a std::vector<double> into a std::queue<std::vector<double>>
and the main thread moves on (to the next Stan iteration). The worker threads then take iterations off the queue and now that I've gotten std::chrono figured out it looks like it takes ~1 second per 1,000 parameters CPU time per worker thread. Since this is 1 insert per parameter it's about on par with what people seem to cite as the kind of performance you get form unoptimized inserts. It is a little shocking how fast the .csv writer is in comparison (0.3ms for 1,000 parameters with similar instrumentation on the operator()(std::vector<double>))
I'm thinking of this in the context of a model with ~40k parameters that
takes maybe 8+ hours (?) to run. With a single worker thread that
would be about 11 hours of time getting everything written into the DB
and no real increase to the Stan run-time---the workers just finish
processing/writing iterations after the sampler is done. With two worker
threads they stay ahead of the sampler. On a multi-core desktop with 32GB RAM it's really fine.
I tend to work from a laptop and run Stan on a server/desktop I access from the command line. The second half of the workflow is that I can access Stan output in postgres directly from my laptop with R/dplyr (https://cran.r-project.org/web/packages/dplyr/vignettes/databases.html) without worrying whether I need to move a 500MB csv file or try to read it over a network filesystem.
There's a moderate size cottage industry (and
slew of stack overflow Q&A) around optimizing writes to postgres
with a lot of good stuff I could use. The code I wrote is clearly doing some things in the least efficient way possible. Worst case I could be reducing the write time quite a bit by writing each iteration as binary to the db and having a trigger that unpacks it on the db side. Point is, there are lots of ways of optimizing this, but in the meantime it's not looking too bad.
Bob Carpenter
I would have thought the right transaction granularity (for
both semantics and streaming speed) would be a row (iteration) rather
than a parameter. Given that there's per-transaction disk overhead
plus handshake latency, this could make a big difference (that'll vary
by number of parameters).
I've never really worked directly with databases, so don't really
know what to expect from throughput. Inverted indexes for text
search are pretty fast to write by comparison (without transactional
support).
The CSV writer is already slow compared to a binary writer;
.3ms per 1K params is 3M double values/second, which is
only 24MB/second of raw binary data, which is something like an order of
of magnitude under disk throughput. It's usually the latency of writes
that's the killer, with solid states being way better than spinny disks,
which are in turn way better than most network storage options.
- Bob
Krzysztof Sakrejda
> Thanks for the analysis. That's really helpful.
>
> I would have thought the right transaction granularity (for
> both semantics and streaming speed) would be a row (iteration) rather
> than a parameter.
Yes, that is how it is currently implemented. Sorry if I gave the wrong
impression. The other major speed improvement would be to make
each insert a multi-value insert, but then the class has more complicated
SQL in it so... tests before optimization. Apparently (http://www.depesz.com/2007/07/05/how-to-insert-data-to-database-as-fast-as-possible/) multi-value inserts up to 25 or
so at a time give a really dramatic speed-up (a little over twice?)
> Given that there's per-transaction disk overhead
> plus handshake latency, this could make a big difference (that'll vary
> by number of parameters).
>
> I've never really worked directly with databases, so don't really
> know what to expect from throughput. Inverted indexes for text
> search are pretty fast to write by comparison (without transactional
> support).
>
> The CSV writer is already slow compared to a binary writer;
> .3ms per 1K params is 3M double values/second, which is
> only 24MB/second of raw binary data, which is something like an order of
> of magnitude under disk throughput. It's usually the latency of writes
> that's the killer, with solid states being way better than spinny disks,
> which are in turn way better than most network storage options.
That's a good reality check.
I'm not really sure how much the SQL writer is worth optimizing really: Even with just
one worker thread the writer does not slow down Stan and if getting the
last iterations out of memory and into the DB happens two hours after a six
hour run... I'm not sure that I care? Sure it would be nice to avoid but
the run is available for rough analysis from the database mostly as it's produced.
Bob Carpenter
Yes, buffering makes a huge difference because it removes
the extra latency per transaction.
How many chains are you running in parallel and how many cores
are you using? I'd think if there were two threads per chain
being heavily utilized, you'd need eight cores to get decent
performance.
- Bob