concurrent hdf5 access
15 views
Skip to first unread message
Kyle Foreman
Jul 25, 2010, 1:29:07 PM7/25/10
to labeled-array
Since h5py only supports the serial build of HDF5, should I be worried
about concurrent reads of a file archived via larry? Or would
concurrent writes be the only thing to watch out for?
about concurrent reads of a file archived via larry? Or would
concurrent writes be the only thing to watch out for?
Keith Goodman
Jul 25, 2010, 1:36:29 PM7/25/10
to labele...@googlegroups.com
My guess is the same as yours: concurrent reads are safe. But I'm just
guessing. You might want to ask on the h5py list to make sure.
Kyle Foreman
Jul 31, 2010, 4:23:24 AM7/31/10
to labeled-array
I haven't run anything massively parallel yet, but it seems that about
10 concurrent reads works just fine. I opted to just code around the
need for concurrent writes on the assumption it'd cause problems.
Thanks!
On Jul 25, 10:36 am, Keith Goodman <kwgood...@gmail.com> wrote:
10 concurrent reads works just fine. I opted to just code around the
need for concurrent writes on the assumption it'd cause problems.
Thanks!
On Jul 25, 10:36 am, Keith Goodman <kwgood...@gmail.com> wrote:
Keith Goodman
Jul 31, 2010, 9:49:00 AM7/31/10
to labele...@googlegroups.com
On Sat, Jul 31, 2010 at 1:23 AM, Kyle Foreman <kylef...@gmail.com> wrote:
> I haven't run anything massively parallel yet, but it seems that about
> 10 concurrent reads works just fine. I opted to just code around the
> need for concurrent writes on the assumption it'd cause problems.
> Thanks!
> I haven't run anything massively parallel yet, but it seems that about
> 10 concurrent reads works just fine. I opted to just code around the
> need for concurrent writes on the assumption it'd cause problems.
> Thanks!
It's good to know that it doesn't crash.
Are you reading in parallel for speed? Or do you have multiple
programs and one data archive?
Kyle Foreman
Aug 1, 2010, 8:41:27 AM8/1/10
to labeled-array
I'm running local regression across 3 dimensions, which is pretty
slow. In Stata it took about 40 hours in serial to do the whole
dataset, so I parallelized it on our cluster and got it down to about
an hour. I'm now trying to turn it into a web application and am
parallelizing from the start.
On Jul 31, 6:49 am, Keith Goodman <kwgood...@gmail.com> wrote:
slow. In Stata it took about 40 hours in serial to do the whole
dataset, so I parallelized it on our cluster and got it down to about
an hour. I'm now trying to turn it into a web application and am
parallelizing from the start.
On Jul 31, 6:49 am, Keith Goodman <kwgood...@gmail.com> wrote:
Reply all
Reply to author
Forward
0 new messages