Repairing bcolz carray after crash

[Brad Buran]

During a data acquisition experiment I'm streaming data to a bcolz carray. I can see 44 blocks saved to disk; however, if you look at the "meta/sizes" file, it reports the following:

{"shape": [0], "nbytes": 0, "cbytes": 1048576}

It seems that the crash is not causing the metadata to be updated properly before exiting. However, the data is clearly there since I can take the chunklen (131072) and multiply it by the number of blocks (44) and set the shape parameter to that. I can then read in the data from the file.

Are there any caveats to this approach? Is there a smarter way to do this? Do I need to worry about the "nbytes" parameter?

Thanks,

Brad

[Francesc Alted]

Hi,

Yeah, there is not any guarantee for safety after a crash.  Having said that, I think that if you know the number of chunks and the chunksize, you can derive perfectly the nbytes and the shape. So, I'd write this manually in the 'meta/sizes' and try.

Luck,

Francesc

--
You received this message because you are subscribed to the Google Groups "bcolz" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bcolz+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Francesc Alted