Stability improvements on HDFS and performance tradeoff in 0.9.8.8

[Doug Judd]

With the 0.9.8.8 release we've improved Hypertable stability when running on top of HDFS.  In particular, we're now passing the SYNC_BLOCK flag to all file open calls and flushing METADATA commit log writes happens with hsync() instead of hflush().  To learn exactly what this means, please read the HDFS and Durability section of the Hypertable Administrator's Guide.

The stability improvements have come at a cost in performance (it's more expensive to call hysync() than it is to call hflush() for example).  However, we've made a number of performance improvements that should mostly offset the performance degradation introduced by the stability improvements.  If after upgrading to 0.9.8.8, you're not getting adequate performance, you can can effectively roll back the performance degrading changes by adding the following two property settings to your hypertable.cfg file and restarting:

HdfsBroker.SyncBlock=false

Hypertable.LogFlushMethod.Meta=FLUSH

- Doug