Hypertable and HDFS

[esvee]

How does Hypertable talk to HDFS? Does it use libhdfs?

Thanks!

[Doug Judd]

Hypertable communicates with HFDS via the HDFSBroker process.  The HDFSBroker application is written in Java and uses Hadoop's Java FileSystem API (e.g. not libhdfs).

- Doug

[esvee]

Thanks Doug for the quick reply. How is the performance of the Broker
as compared to libhdfs since we know libhdfs (uses JNI) is slow.

Thanks!

On Apr 3, 10:57 am, "Doug Judd" <d...@zvents.com> wrote:
> Hypertable communicates with HFDS via the HDFSBroker process. The
> HDFSBroker application is written in Java and uses Hadoop's Java FileSystem
> API (e.g. not libhdfs).
>
> - Doug
>

[Doug Judd]

I'm not sure how the performance would compare.  The one problem I see with libhdfs is that it provides only a synchronous Posix file interface.  The nice thing about Hypertable's brokered DFS architecture that it provides an asynchronous interface as well.  Hypertable is designed to use asynchronous communication throughout and is how the system achieves high performance.  You could probably build an asynchronous interface on top of libhdfs, but it would take some work.  Also, it maintains a clean separation between Hypertable and the underlying DFS.  If you have any interest in experimenting with implementing a directly linked libhdfs version, feel free.  I can get you pointed in the right direction and help with building and testing it for comparison purposes.  It would be interesting to see if there is a significant performance advantage.  If so, we might have to rethink the broker architecture.

- Doug