krati with hadoop hdfs backend

[qmx]

Hi!

What do you all think about it? Does it sound feasible?

Cheers!

-- qmx

[Sam Tingleff]

What are you trying to do?

[Douglas Campos]

Well, my boss asked me about it, then I started looking if it was possible to have the same cool api, having hdfs as the backend (we're on 3TB/day data volume here, and they want to mine the data later)

The main reason for not going straight to hbase / cassandra is that all the clustering / columns stuff adds significant overhead, and krati gets close to our desired throughput rate.

Am I crazy? :)

[Sam Tingleff]

It sounds crazy, yes :) HDFS has very different behavior than local disk. It is designed for reliability and throughput more than random seek performance.

There is ElephantDB [1] and I have been working on a thrift-based service to serve read-only exports from HDFS using Krati as local storage. In general these kind of systems (also see Voldemort) typically export out of HDFS to expose as a real time service.

[1] https://github.com/nathanmarz/elephantdb

[2] https://github.com/samtingleff/valkyriedb