Is Sensei a good candiate for long term log storage?

[Ryan Chan]

Currently we output web server access log to Kafka and consumed by a program that forward to SAN (compressed) for long term storage.

This method is space efficient and scalable but lack of real time searching functionality (need to use Hadoop for query). I am thinking if we should forward to SenseiDB directly instead of SAN since it provide the handy search functionality.

But I am worry the following factors:

1. We output 50GB log per day, around 1M rows of record per day, our target is to store at least 2 year active data, so around 4TB, 700M rows of data, is this value huge?

2. As we might need to do house keeping (e.g. removing old data) in order to make the data fit in SenseiDB, what are the best way to handle this?

Thanks

[John Wang]

HI Ryan:

 1)   1M records per day is not much at all. I tried to index that once on my laptop, it took only a few min. To support all of 700M rows, you might need to considering sharding and run a Sensei cluster. 100M per machine is usually reasonable.

  2)  Sensei can do this automatically for you. Take a look at the zoie retention setting in the Tweets example.

-John