Splittable files with LZ4 compression

[Carlo Medas]

Hello guys.

I've been working with hadoop-lzo and EB a couple years. First of all thank you all for that.

Pushed by the need of next level performances I stumbled upon LZ4.

I ended up with https://github.com/carlomedas/4mc

By manually providing decent adapters that bridge 4mc with EB, it is possible to leverage part of EB library (e.g. protobuf works perfectly), since you did design it with very good abstraction; although many classes start with Lzo* they are not really bound to that format.

I am looking forward to know if you see a possible collaboration/integration, to e.g. have EB support 4mc as alternative to lzop.

I saw some discussion where you've been chatting about supporting non-LZO files in EB, so I hope you are available to discuss it.

Let me know what you think and thanks in advance for your kind consideration.

Best regards,

Carlo Medas