Using HBaseHUT with counters

[Ionut I.]

Hi Alex,

I didn't find something in HUT documentation about using it for
table.incrementColumnValue operation.
In my opinion this is not possible since this operation is atomic, but
I decide to ask you to confirm this.

Regards,
Ionut I.

[Alex Baranau]

Hi!

Not sure I follow the question, sorry. Could you give an example of how/where you'd use table.incrementColumnValue in your case?

Thanx,

Alex

[Ionut I.]

Hi again,

I'm working on a project that has to keep a large number of
counters(~2-3M). Big part of them are updating frequently (2-3 times/
minut).
My intention is to reduce the number of call to HBase and behind in-
memory aggregation, I'm looking for other solution to reduce the
throughput.
Is this a valid usecase for your project?

[Alex Baranau]

Aha. I guess "Big part of them" is really big, so that simply using HBase counters directly is not fast enough.

The answer would be yes, HBaseHUT is meant for such situations as well. Though it is usually used with more complex calculations than just counters.

Currently with HBaseHUT there're no calculations (agg functions and such) out-of-the-box, so the merge code (processing updates) have to be written by client code. In you situation this means:

1) while processing input data, you write records (with the help of HutPut.adjustRowKey() as shown on wiki) which contain values like: column->inc_delta. I.e. by which number to increase values stored in specific column

2) while reading data or during periodic compactions you have to provide UpdateProcessor implementation, which for a given iterable list of records merges deltas and calculates the final value

Please let me know if you have questions for any of these parts.

Btw, such UpdateProcessor implementation looks quite generic for me, we could include it in HBaseHUT lib. So don't hesitate to create a pull request in github once you implement it ;)

Alex

[Alex Baranau]

jfyi: committed  example implementation for this generic UpdatesProcessor [1]. Note: it is just an example, not the most optimal one (see also TODOs in comments), you may want to adjust it for your needs.

Hope this helps!

Alex

[1]

https://github.com/sematext/HBaseHUT/blob/master/src/main/java/com/sematext/hbase/hut/up/SumColumnsUp.java