Kite HBase Counter

[Buntu Dev]

I got Kafka with events that have attributes per URL. Some of the attributes need to be aggregated and put into HBase column.

For example:

{"url": "http://google.com", "attrs": { "cost" : 1.25, "timestamp": 111 ..... } }

{"url": "http://google.com", "attrs": { "cost" : 0.50, "timestamp": 222..... } }

{"url": "http://yahoo.com", "attrs": { "cost" : 0.75, "timestamp": 333..... } }

{"url": "http://google.com", "attrs": { "cost" : -0.25, "timestamp": 222..... } }

I would like to have a 'total_cost' column per 'url' (key). Is there anyway to achieve this using the Counter mapping type in Kite HBase dataset? 

Thanks!

[Ryan Blue]

I think the counter mapping works with Integers using HBase's atomic
increment. You could carry out the operation in cents using the it though.

rb

> --
> You received this message because you are subscribed to the Google
> Groups "CDK Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cdk-dev+u...@cloudera.org
> <mailto:cdk-dev+u...@cloudera.org>.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.


--
Ryan Blue
Software Engineer
Cloudera, Inc.

[Buntu Dev]

Makes sense thanks, will experiment with Counter mapping.

[Buntu Dev]

If I were to use Kite dataset sink via Flume, is there a way to increment the counter value or do I still need to use the increment API to handle this?

Thanks!

[Ryan Blue]

I think you'd need to use the API for this. You could also patch the
Flume sink so that it understands counters when writing to HBase.

rb

> <mailto:cdk-dev%2Bunsu...@cloudera.org>
> <mailto:cdk-dev+u...@cloudera.org
> <mailto:cdk-dev%2Bunsu...@cloudera.org>>.