get set bits index on redis bitmap

[Ivan Etchart]

I'm wondering if there is a way to get the index of the set bits on redis. You can get/set an individual bit, and event count it but there is not way to retrieve all the set bits (at least, not using redis commands).

So my solution was iterate over all the possible index and ask if it is or not set. This lead to a huge amount of getbits requests.

Pipelining those request is fast, but I need to know beforehand which indexes I have to get, and that is what I want to avoid.

I would really appreciate any comment.

Regards

[Josiah Carlson]

If you are calling 'getbit' for all bits in a bitmap, you would be better off performing a single "get" and doing the scan on the client side.

It would be significantly faster than what you are doing, use less bandwidth, etc. You could also write your scan using Lua scripting, though I would suggest that you use 'bitcount' to determine if a block has any bits set, and if so, then scan a block. If you size your blocks properly based on the actual density of bits set in your data, then you may save significant time in scanning blocks on the Lua side. I would also suggest that you not scan the whole bitmap at once, and instead provide a "start" position so you can scan your data incrementally, receiving say 1000 bit set positions at a time.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Chris Curtin]

If you do pull the value to the client, be aware of the 'endian' issues you may encounter.

In Java: 

https://github.com/xetorthio/jedis/issues/301

Example and workaround

https://gist.github.com/gmuller/2933940

I'm sure other languages may have the same problem.

Thanks,

Chris

[Ivan Etchart]

Yes, get the position (index in the map) of the bit set as 1. For example 0001001, and get [3, 6]. 

I really like the approach of processing batches and using bitcount given the densitiy of the bits set, this can avoid some processing. About using less bandwith, you say this can be done using the lua script, right?

Thank for your answer!!