ZSCAN: fold over ZSET member's scores

[Matteo Moci]

Hi All, 

Thanks for this great project! 

Basically, I want to do a simple sum of all the scores of a sorted set members. 

I read in the docs I can do a ZSCAN [1], keeping the status of the members I've seen and adding only if it's not a duplicate. 

Are there other possibilities? 

I'll dig here [2], maybe bitmaps can help here. 

Thanks a lot! 

Matteo

Some more details. 

My dataset looks like the following: 

ZSET-ONE

a 12

b 34

ZSET-TWO

b 56

c 78

then, based on the previous two zsets, I'd like to update another zset like this: 

ZSET-123-456

ONE   46  (=12+34)

TWO 134  (=56+78)

[1] http://redis.io/commands/zscan

[2] http://redis.io/commands/bitcount

--

Matteo Moci
http://mox.fm

[Itamar Haber]

You can use a Lua script to do your bidding - classic use case

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Alexey Shuksto]

No, he won't be able to use any 'write' Redis command after ZSCAN invocation, as all SCAN-family commands considered 'non-deterministic' by Redis LUA-scripting laws (same as RANDOMKEY, TIME etc.).

This is quite sad, actually. :(

Simple test:

127.0.0.1:6379> eval "redis.call('ZSCAN', KEYS[1], 0); redis.call('ZADD', KEYS[1], '1', '1');" 1 test-zset

(error) ERR Error running script (call to f_14e6c9530282a6b1d19252ab271bcaf843b942b2): @user_script:1: @user_script: 1: Write commands not allowed after non deterministic commands

четверг, 25 июня 2015 г., 20:46:55 UTC+3 пользователь Itamar Haber написал:

[Itamar Haber]

Very true - `ZSCAN` won't mix with a Lua that writes. OTOH, you can use something else like `ZRANGE key 0 -1 WITHSCORES` in the script instead of it.

Alternatively, if you absolutely have to use `ZSCAN` in your script (one reason to do so is an obscenely large Sorted Set), you can have it return the sum to the client and have the client do the write operations.

--

Itamar Haber | Chief Developers Advocate
Redis Watch Newsletter - Curator and Janitor
Redis Labs - Enterprise-Class Redis for Developers

Mobile: +1 (415) 688 2443
Mobile (IL): +972 (54) 567 9692
Email: ita...@redislabs.com
Skype: itamar.haber

Blog  |  Twitter  |  LinkedIn

[The Baldguy]

On Thursday, June 25, 2015 at 12:06:42 PM UTC-5, Matteo Moci wrote:

Hi All, 

Thanks for this great project! 

Basically, I want to do a simple sum of all the scores of a sorted set members. 

I read in the docs I can do a ZSCAN [1], keeping the status of the members I've seen and adding only if it's not a duplicate. 

Are there other possibilities? 

I'll dig here [2], maybe bitmaps can help here. 

Greetings,

ZSCAN combined with local set membership (in the client)  and updates to the server to be the best route here for it as described, but there are alternatives which may be better depending on your code.

If the sorted set is large, or might become large,  and especially if you have many connections to redis, using ZSCAN will allow other operations to continue during the non-data gathering and non-data storing sections - the summing specifically. If you were to do it in a Lua script you will decrease the capability of other connections to execute in a timely fashion.

For this route I would go with the client calling ZSCAN, checking to see if it has processed that member, then calling an INCRBY on your destination key (ie. ZSET-ONE would INCRBY ONE <member-score>" in your example). This will keep the logic and client side code down while not reducing the concurrency of the system by invoking a potentially "long running" Lua script.  If your "ONE" key needs to be reset each time you run this, simply delete it in the beginning, and it will be effectively "0" and get created with the first INCRBY call.

However, you could eliminate this sequence entirely by having whatever is adding the data to ZSET-ONE update the ONE key at the same time. Redis is fast enough this is much more efficient than the above process. For example if you are using ZINCRBY to store the data, simply INCRBY the secondary key on the next line with the same value. This eliminates the need for the scan/sum/set sequence entirely.

If, however, you populate the sorted set with ZADD instead you could pull the score, diff the old value with the new, and then call ZINCRBY and you're back the the above method. The code complexity for this is still less than the complexity of Lua, or ZSCAN iteration, summation, and result storing. It also has the code benefit of having relevant actions visible to the developer and mentally tying the two (or more) keys together.

In both of these alternatives you are amortizing the cost of the scan/sum/set over time rather than at once.  In either case you get the added benefit of your 'ONE' and 'TWO' keys being constantly current, and not needing the extra code/job to update data.