Updating a large scoreboard

[Behrad Zari]

Consider we have a million of users participate in a voting. storing user votes in a large HASH

Now to calculate each user's score after voting is finished, we HSCAN that HASH and then calculate each user's score and finally do

MULTI

HSET point:{userId} {voteId} {score}

ZINCRBY scores {score} {userId}

EXEC

However this takes almost 40secs (via pipelining) for 500K users. Removing MULTI/EXEC reduces to 25/30secs.

Is there any better solution/data modeling that enhances this?

for example incrementing in memory and then bulk-upload to redis?

LUA script is dangerous running on a large set of commands also.

[Tuco]

Just a question...do you really need to wait till the voting is finished to count the results, or you can keep counting when the votes are coming in.

Maybe, when the votes are coming in, add them in the hash and also add them in the sorted set.

[Behrad Zari]

Voting result is on when voting time is finished, and only then we can calculate scores. At the same time, users are rushing to check their scores :)

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/_x5HrVYuow4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Tuco]

sound like a case of distributed computing, map and reduce.

may be out of the steps which you mentioned, you could check which step is most time consuming. 

If the number of candidates in the election are less and the voters are more, i am guessing the slow part will be iterating the hash and not the adding to sorted set.

If you want to calculate the results after the voting is over and not even try to partially calculate the results beforehand, and intend to use redis, i think you could try to have multiple redis instances and kind of evenly distribute the votes, and the calculation. 

Finally when you calculate the votes separately on separate nodes, you could add them to one single sorted set.

This will distribute the calculation and only non-distributed part could be addition to the sorted set.

Also, use sorted set only if you intend to sort all the records. If you need to get only the top few items, do something which can be done in linear time.

May be i am just rambling here... :)

[Yuhe Chen]

hi, Behrad Zari

Did you use synchronous socket? Which I mean every time you send a this transaction, wait for the reply and then send another.

In that case, you may try pipelining (like a batch). Or you can try asynchronous socket. You can save a lot of time on the Round Trip Delay.

[Yuhe Chen]

You probably don't need TRANSACTION in your can and you can remove MULTI EXEC.

On Wednesday, June 7, 2017 at 7:31:30 AM UTC-4, Behrad Zari wrote: