Pop from sorted set by score

[Serega Sheypak]

Hi, is there any possibility to pull elements from sorted set by score.

For example, my sorted set consists of:

[{1:a}, {2:b}, {3:c}, {5:d}, {7:d}]

I do "pull" using score: (2,5)

and set changes to:

[{1:a}, {2:b},  {5:d}, {7:d}]

and returned element is: {3:c}

Is there any possibility to implement such functionality using other  Redis data structures?

Also it's ok to pull elements from sorted head or tail.

I have only one solution in my mind:

There are X concurrent sorted-set readers

1. x1 sends zrangebyscore 

2. x1 iterates over elements and deletes them. If delete operation == 1, then x1 gets element exclusively

3. Repeat 1..2 changing zrangebyscore range to get enough element.

Looks ugly but could work?

[Josiah Carlson]

Replies inline.

On Thu, May 14, 2015 at 5:30 AM, Serega Sheypak <serega....@gmail.com> wrote:

Hi, is there any possibility to pull elements from sorted set by score.

For example, my sorted set consists of:

[{1:a}, {2:b}, {3:c}, {5:d}, {7:d}]

I do "pull" using score: (2,5)

and set changes to:

[{1:a}, {2:b},  {5:d}, {7:d}]

and returned element is: {3:c}

Is there any possibility to implement such functionality using other  Redis data structures?

You can, but it won't offer the same performance.

Also it's ok to pull elements from sorted head or tail.

This sentence is missing context.

I have only one solution in my mind:

There are X concurrent sorted-set readers

1. x1 sends zrangebyscore 

2. x1 iterates over elements and deletes them. If delete operation == 1, then x1 gets element exclusively

3. Repeat 1..2 changing zrangebyscore range to get enough element.

Looks ugly but could work?

You haven't said what problem you are trying to solve, why you are trying to solve it, or anything else. I have no idea whether this does what you want it to, because there is at least a paragraph or two of missing context of problem and purpose.

That said, generally if you want to add functionality like "scan for and optionally remove elements from a ZSET, returning the results to the caller", you should use a Lua script. You can embed all of your scan/delete/return logic and run it in Redis so you don't have to worry about data race conditions.

 - Josiah

-- 

You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Serega Sheypak]

Hi, thanks for the reply.

The problem I'm trying to solve:

I have events, each event has id. 

I put event to sorted set, where value is id and score = timestamp.

The idea is to keep unique events id sorted by timestamp.

I have application which wakes up each minute and gets events from sorted set by score range.

The problem is that I can't run several instances of that application.

I want to pull by score range so each application could get values from set.

>Also it's ok to pull elements from sorted head or tail.

I mean that I can redesign my algorithm and pull elements from head or tail of sorted set if it's hard to do pull by range.

Right now I do get elements from sorted list by custom range.

> "scan for and optionally remove elements from a ZSET, returning the results to the caller", you should use a Lua script.

Set should be sorted by score.

And I need to pull it tail/head or by score range. 

From business point, it's triggers queue. Each trigger has id. Score = execution timestamp. I scan sorted set using score range to get triggers that could be executed (their score is in range).

четверг, 14 мая 2015 г., 16:32:58 UTC+3 пользователь Serega Sheypak написал:

[Josiah Carlson]

Replies inline.

On Fri, May 15, 2015 at 2:18 AM, Serega Sheypak <serega....@gmail.com> wrote:

Hi, thanks for the reply.

The problem I'm trying to solve:

I have events, each event has id. 

I put event to sorted set, where value is id and score = timestamp.

The idea is to keep unique events id sorted by timestamp.

I have application which wakes up each minute and gets events from sorted set by score range.

The problem is that I can't run several instances of that application.

I want to pull by score range so each application could get values from set.

>Also it's ok to pull elements from sorted head or tail.

I mean that I can redesign my algorithm and pull elements from head or tail of sorted set if it's hard to do pull by range.

Right now I do get elements from sorted list by custom range.

There might be a fraction of a percent improvement in performance if you are pulling from the ends, but that isn't worth worrying about or changing your application logic (1000x more overhead is experienced in network overhead).

> "scan for and optionally remove elements from a ZSET, returning the results to the caller", you should use a Lua script.

Set should be sorted by score.

And I need to pull it tail/head or by score range. 

From business point, it's triggers queue. Each trigger has id. Score = execution timestamp. I scan sorted set using score range to get triggers that could be executed (their score is in range).

Great. Use a Lua script. You can pull any range you want. The start, the end, the middle. You can pull by score or by index.

 - Josiah

четверг, 14 мая 2015 г., 16:32:58 UTC+3 пользователь Serega Sheypak написал:

Hi, is there any possibility to pull elements from sorted set by score.

For example, my sorted set consists of:

[{1:a}, {2:b}, {3:c}, {5:d}, {7:d}]

I do "pull" using score: (2,5)

and set changes to:

[{1:a}, {2:b},  {5:d}, {7:d}]

and returned element is: {3:c}

Is there any possibility to implement such functionality using other  Redis data structures?

Also it's ok to pull elements from sorted head or tail.

I have only one solution in my mind:

There are X concurrent sorted-set readers

1. x1 sends zrangebyscore 

2. x1 iterates over elements and deletes them. If delete operation == 1, then x1 gets element exclusively

3. Repeat 1..2 changing zrangebyscore range to get enough element.

Looks ugly but could work?

[Serega Sheypak]

Thank you, will try!

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/D5AwILCV8SA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.

[Serega Sheypak]

>Great. Use a Lua script. You can pull any range you want. The start, the end, the middle. You can pull by score or by index.

I can't find api to pull from sorted set by score or index.

I have to write lua script, that:

1. zrangebyscore using score range I need

2. zrem elements from zrangebyscore result

3. return zrangebyscore result for processing.

And it guarantees me that each sorted list consumer using lua script would get unique result, correct?

понедельник, 18 мая 2015 г., 22:21:03 UTC+3 пользователь Josiah Carlson написал:

[Serega Sheypak]

I did it this way:

local triggers = redis.call('ZRANGEBYSCORE', '$queueName' , '$leftBorderTs', '$rightBorderTs','WITHSCORES', 'LIMIT', '$fromLimit', '$toLimit')

local fromTs = 0

if table.getn(triggers) > 1 then

  fromTs = triggers[2]

end

local toTs = table.getn(triggers)

redis.call('ZREMRANGEBYSCORE', '$queueName' ,fromTs, toTs)

return triggers

is it OK?

вторник, 19 мая 2015 г., 12:52:45 UTC+3 пользователь Serega Sheypak написал:

[Josiah Carlson]

On Tue, May 19, 2015 at 2:52 AM, Serega Sheypak <serega....@gmail.com> wrote:

>Great. Use a Lua script. You can pull any range you want. The start, the end, the middle. You can pull by score or by index.

I can't find api to pull from sorted set by score or index.

ZRANGEBYSCORE

ZREVRANGEBYSCORE

ZRANGEBYRANK

ZREVRANGEBYRANK 

I have to write lua script, that:

1. zrangebyscore using score range I need

2. zrem elements from zrangebyscore result

3. return zrangebyscore result for processing.

And it guarantees me that each sorted list consumer using lua script would get unique result, correct?

Lua scripts in Redis execute atomically. Assuming your Lua script is written correctly, yes, everyone should get unique results.

 - Josiah

[Josiah Carlson]

Rather than generate your Lua script with every call, write your Lua script to accept the standard KEYS and ARGV arguments to ensure that you don't waste memory in Redis keeping a copy of all of your script variants. But other than that, it looks like you have a script that does what you want. :)

 - Josiah

[Serega Sheypak]

Thanks!!!