HSCAN with redis and jedis has a lot of results with zero elements

[Oscar Besga]

I'm using Jedis Java library to make the iterative hscans operations

I'm experiencing an strange behaviour with HSCAN command with a custom COUNT on a hashmap on redis, with no search pattern. The hashmap has a size of 350000 elements (every element has 1Kb of size)

I'm iterating over the redis hasmap elements. In every iteration, I store the elements into a list. When I reach a limit of scanned elements (2000 elements) I stop the hscan.Then I make some calculations and delete (hdel) the elements into the list. Then I return to execute another hscan operation (initied with a new zewro cursor) and store elements in a list and etc... This cycle repeats until the hashmap is depleted

I've done tests with different values of COUNT, counting the number of hscan operations and the elements retrieved in each hscan call. That to calculate a average elements/hscan and see how many redis calls I'm doing

X = COUNT value in each test

Y = mean number of elements returned in each HSCAN iteration

Z = number of HSCAN iterations until the map is depleted

X   | Y    | Z 

----+------+------- 

25  | 4.0  | 82720 

50  | 8.0  | 41404 

100 | 17.0 | 20766

----+------+------- 

  

I'm recieving A LOT of iterations with 0 results (but cursor with non-zero value). Then, I recieve some iterations with the element number as COUNT value, no problem. The number of HSCAN operations isvery high and that's impacting severely the performance of my application. Memory and cpu usage are monitorized and I've seen no issues on this regard.

I can not find any explanation to this strange behaviour, any ideas ?

[Allan Wax]

As with any Redis SCAN, the COUNT is the maximum to return on the request.  You won't always get that many.    The scan is done when the cursor returned is "0".

See https://stackoverflow.com/questions/33842026/how-to-use-scan-commands-in-jedis

[Oscar Besga]

Hi

Sorry for the time passed, Ive been doing other tasks at my project.

But the problem remains; and I've beem investigating it.

My code does a number of scans to retrieve a bunch of data from a map (10 or more scans of ten elements each, to retrieve 100 elements) with HSCAN (with a new "0" cursor)

Then, it makes some operations with those elements and stores the results and the processed elements elsewere.

The processed elements are deleted from the redis hash with HDEL.

And we return to execute a new scan on the hash, with HSCAN and a new "0" cursor. 

This until the redis hash is empty or we reach a limit of element processed (for example 100.000 elements)

The problem is that Redis returns a lot of empty lists of results (with a non-zero cursor),

after some blocks of HSCAN operations.

So, to get a bunch of 100 elements, at the beggining of the execution I must execute 10 scans (more or less); as normal.

But later, I recieve a lot a empty lists; so to have a bunch of 100 elements, I must execute 50 scans or more.

But if I store the cursor between the bunch of data operations, and carry the last cursor over the operations-delete and apply to the next bunch of HSCAN (instead of using a new "0" cursor), 

the problem dissapears.

Also, I've tested this behaviour in Java(Jedis) Jedis and Python; the problem still appears in both languages

Example code

* Groovy https://pastebin.com/UdwSDfh6

* Python https://pastebin.com/D8hVhtst

Seriously, this problem has been driving me crazy

Anyone knows why this happens ?

(Sorry for the long message)

[Sazzadul Hoque]

There is an answer in your stackoverflow question. Did you try that?

Restart HSCAN operation with the cursor that was returned by last HSCAN operation.

--
You received this message because you are subscribed to the Google Groups "Jedis" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jedis_redis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jedis_redis/428f39e1-7030-4559-8241-44d943b4d449n%40googlegroups.com.

[Oscar Besga]

Thanks !

But that was my question on StackOverflow.

Now, I know that solution works, but why ?

I don't see that I've done something wrong with redis / jedis / my code.

(Or simply us how redis works and must accept it..)

[Sazzadul Hoque]

I know that solution works

So, Jedis seem to be working fine.

To gain further knowledge about HSCAN, I guess you can try contacting "Redis" developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/jedis_redis/72688348-9e7b-4f5e-99d7-9a8057a478f5n%40googlegroups.com.

[Oscar Besga]

Yes, as I said, the problem is also present in a python script

Jedis for me is working rigth and close to the metal,.

Thanks

[Allan Wax]

Perhaps a different point of view is needed to perform the functionality you want.  If your data is a list/queue and you are trying to get a certain number of elements from the list, process them, and delete it from the list it might be better you use the approach of using LPOP (or RPOP) instead.  Loop through the list and pop an element off the list.  Process it and move onto the next until there are no more elements or it times out.  If this is a constant processing situation then you can put this into its own thread and let it run forever (with the usual protections  for interruption and shutdown).

The above works if you never have to leave an element on the list after processing.  Basically you are processing a queue.

[Oscar Besga]

Yep, sometimes  I think that if I've used a LPOP; but I preferred HSCAN 

- have more results for one operation. *

- do not remove the data until it is processed

* yes, in version 6.x a LPOP operation can return more than one result. But we are using an older version....

** yes, I have multi and pipeline at my disposal, but when I started I didn't know those well, so...

Tks