MEXISTS

[Joran Greef]

I need to test for the existence of multiple keys, and calling EXISTS
multiple times (by pipelining or over multiple existing connections
simultaneously) is slower than doing a single MGET. Would an MEXISTS
command be faster than an MGET? Would this be something that could be
added as a command?

Cheers

Joran Greef

[Josiah Carlson]

How much slower is EXISTS over a pipeline vs. an MGET call? What
client library are you using?

Have you been using MULTI/EXEC pipelines, or just plain client-side
pipelines (the latter doesn't use the MULTI/EXEC pair, it merely
batches up all of the commands and sends them with a single
round-trip).

- Josiah

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[Joran Greef]

> How much slower is EXISTS over a pipeline vs. an MGET call?

Thanks Josiah, it's about 3ms to check 10 keys using EXISTS vs < 1ms
for MGET. The values returned by MGET are around 250 bytes.

> What client library are you using?

My own client which does a ZRANGE for a 10000 key sorted set in about
60ms, and MGET for 10000 keys with 4 kilobyte values in about 570ms.

> Have you been using MULTI/EXEC pipelines, or just plain client-side
> pipelines (the latter doesn't use the MULTI/EXEC pair, it merely
> batches up all of the commands and sends them with a single
> round-trip).

Pipelining without MULTI/EXEC, sending 10 commands in one go using
Node.js, and collecting the responses as they come in. I tried
multiplexing over 10 open connections, 1 connection etc.

I think it makes sense that MGET is faster, since it's a single
command going in, and not much coming back. I think an MEXISTS would
be faster still, without the network overhead.

Cheers

Joran Greef

[Josiah Carlson]

For a matter of comparison, this is the performance I see inside a
Linux VM using Redis 2.0.2 and a modified version of the redis-python
client (which I've benchmarked to be twice as fast for bulk
operations).

>>> len(k)
>>> def f1():
... s1 = stats.Stats()
... for i in xrange(100):
... t = time.time()
... x = rc.mget(10*k)
... s1.add(time.time()-t)
...
... s2 = stats.Stats()
... for i in xrange(100):
... p = rc.pipeline(False)
... for kk in 10*k:
... p = p.exists(kk)
... t = time.time()
... c = p.execute()
... s2.add(time.time()-t)
...
... s3 = stats.Stats()
... for i in xrange(100):
... t = time.time()
... p = rc.pipeline(False)
... for kk in 10*k:
... p = p.exists(kk)
... c = p.execute()
... s3.add(time.time()-t)
...
... print s1
... print s2
... print s3
...
>>> f1()
<min:0.13 max:0.37 avg:0.15 std:0.03 gavg:0.15 gstd:0.34 n:100>
<min:0.12 max:0.22 avg:0.14 std:0.02 gavg:0.14 gstd:0.31 n:100>
<min:0.21 max:0.31 avg:0.24 std:0.03 gavg:0.23 gstd:0.26 n:100>

We get 10,000 keys (each of which are duplicated 10x), 100 times for
each of the different versions. If you don't count the time it takes
to set up the pipelined call in Python, the pipeline is a little bit
faster. If you do count the time it takes to set up the pipeline,
then it's about half the speed. I suspect that your client, like
mine, spends a lot of time setting up the state for pipelining
everything out (and really, just making all of the calls). If you
write your own command for bulk requests, that just manipulate the
pipeline's internal state in a single call, I suspect that you could
get the pipeline to perform about the same speed, if not a little
faster than the mget call.

... which gives me some ideas. :)

- Josiah

[Josiah Carlson]

Sorry, len(k) is 1000, and we get each key in k, ten times. Sorry if
there was any confusion.

- Josiah

[Joran Greef]

Thanks for the stats Josiah.

> If you write your own command for bulk requests, that just manipulate the
> pipeline's internal state in a single call, I suspect that you could
> get the pipeline to perform about the same speed, if not a little
> faster than the mget call.
> ... which gives me some ideas. :)

Good idea.