For some time I am trying to use Redis in a high volume system, but till now I am not happy about the performance of Redis. I wonder what the issue is, am I just pushing Redis to it's limits or am I doing something wrong?
I am using Redis to store a great amount of counters (of which some have an expiry timestamp). However, after some time Redis starts to block new connections for short periods of time and the CPU usage becomes 100%. The overall performance is degraded at those times.
The following things were already improved/tried:
- Improved the client code
- Enabled PIPELINE
- Spreading the keys among 4 instances instead of just one instance
- Disabled snapshotting to the RDB file
- Updated Redis from 2.4 to 2.6 RC.
- Enabled the Redis Watchdog (see logs below)
* Some key numbers:
Redis instances: 4
Memory usage per instance: ~ 3GB
Amount of keys per instance: db0:keys=18644443,expires=9137407
Amount of set operations: 1200 - 1500 per second
Try not to create a new TCP connection each time. My guess is that your
problem is related to that. Unfortunately, I don't know how to use a
connection pool (aka persistent connections) for Redis in PHP (if at all
possible). To check if this is the problem, perhaps you could try to
connect using a local unix socket instead of TCP and see if the problem
goes away?
> For some time I am trying to use Redis in a high volume system, but
> till now I am not happy about the performance of Redis. I wonder what
> the issue is, am I just pushing Redis to it's limits or am I doing
> something wrong?
> I am using Redis to store a great amount of counters (of which some
> have an expiry timestamp). However, after some time Redis starts to
> block new connections for short periods of time and the CPU usage
> becomes 100%. The overall performance is degraded at those times.
> The following things were already improved/tried:
> - Improved the client code
> - Enabled PIPELINE
> - Spreading the keys among 4 instances instead of just one instance
> - Disabled snapshotting to the RDB file
> - Updated Redis from 2.4 to 2.6 RC.
> - Enabled the Redis Watchdog (see logs below)
> * Some key numbers:
> Redis instances: 4
> Memory usage per instance: ~ 3GB
> Amount of keys per instance: db0:keys=18644443,expires=9137407
> Amount of set operations: 1200 - 1500 per second
> Try not to create a new TCP connection each time. My guess is that your
> problem is related to that. Unfortunately, I don't know how to use a
> connection pool (aka persistent connections) for Redis in PHP (if at all
> possible). To check if this is the problem, perhaps you could try to
> connect using a local unix socket instead of TCP and see if the problem
> goes away?
> BR,
> Hampus
> On 05/21/12 19:55, Melvin Rook wrote:
> > Hi all,
> > For some time I am trying to use Redis in a high volume system, but
> > till now I am not happy about the performance of Redis. I wonder what
> > the issue is, am I just pushing Redis to it's limits or am I doing
> > something wrong?
> > I am using Redis to store a great amount of counters (of which some
> > have an expiry timestamp). However, after some time Redis starts to
> > block new connections for short periods of time and the CPU usage
> > becomes 100%. The overall performance is degraded at those times.
> > The following things were already improved/tried:
> > - Improved the client code
> > - Enabled PIPELINE
> > - Spreading the keys among 4 instances instead of just one instance
> > - Disabled snapshotting to the RDB file
> > - Updated Redis from 2.4 to 2.6 RC.
> > - Enabled the Redis Watchdog (see logs below)
> > * Some key numbers:
> > Redis instances: 4
> > Memory usage per instance: ~ 3GB
> > Amount of keys per instance: db0:keys=18644443,expires=9137407
> > Amount of set operations: 1200 - 1500 per second
Your reporting information mentions sets. How many sets are you
passing to your operations, what operations are you performing, and
how large are your sets?
On Mon, May 21, 2012 at 11:55 AM, Marc Byrd <dr.marc.b...@gmail.com> wrote:
> You mentioned going from 1 instance to 4 - but these are all on the same
> hardware?
> What does # ulimit -n return? If it's the default 1024, that might be your
> problem.
> m
> On Mon, May 21, 2012 at 11:52 AM, Hampus Wessman <ham...@hampuswessman.se>
> wrote:
>> Try not to create a new TCP connection each time. My guess is that your
>> problem is related to that. Unfortunately, I don't know how to use a
>> connection pool (aka persistent connections) for Redis in PHP (if at all
>> possible). To check if this is the problem, perhaps you could try to
>> connect using a local unix socket instead of TCP and see if the problem
>> goes away?
>> BR,
>> Hampus
>> On 05/21/12 19:55, Melvin Rook wrote:
>> > Hi all,
>> > For some time I am trying to use Redis in a high volume system, but
>> > till now I am not happy about the performance of Redis. I wonder what
>> > the issue is, am I just pushing Redis to it's limits or am I doing
>> > something wrong?
>> > I am using Redis to store a great amount of counters (of which some
>> > have an expiry timestamp). However, after some time Redis starts to
>> > block new connections for short periods of time and the CPU usage
>> > becomes 100%. The overall performance is degraded at those times.
>> > The following things were already improved/tried:
>> > - Improved the client code
>> > - Enabled PIPELINE
>> > - Spreading the keys among 4 instances instead of just one instance
>> > - Disabled snapshotting to the RDB file
>> > - Updated Redis from 2.4 to 2.6 RC.
>> > - Enabled the Redis Watchdog (see logs below)
>> > * Some key numbers:
>> > Redis instances: 4
>> > Memory usage per instance: ~ 3GB
>> > Amount of keys per instance: db0:keys=18644443,expires=9137407
>> > Amount of set operations: 1200 - 1500 per second
I don't know the client library, is it phpredis? looks like not only you
are not using persistent connections, you are closing the socket on each
request, that's really bad.
this gives a hint:
total_connections_received:**119569
total_commands_processed:**1047293
around 1:9 connection/commands ratio. I counted and your code has 9
commands. :)
so what you have here is tons of TCP sockets probably in time wait state,
blocking redis or the kernel or whatever.
anyway, phpredis has the method *pconnect*() instead of connect, that tries
to pool connections.
use it and not connect() and don't close your connection in the end.
On Mon, May 21, 2012 at 9:52 PM, Hampus Wessman <ham...@hampuswessman.se>wrote:
> Try not to create a new TCP connection each time. My guess is that your
> problem is related to that. Unfortunately, I don't know how to use a
> connection pool (aka persistent connections) for Redis in PHP (if at all
> possible). To check if this is the problem, perhaps you could try to
> connect using a local unix socket instead of TCP and see if the problem
> goes away?
> BR,
> Hampus
> On 05/21/12 19:55, Melvin Rook wrote:
> > Hi all,
> > For some time I am trying to use Redis in a high volume system, but
> > till now I am not happy about the performance of Redis. I wonder what
> > the issue is, am I just pushing Redis to it's limits or am I doing
> > something wrong?
> > I am using Redis to store a great amount of counters (of which some
> > have an expiry timestamp). However, after some time Redis starts to
> > block new connections for short periods of time and the CPU usage
> > becomes 100%. The overall performance is degraded at those times.
> > The following things were already improved/tried:
> > - Improved the client code
> > - Enabled PIPELINE
> > - Spreading the keys among 4 instances instead of just one instance
> > - Disabled snapshotting to the RDB file
> > - Updated Redis from 2.4 to 2.6 RC.
> > - Enabled the Redis Watchdog (see logs below)
> > * Some key numbers:
> > Redis instances: 4
> > Memory usage per instance: ~ 3GB
> > Amount of keys per instance: db0:keys=18644443,expires=9137407
> > Amount of set operations: 1200 - 1500 per second
> You mentioned going from 1 instance to 4 - but these are all on the > same hardware?
> What does # ulimit -n return? If it's the default 1024, that might be > your problem.
> m
> On Mon, May 21, 2012 at 11:52 AM, Hampus Wessman > <ham...@hampuswessman.se <mailto:ham...@hampuswessman.se>> wrote:
> Try not to create a new TCP connection each time. My guess is that
> your
> problem is related to that. Unfortunately, I don't know how to use a
> connection pool (aka persistent connections) for Redis in PHP (if
> at all
> possible). To check if this is the problem, perhaps you could try to
> connect using a local unix socket instead of TCP and see if the
> problem
> goes away?
> BR,
> Hampus
> On 05/21/12 19:55, Melvin Rook wrote:
> > Hi all,
> > For some time I am trying to use Redis in a high volume system, but
> > till now I am not happy about the performance of Redis. I wonder
> what
> > the issue is, am I just pushing Redis to it's limits or am I doing
> > something wrong?
> > I am using Redis to store a great amount of counters (of which some
> > have an expiry timestamp). However, after some time Redis starts to
> > block new connections for short periods of time and the CPU usage
> > becomes 100%. The overall performance is degraded at those times.
> > The following things were already improved/tried:
> > - Improved the client code
> > - Enabled PIPELINE
> > - Spreading the keys among 4 instances instead of just one instance
> > - Disabled snapshotting to the RDB file
> > - Updated Redis from 2.4 to 2.6 RC.
> > - Enabled the Redis Watchdog (see logs below)
> > * Some key numbers:
> > Redis instances: 4
> > Memory usage per instance: ~ 3GB
> > Amount of keys per instance: db0:keys=18644443,expires=9137407
> > Amount of set operations: 1200 - 1500 per second
> I don't know the client library, is it phpredis? looks like not only > you are not using persistent connections, you are closing the socket > on each request, that's really bad.
> this gives a hint:
> total_connections_received:119569
> total_commands_processed:1047293
> around 1:9 connection/commands ratio. I counted and your code has 9 > commands. :)
> so what you have here is tons of TCP sockets probably in time wait > state, blocking redis or the kernel or whatever.
> anyway, phpredis has the method *pconnect*() instead of connect, that > tries to pool connections.
> use it and not connect() and don't close your connection in the end.
> On Mon, May 21, 2012 at 9:52 PM, Hampus Wessman > <ham...@hampuswessman.se <mailto:ham...@hampuswessman.se>> wrote:
> Try not to create a new TCP connection each time. My guess is that
> your
> problem is related to that. Unfortunately, I don't know how to use a
> connection pool (aka persistent connections) for Redis in PHP (if
> at all
> possible). To check if this is the problem, perhaps you could try to
> connect using a local unix socket instead of TCP and see if the
> problem
> goes away?
> BR,
> Hampus
> On 05/21/12 19:55, Melvin Rook wrote:
> > Hi all,
> > For some time I am trying to use Redis in a high volume system, but
> > till now I am not happy about the performance of Redis. I wonder
> what
> > the issue is, am I just pushing Redis to it's limits or am I doing
> > something wrong?
> > I am using Redis to store a great amount of counters (of which some
> > have an expiry timestamp). However, after some time Redis starts to
> > block new connections for short periods of time and the CPU usage
> > becomes 100%. The overall performance is degraded at those times.
> > The following things were already improved/tried:
> > - Improved the client code
> > - Enabled PIPELINE
> > - Spreading the keys among 4 instances instead of just one instance
> > - Disabled snapshotting to the RDB file
> > - Updated Redis from 2.4 to 2.6 RC.
> > - Enabled the Redis Watchdog (see logs below)
> > * Some key numbers:
> > Redis instances: 4
> > Memory usage per instance: ~ 3GB
> > Amount of keys per instance: db0:keys=18644443,expires=9137407
> > Amount of set operations: 1200 - 1500 per second
Actually, at the moment only the 'writing' part is enabled. The data isn't being read actively yet.
I do not use sets. The following operations are performed within one multi pipeline:
- 1x SET
- 4x INCR
- 3x EXPIREAT (for 3 of the incremented keys)
Note that the EXPIREAT operation happens after each increment and not only for the first increment. Performing the EXPIREAT only the first time opens a way for race conditions as the key might be incremented at the same time by another client.
I wonder where in my reporting information sets are mentioned, because as far as I know I am not using sets in my client code. Might be that sets are used 'under water' by PHPRedis or Redis itself?
> Your reporting information mentions sets. How many sets are you
> passing to your operations, what operations are you performing, and
> how large are your sets?
> Regards,
> - Josiah
> On Mon, May 21, 2012 at 11:55 AM, Marc Byrd<dr.marc.b...@gmail.com> wrote:
>> You mentioned going from 1 instance to 4 - but these are all on the same
>> hardware?
>> What does # ulimit -n return? If it's the default 1024, that might be your
>> problem.
>> m
>> On Mon, May 21, 2012 at 11:52 AM, Hampus Wessman<ham...@hampuswessman.se>
>> wrote:
>>> Try not to create a new TCP connection each time. My guess is that your
>>> problem is related to that. Unfortunately, I don't know how to use a
>>> connection pool (aka persistent connections) for Redis in PHP (if at all
>>> possible). To check if this is the problem, perhaps you could try to
>>> connect using a local unix socket instead of TCP and see if the problem
>>> goes away?
>>> BR,
>>> Hampus
>>> On 05/21/12 19:55, Melvin Rook wrote:
>>>> Hi all,
>>>> For some time I am trying to use Redis in a high volume system, but
>>>> till now I am not happy about the performance of Redis. I wonder what
>>>> the issue is, am I just pushing Redis to it's limits or am I doing
>>>> something wrong?
>>>> I am using Redis to store a great amount of counters (of which some
>>>> have an expiry timestamp). However, after some time Redis starts to
>>>> block new connections for short periods of time and the CPU usage
>>>> becomes 100%. The overall performance is degraded at those times.
>>>> The following things were already improved/tried:
>>>> - Improved the client code
>>>> - Enabled PIPELINE
>>>> - Spreading the keys among 4 instances instead of just one instance
>>>> - Disabled snapshotting to the RDB file
>>>> - Updated Redis from 2.4 to 2.6 RC.
>>>> - Enabled the Redis Watchdog (see logs below)
>>>> * Some key numbers:
>>>> Redis instances: 4
>>>> Memory usage per instance: ~ 3GB
>>>> Amount of keys per instance: db0:keys=18644443,expires=9137407
>>>> Amount of set operations: 1200 - 1500 per second
"So the issue is, when this happens Redis currently will block
everything and expire keys in a busy loop til the number of already
expired keys (in the total of keys with an expire) became 25% or less.
25% is considered an acceptable memory waste to continue serving
queries again, expiring the remaining keys 100 per second,
incrementally. "
if you're creating tons of expires all the time, maybe the expire loop gets
too heavy?
have you tried this without expires?
On Tue, May 22, 2012 at 3:21 PM, Melvin Rook <mel...@melvinrook.nl> wrote:
> Actually, at the moment only the 'writing' part is enabled. The data isn't
> being read actively yet.
> I do not use sets. The following operations are performed within one multi
> pipeline:
> - 1x SET
> - 4x INCR
> - 3x EXPIREAT (for 3 of the incremented keys)
> Note that the EXPIREAT operation happens after each increment and not only
> for the first increment. Performing the EXPIREAT only the first time opens
> a way for race conditions as the key might be incremented at the same time
> by another client.
> I wonder where in my reporting information sets are mentioned, because as
> far as I know I am not using sets in my client code. Might be that sets are
> used 'under water' by PHPRedis or Redis itself?
> Kind regards,
> Melvin
> Op 21-5-2012 21:20, Josiah Carlson schreef:
> Your reporting information mentions sets. How many sets are you
>> passing to your operations, what operations are you performing, and
>> how large are your sets?
>> Regards,
>> - Josiah
>> On Mon, May 21, 2012 at 11:55 AM, Marc Byrd<dr.marc.b...@gmail.com>
>> wrote:
>>> You mentioned going from 1 instance to 4 - but these are all on the same
>>> hardware?
>>> What does # ulimit -n return? If it's the default 1024, that might be
>>> your
>>> problem.
>>> m
>>> On Mon, May 21, 2012 at 11:52 AM, Hampus Wessman<hampus@hampuswessman.**
>>> se <ham...@hampuswessman.se>>
>>> wrote:
>>>> Try not to create a new TCP connection each time. My guess is that your
>>>> problem is related to that. Unfortunately, I don't know how to use a
>>>> connection pool (aka persistent connections) for Redis in PHP (if at all
>>>> possible). To check if this is the problem, perhaps you could try to
>>>> connect using a local unix socket instead of TCP and see if the problem
>>>> goes away?
>>>> BR,
>>>> Hampus
>>>> On 05/21/12 19:55, Melvin Rook wrote:
>>>>> Hi all,
>>>>> For some time I am trying to use Redis in a high volume system, but
>>>>> till now I am not happy about the performance of Redis. I wonder what
>>>>> the issue is, am I just pushing Redis to it's limits or am I doing
>>>>> something wrong?
>>>>> I am using Redis to store a great amount of counters (of which some
>>>>> have an expiry timestamp). However, after some time Redis starts to
>>>>> block new connections for short periods of time and the CPU usage
>>>>> becomes 100%. The overall performance is degraded at those times.
>>>>> The following things were already improved/tried:
>>>>> - Improved the client code
>>>>> - Enabled PIPELINE
>>>>> - Spreading the keys among 4 instances instead of just one instance
>>>>> - Disabled snapshotting to the RDB file
>>>>> - Updated Redis from 2.4 to 2.6 RC.
>>>>> - Enabled the Redis Watchdog (see logs below)
>>>>> * Some key numbers:
>>>>> Redis instances: 4
>>>>> Memory usage per instance: ~ 3GB
>>>>> Amount of keys per instance: db0:keys=18644443,expires=**9137407
>>>>> Amount of set operations: 1200 - 1500 per second
On Tue, May 22, 2012 at 5:21 AM, Melvin Rook <mel...@melvinrook.nl> wrote:
> Actually, at the moment only the 'writing' part is enabled. The data isn't
> being read actively yet.
> I do not use sets. The following operations are performed within one multi
> pipeline:
> - 1x SET
> - 4x INCR
> - 3x EXPIREAT (for 3 of the incremented keys)
> Note that the EXPIREAT operation happens after each increment and not only
> for the first increment. Performing the EXPIREAT only the first time opens a
> way for race conditions as the key might be incremented at the same time by
> another client.
> I wonder where in my reporting information sets are mentioned, because as
> far as I know I am not using sets in my client code. Might be that sets are
> used 'under water' by PHPRedis or Redis itself?
"Amount of set operations: 1200 - 1500 per second"
It seemed ambiguous, which is why I asked. My guess is that Dvir is
right about the expirations with the lockups, and that you could
likely halve your command latency by taking Hampus' suggestion to use
connection pooling (which should have a viable implementation in php
at this point).
the watchdog output seems odd, like if the Redis process is blocked in
the middle of normal operations by swapping or some other OS activity
that does not allow it to run comfortably.
On Mon, May 21, 2012 at 7:55 PM, Melvin Rook <mel...@melvinrook.nl> wrote:
> Hi all,
> For some time I am trying to use Redis in a high volume system, but till now
> I am not happy about the performance of Redis. I wonder what the issue is,
> am I just pushing Redis to it's limits or am I doing something wrong?
> I am using Redis to store a great amount of counters (of which some have an
> expiry timestamp). However, after some time Redis starts to block new
> connections for short periods of time and the CPU usage becomes 100%. The
> overall performance is degraded at those times.
> The following things were already improved/tried:
> - Improved the client code
> - Enabled PIPELINE
> - Spreading the keys among 4 instances instead of just one instance
> - Disabled snapshotting to the RDB file
> - Updated Redis from 2.4 to 2.6 RC.
> - Enabled the Redis Watchdog (see logs below)
> * Some key numbers:
> Redis instances: 4
> Memory usage per instance: ~ 3GB
> Amount of keys per instance: db0:keys=18644443,expires=9137407
> Amount of set operations: 1200 - 1500 per second
I looked into that issue as well, but the expiry loop isn't that heavy as you might think. Actually, to be sure I checked the redis-server source code and the expiry algorithm is pretty solid.
> "So the issue is, when this happens Redis currently will block
> everything and expire keys in a busy loop til the number of already
> expired keys (in the total of keys with an expire) became 25% or less.
> 25% is considered an acceptable memory waste to continue serving
> queries again, expiring the remaining keys 100 per second,
> incrementally. "
> if you're creating tons of expires all the time, maybe the expire loop > gets too heavy?
> have you tried this without expires?
> On Tue, May 22, 2012 at 3:21 PM, Melvin Rook <mel...@melvinrook.nl > <mailto:mel...@melvinrook.nl>> wrote:
> Actually, at the moment only the 'writing' part is enabled. The
> data isn't being read actively yet.
> I do not use sets. The following operations are performed within
> one multi pipeline:
> - 1x SET
> - 4x INCR
> - 3x EXPIREAT (for 3 of the incremented keys)
> Note that the EXPIREAT operation happens after each increment and
> not only for the first increment. Performing the EXPIREAT only the
> first time opens a way for race conditions as the key might be
> incremented at the same time by another client.
> I wonder where in my reporting information sets are mentioned,
> because as far as I know I am not using sets in my client code.
> Might be that sets are used 'under water' by PHPRedis or Redis itself?
> Kind regards,
> Melvin
> Op 21-5-2012 21:20, Josiah Carlson schreef:
> Your reporting information mentions sets. How many sets are you
> passing to your operations, what operations are you
> performing, and
> how large are your sets?
> Regards,
> - Josiah
> On Mon, May 21, 2012 <tel:2012> at 11:55 AM, Marc
> Byrd<dr.marc.b...@gmail.com <mailto:dr.marc.b...@gmail.com>>
> wrote:
> You mentioned going from 1 instance to 4 - but these are
> all on the same
> hardware?
> What does # ulimit -n return? If it's the default 1024,
> that might be your
> problem.
> m
> On Mon, May 21, 2012 <tel:2012> at 11:52 AM, Hampus
> Wessman<ham...@hampuswessman.se
> <mailto:ham...@hampuswessman.se>>
> wrote:
> Try not to create a new TCP connection each time. My
> guess is that your
> problem is related to that. Unfortunately, I don't
> know how to use a
> connection pool (aka persistent connections) for Redis
> in PHP (if at all
> possible). To check if this is the problem, perhaps
> you could try to
> connect using a local unix socket instead of TCP and
> see if the problem
> goes away?
> BR,
> Hampus
> On 05/21/12 19:55, Melvin Rook wrote:
> Hi all,
> For some time I am trying to use Redis in a high
> volume system, but
> till now I am not happy about the performance of
> Redis. I wonder what
> the issue is, am I just pushing Redis to it's
> limits or am I doing
> something wrong?
> I am using Redis to store a great amount of
> counters (of which some
> have an expiry timestamp). However, after some
> time Redis starts to
> block new connections for short periods of time
> and the CPU usage
> becomes 100%. The overall performance is degraded
> at those times.
> The following things were already improved/tried:
> - Improved the client code
> - Enabled PIPELINE
> - Spreading the keys among 4 instances instead of
> just one instance
> - Disabled snapshotting to the RDB file
> - Updated Redis from 2.4 to 2.6 RC.
> - Enabled the Redis Watchdog (see logs below)
> * Some key numbers:
> Redis instances: 4
> Memory usage per instance: ~ 3GB
> Amount of keys per instance:
> db0:keys=18644443,expires=9137407
> Amount of set operations: 1200 - 1500 per second
Just out of curiosity, I want to run your code and see if I can recreate
the problem.
what's your server setup and how to grind it? will "ab -n 100" do?
On Wed, May 23, 2012 at 5:31 PM, Melvin Rook <mel...@melvinrook.nl> wrote:
> I looked into that issue as well, but the expiry loop isn't that heavy as
> you might think. Actually, to be sure I checked the redis-server source
> code and the expiry algorithm is pretty solid.
> "So the issue is, when this happens Redis currently will block
> everything and expire keys in a busy loop til the number of already
> expired keys (in the total of keys with an expire) became 25% or less.
> 25% is considered an acceptable memory waste to continue serving
> queries again, expiring the remaining keys 100 per second,
> incrementally. "
> if you're creating tons of expires all the time, maybe the expire loop
> gets too heavy?
> have you tried this without expires?
> On Tue, May 22, 2012 at 3:21 PM, Melvin Rook <mel...@melvinrook.nl> wrote:
>> Actually, at the moment only the 'writing' part is enabled. The data
>> isn't being read actively yet.
>> I do not use sets. The following operations are performed within one
>> multi pipeline:
>> - 1x SET
>> - 4x INCR
>> - 3x EXPIREAT (for 3 of the incremented keys)
>> Note that the EXPIREAT operation happens after each increment and not
>> only for the first increment. Performing the EXPIREAT only the first time
>> opens a way for race conditions as the key might be incremented at the same
>> time by another client.
>> I wonder where in my reporting information sets are mentioned, because as
>> far as I know I am not using sets in my client code. Might be that sets are
>> used 'under water' by PHPRedis or Redis itself?
>> Kind regards,
>> Melvin
>> Op 21-5-2012 21:20, Josiah Carlson schreef:
>> Your reporting information mentions sets. How many sets are you
>>> passing to your operations, what operations are you performing, and
>>> how large are your sets?
>>> Regards,
>>> - Josiah
>>> On Mon, May 21, 2012 at 11:55 AM, Marc Byrd<dr.marc.b...@gmail.com>
>>> wrote:
>>>> You mentioned going from 1 instance to 4 - but these are all on the same
>>>> hardware?
>>>> What does # ulimit -n return? If it's the default 1024, that might be
>>>> your
>>>> problem.
>>>> m
>>>> On Mon, May 21, 2012 at 11:52 AM, Hampus Wessman<
>>>> ham...@hampuswessman.se>
>>>> wrote:
>>>>> Try not to create a new TCP connection each time. My guess is that your
>>>>> problem is related to that. Unfortunately, I don't know how to use a
>>>>> connection pool (aka persistent connections) for Redis in PHP (if at
>>>>> all
>>>>> possible). To check if this is the problem, perhaps you could try to
>>>>> connect using a local unix socket instead of TCP and see if the problem
>>>>> goes away?
>>>>> BR,
>>>>> Hampus
>>>>> On 05/21/12 19:55, Melvin Rook wrote:
>>>>>> Hi all,
>>>>>> For some time I am trying to use Redis in a high volume system, but
>>>>>> till now I am not happy about the performance of Redis. I wonder what
>>>>>> the issue is, am I just pushing Redis to it's limits or am I doing
>>>>>> something wrong?
>>>>>> I am using Redis to store a great amount of counters (of which some
>>>>>> have an expiry timestamp). However, after some time Redis starts to
>>>>>> block new connections for short periods of time and the CPU usage
>>>>>> becomes 100%. The overall performance is degraded at those times.
>>>>>> The following things were already improved/tried:
>>>>>> - Improved the client code
>>>>>> - Enabled PIPELINE
>>>>>> - Spreading the keys among 4 instances instead of just one instance
>>>>>> - Disabled snapshotting to the RDB file
>>>>>> - Updated Redis from 2.4 to 2.6 RC.
>>>>>> - Enabled the Redis Watchdog (see logs below)
>>>>>> * Some key numbers:
>>>>>> Redis instances: 4
>>>>>> Memory usage per instance: ~ 3GB
>>>>>> Amount of keys per instance: db0:keys=18644443,expires=9137407
>>>>>> Amount of set operations: 1200 - 1500 per second
You are right about the watchdog output, I suspect the blocking is caused by the BGSAVE fork. I tracked down that a great number of blocked connections were happening during the fork.
With all the adjustments I ended up with quite a mess. Just to be sure I reconfigured everything from scratch again and performed all my configuration cases again. Seems I overlooked a few cases in combination with the 2.6 RC version (which I did performed with 2.4).
I found out that pconnect didn't work well on 2.4 with BGSAVE on. However, with 2.6 the number of blocked connections are less, but not yet gone. All the blocked connections disappeared after I disabled BGSAVE completely.
I will probably initiate BGSAVE manually during off-peak times to at least have a daily dump of the data.
To recap:
- BGSAVE could be a cause of blocked connections (-> disable BGSAVE)
- connect could be a cause of blocked connections (-> use pconnect)
- a combination of the above two reasons, used in a high volume system, could be the reason for a bad overall performance.
Other improvements I made:
- According to my tests 2.6 seems to perform better in comparison with 2.4
- Configured the number of databases to 1. If I understand the code correctly, it will save a few loop iterations in the expiry algorithm.
- Spreading the keys among multiple Redis instances (using modulo). A rule of thumb I used:
- With BGSAVE on: # instances = physical CPU cores / 2
- Without BGSAVE: # instances = physical CPU cores / 1
> the watchdog output seems odd, like if the Redis process is blocked in
> the middle of normal operations by swapping or some other OS activity
> that does not allow it to run comfortably.
> Cheers,
> Salvatore
> On Mon, May 21, 2012 at 7:55 PM, Melvin Rook<mel...@melvinrook.nl> wrote:
>> Hi all,
>> For some time I am trying to use Redis in a high volume system, but till now
>> I am not happy about the performance of Redis. I wonder what the issue is,
>> am I just pushing Redis to it's limits or am I doing something wrong?
>> I am using Redis to store a great amount of counters (of which some have an
>> expiry timestamp). However, after some time Redis starts to block new
>> connections for short periods of time and the CPU usage becomes 100%. The
>> overall performance is degraded at those times.
>> The following things were already improved/tried:
>> - Improved the client code
>> - Enabled PIPELINE
>> - Spreading the keys among 4 instances instead of just one instance
>> - Disabled snapshotting to the RDB file
>> - Updated Redis from 2.4 to 2.6 RC.
>> - Enabled the Redis Watchdog (see logs below)
>> * Some key numbers:
>> Redis instances: 4
>> Memory usage per instance: ~ 3GB
>> Amount of keys per instance: db0:keys=18644443,expires=9137407
>> Amount of set operations: 1200 - 1500 per second
Please note my other reply as well, as for now the overall performance issue seems to be resolved. As for your curiosity:
Let's say you want to reproduce it on just one instance (I am using multiple instances), then "ab -n 200" will come pretty close I think. Make sure that you reach about 18 million keys and a DB size of about 3 GB.
You will notice that the expiry algorithm isn't that heavy. If you play around with BGSAVE and (p)connect you will see that those affect performance really a lot.
With BGSAVE enabled you will see that Redis will start to block connections during the fork. Another solution would be to catch that event in the client code, then wait a bit and try to reconnect.
> Just out of curiosity, I want to run your code and see if I can > recreate the problem.
> what's your server setup and how to grind it? will "ab -n 100" do?
> On Wed, May 23, 2012 at 5:31 PM, Melvin Rook <mel...@melvinrook.nl > <mailto:mel...@melvinrook.nl>> wrote:
> I looked into that issue as well, but the expiry loop isn't that
> heavy as you might think. Actually, to be sure I checked the
> redis-server source code and the expiry algorithm is pretty solid.
>> "So the issue is, when this happens Redis currently will block
>> everything and expire keys in a busy loop til the number of already
>> expired keys (in the total of keys with an expire) became 25% or
>> less.
>> 25% is considered an acceptable memory waste to continue serving
>> queries again, expiring the remaining keys 100 per second,
>> incrementally. "
>> if you're creating tons of expires all the time, maybe the expire
>> loop gets too heavy?
>> have you tried this without expires?
>> On Tue, May 22, 2012 <tel:2012> at 3:21 PM, Melvin Rook
>> <mel...@melvinrook.nl <mailto:mel...@melvinrook.nl>> wrote:
>> Actually, at the moment only the 'writing' part is enabled.
>> The data isn't being read actively yet.
>> I do not use sets. The following operations are performed
>> within one multi pipeline:
>> - 1x SET
>> - 4x INCR
>> - 3x EXPIREAT (for 3 of the incremented keys)
>> Note that the EXPIREAT operation happens after each increment
>> and not only for the first increment. Performing the EXPIREAT
>> only the first time opens a way for race conditions as the
>> key might be incremented at the same time by another client.
>> I wonder where in my reporting information sets are
>> mentioned, because as far as I know I am not using sets in my
>> client code. Might be that sets are used 'under water' by
>> PHPRedis or Redis itself?
>> Kind regards,
>> Melvin
>> Op 21-5-2012 21:20, Josiah Carlson schreef:
>> Your reporting information mentions sets. How many sets
>> are you
>> passing to your operations, what operations are you
>> performing, and
>> how large are your sets?
>> Regards,
>> - Josiah
>> On Mon, May 21, 2012 <tel:2012> at 11:55 AM, Marc
>> Byrd<dr.marc.b...@gmail.com
>> <mailto:dr.marc.b...@gmail.com>> wrote:
>> You mentioned going from 1 instance to 4 - but these
>> are all on the same
>> hardware?
>> What does # ulimit -n return? If it's the default
>> 1024, that might be your
>> problem.
>> m
>> On Mon, May 21, 2012 <tel:2012> at 11:52 AM, Hampus
>> Wessman<ham...@hampuswessman.se
>> <mailto:ham...@hampuswessman.se>>
>> wrote:
>> Try not to create a new TCP connection each time.
>> My guess is that your
>> problem is related to that. Unfortunately, I
>> don't know how to use a
>> connection pool (aka persistent connections) for
>> Redis in PHP (if at all
>> possible). To check if this is the problem,
>> perhaps you could try to
>> connect using a local unix socket instead of TCP
>> and see if the problem
>> goes away?
>> BR,
>> Hampus
>> On 05/21/12 19:55, Melvin Rook wrote:
>> Hi all,
>> For some time I am trying to use Redis in a
>> high volume system, but
>> till now I am not happy about the performance
>> of Redis. I wonder what
>> the issue is, am I just pushing Redis to it's
>> limits or am I doing
>> something wrong?
>> I am using Redis to store a great amount of
>> counters (of which some
>> have an expiry timestamp). However, after
>> some time Redis starts to
>> block new connections for short periods of
>> time and the CPU usage
>> becomes 100%. The overall performance is
>> degraded at those times.
>> The following things were already improved/tried:
>> - Improved the client code
>> - Enabled PIPELINE
>> - Spreading the keys among 4 instances
>> instead of just one instance
>> - Disabled snapshotting to the RDB file
>> - Updated Redis from 2.4 to 2.6 RC.
>> - Enabled the Redis Watchdog (see logs below)
>> * Some key numbers:
>> Redis instances: 4
>> Memory usage per instance: ~ 3GB
>> Amount of keys per instance:
>> db0:keys=18644443,expires=9137407
>> Amount of set operations: 1200 - 1500 per second
cool, although now that you've figured it out, the curiosity is quite over
:)
re your previous email -
if you can spare the resources, to keep a slave that does all the BGSAVE
and let the master be persistence free. this will solve the forking issues.
On Wed, May 23, 2012 at 6:20 PM, Melvin Rook <mel...@melvinrook.nl> wrote:
> Please note my other reply as well, as for now the overall performance
> issue seems to be resolved. As for your curiosity:
> Let's say you want to reproduce it on just one instance (I am using
> multiple instances), then "ab -n 200" will come pretty close I think. Make
> sure that you reach about 18 million keys and a DB size of about 3 GB.
> You will notice that the expiry algorithm isn't that heavy. If you play
> around with BGSAVE and (p)connect you will see that those affect
> performance really a lot.
> With BGSAVE enabled you will see that Redis will start to block
> connections during the fork. Another solution would be to catch that event
> in the client code, then wait a bit and try to reconnect.
> Kind regards,
> Melvin
> Op 23-5-2012 16:34, Dvir Volk schreef:
> Just out of curiosity, I want to run your code and see if I can recreate
> the problem.
> what's your server setup and how to grind it? will "ab -n 100" do?
> On Wed, May 23, 2012 at 5:31 PM, Melvin Rook <mel...@melvinrook.nl> wrote:
>> I looked into that issue as well, but the expiry loop isn't that heavy
>> as you might think. Actually, to be sure I checked the redis-server source
>> code and the expiry algorithm is pretty solid.
>> "So the issue is, when this happens Redis currently will block
>> everything and expire keys in a busy loop til the number of already
>> expired keys (in the total of keys with an expire) became 25% or less.
>> 25% is considered an acceptable memory waste to continue serving
>> queries again, expiring the remaining keys 100 per second,
>> incrementally. "
>> if you're creating tons of expires all the time, maybe the expire loop
>> gets too heavy?
>> have you tried this without expires?
>> On Tue, May 22, 2012 at 3:21 PM, Melvin Rook <mel...@melvinrook.nl>wrote:
>>> Actually, at the moment only the 'writing' part is enabled. The data
>>> isn't being read actively yet.
>>> I do not use sets. The following operations are performed within one
>>> multi pipeline:
>>> - 1x SET
>>> - 4x INCR
>>> - 3x EXPIREAT (for 3 of the incremented keys)
>>> Note that the EXPIREAT operation happens after each increment and not
>>> only for the first increment. Performing the EXPIREAT only the first time
>>> opens a way for race conditions as the key might be incremented at the same
>>> time by another client.
>>> I wonder where in my reporting information sets are mentioned, because
>>> as far as I know I am not using sets in my client code. Might be that sets
>>> are used 'under water' by PHPRedis or Redis itself?
>>> Kind regards,
>>> Melvin
>>> Op 21-5-2012 21:20, Josiah Carlson schreef:
>>> Your reporting information mentions sets. How many sets are you
>>>> passing to your operations, what operations are you performing, and
>>>> how large are your sets?
>>>> Regards,
>>>> - Josiah
>>>> On Mon, May 21, 2012 at 11:55 AM, Marc Byrd<dr.marc.b...@gmail.com>
>>>> wrote:
>>>>> You mentioned going from 1 instance to 4 - but these are all on the
>>>>> same
>>>>> hardware?
>>>>> What does # ulimit -n return? If it's the default 1024, that might be
>>>>> your
>>>>> problem.
>>>>> m
>>>>> On Mon, May 21, 2012 at 11:52 AM, Hampus Wessman<
>>>>> ham...@hampuswessman.se>
>>>>> wrote:
>>>>>> Try not to create a new TCP connection each time. My guess is that
>>>>>> your
>>>>>> problem is related to that. Unfortunately, I don't know how to use a
>>>>>> connection pool (aka persistent connections) for Redis in PHP (if at
>>>>>> all
>>>>>> possible). To check if this is the problem, perhaps you could try to
>>>>>> connect using a local unix socket instead of TCP and see if the
>>>>>> problem
>>>>>> goes away?
>>>>>> BR,
>>>>>> Hampus
>>>>>> On 05/21/12 19:55, Melvin Rook wrote:
>>>>>>> Hi all,
>>>>>>> For some time I am trying to use Redis in a high volume system, but
>>>>>>> till now I am not happy about the performance of Redis. I wonder what
>>>>>>> the issue is, am I just pushing Redis to it's limits or am I doing
>>>>>>> something wrong?
>>>>>>> I am using Redis to store a great amount of counters (of which some
>>>>>>> have an expiry timestamp). However, after some time Redis starts to
>>>>>>> block new connections for short periods of time and the CPU usage
>>>>>>> becomes 100%. The overall performance is degraded at those times.
>>>>>>> The following things were already improved/tried:
>>>>>>> - Improved the client code
>>>>>>> - Enabled PIPELINE
>>>>>>> - Spreading the keys among 4 instances instead of just one instance
>>>>>>> - Disabled snapshotting to the RDB file
>>>>>>> - Updated Redis from 2.4 to 2.6 RC.
>>>>>>> - Enabled the Redis Watchdog (see logs below)
>>>>>>> * Some key numbers:
>>>>>>> Redis instances: 4
>>>>>>> Memory usage per instance: ~ 3GB
>>>>>>> Amount of keys per instance: db0:keys=18644443,expires=9137407
>>>>>>> Amount of set operations: 1200 - 1500 per second
