I have an application* *where multiple clients that a) tries to get from redis; b) if it's not there, then it get the value from db and set it to expire in 10 seconds;
the clients run the above logic every 11 seconds. There are only two keys that are used, each with less than 200k bytes value. there are about 50 clients. However, after running the clients for 30 mins, Redis memory usage went way up to exceed the memory limit.
It seems that redis doesn't handle expiry gracefully -- is it because it doesn't do de-fragment aggressively or is there a bug? thanks.
On Thu, Oct 25, 2012 at 1:47 PM, c.z <c...@rocketfuelinc.com> wrote:
> I have an application where multiple clients that
> a) tries to get from redis;
> b) if it's not there, then it get the value from db and set it to expire in
> 10 seconds;
> the clients run the above logic every 11 seconds. There are only two keys
> that are used, each with less than 200k bytes value.
> there are about 50 clients. However, after running the clients for 30 mins,
> Redis memory usage went way up to exceed the memory limit.
> It seems that redis doesn't handle expiry gracefully -- is it because it
> doesn't do de-fragment aggressively or is there a bug?
> thanks.
> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/mcOTtRjZDwoJ.
> To post to this group, send email to redis-db@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
here is the info output. in this case, we stopped using expiry, just 6 keys, 4 of which gets updated every 10 seconds with new value, the max memory is 256M
On Thursday, October 25, 2012 2:21:06 PM UTC-7, Josiah Carlson wrote:
> When Redis is using a lot of memory, can you send us the output of INFO?
> Regards, > - Josiah
> On Thu, Oct 25, 2012 at 1:47 PM, c.z <cz...@rocketfuelinc.com<javascript:>> > wrote: > > I have an application where multiple clients that > > a) tries to get from redis; > > b) if it's not there, then it get the value from db and set it to expire > in > > 10 seconds;
> > the clients run the above logic every 11 seconds. There are only two > keys > > that are used, each with less than 200k bytes value. > > there are about 50 clients. However, after running the clients for 30 > mins, > > Redis memory usage went way up to exceed the memory limit.
> > It seems that redis doesn't handle expiry gracefully -- is it because it > > doesn't do de-fragment aggressively or is there a bug? > > thanks.
> > -- > > You received this message because you are subscribed to the Google > Groups > > "Redis DB" group. > > To view this discussion on the web visit > > https://groups.google.com/d/msg/redis-db/-/mcOTtRjZDwoJ. > > To post to this group, send email to redi...@googlegroups.com<javascript:>.
> > To unsubscribe from this group, send email to > > redis-db+u...@googlegroups.com <javascript:>. > > For more options, visit this group at > > http://groups.google.com/group/redis-db?hl=en.
On Thu, Oct 25, 2012 at 7:01 PM, c.z <c...@rocketfuelinc.com> wrote:
> here is the info output. in this case, we stopped using expiry, just 6 keys,
> 4 of which gets updated every 10 seconds with new value, the max memory is
> 256M
> On Thursday, October 25, 2012 2:21:06 PM UTC-7, Josiah Carlson wrote:
>> When Redis is using a lot of memory, can you send us the output of INFO?
>> Regards,
>> - Josiah
>> On Thu, Oct 25, 2012 at 1:47 PM, c.z <cz...@rocketfuelinc.com> wrote:
>> > I have an application where multiple clients that
>> > a) tries to get from redis;
>> > b) if it's not there, then it get the value from db and set it to expire
>> > in
>> > 10 seconds;
>> > the clients run the above logic every 11 seconds. There are only two
>> > keys
>> > that are used, each with less than 200k bytes value.
>> > there are about 50 clients. However, after running the clients for 30
>> > mins,
>> > Redis memory usage went way up to exceed the memory limit.
>> > It seems that redis doesn't handle expiry gracefully -- is it because it
>> > doesn't do de-fragment aggressively or is there a bug?
>> > thanks.
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Redis DB" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/redis-db/-/mcOTtRjZDwoJ.
>> > To post to this group, send email to redi...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > redis-db+u...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/redis-db?hl=en.
> To post to this group, send email to redis-db@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
On Thursday, October 25, 2012 8:00:48 PM UTC-7, Josiah Carlson wrote:
> That is odd. If you were to execute a BGSAVE, how large is the resulting > .rdb?
> - Josiah
> On Thu, Oct 25, 2012 at 7:01 PM, c.z <cz...@rocketfuelinc.com<javascript:>> > wrote:
> > here is the info output. in this case, we stopped using expiry, just 6 > keys, > > 4 of which gets updated every 10 seconds with new value, the max memory > is > > 256M
> > On Thursday, October 25, 2012 2:21:06 PM UTC-7, Josiah Carlson wrote:
> >> When Redis is using a lot of memory, can you send us the output of > INFO?
> >> Regards, > >> - Josiah
> >> On Thu, Oct 25, 2012 at 1:47 PM, c.z <cz...@rocketfuelinc.com> wrote: > >> > I have an application where multiple clients that > >> > a) tries to get from redis; > >> > b) if it's not there, then it get the value from db and set it to > expire > >> > in > >> > 10 seconds;
> >> > the clients run the above logic every 11 seconds. There are only two > >> > keys > >> > that are used, each with less than 200k bytes value. > >> > there are about 50 clients. However, after running the clients for 30 > >> > mins, > >> > Redis memory usage went way up to exceed the memory limit.
> >> > It seems that redis doesn't handle expiry gracefully -- is it because > it > >> > doesn't do de-fragment aggressively or is there a bug? > >> > thanks.
> >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Redis DB" group. > >> > To view this discussion on the web visit > >> > https://groups.google.com/d/msg/redis-db/-/mcOTtRjZDwoJ. > >> > To post to this group, send email to redi...@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > redis-db+u...@googlegroups.com. > >> > For more options, visit this group at > >> > http://groups.google.com/group/redis-db?hl=en.
> > To post to this group, send email to redi...@googlegroups.com<javascript:>.
> > To unsubscribe from this group, send email to > > redis-db+u...@googlegroups.com <javascript:>. > > For more options, visit this group at > > http://groups.google.com/group/redis-db?hl=en.
If you have 6 keys/values and a 4.6M dump, that means that each
key/value pair averages about 760k compressed. Your original claim
that each key/value is less than 200k is incorrect.
Could you do us another favor and call STRLEN on each value, and also
calculate the length of the related key? I suspect that you will find
that your <200kbyte key/value pairs are actually substantially larger
than what you expect, as no one has previously experienced the bug you
seem to be experiencing.
On Thu, Oct 25, 2012 at 9:41 PM, c.z <c...@rocketfuelinc.com> wrote:
> dump size is 4658467. Redis has some serious problems.
> On Thursday, October 25, 2012 8:00:48 PM UTC-7, Josiah Carlson wrote:
>> That is odd. If you were to execute a BGSAVE, how large is the resulting
>> .rdb?
>> - Josiah
>> On Thu, Oct 25, 2012 at 7:01 PM, c.z <cz...@rocketfuelinc.com> wrote:
>> > here is the info output. in this case, we stopped using expiry, just 6
>> > keys,
>> > 4 of which gets updated every 10 seconds with new value, the max memory
>> > is
>> > 256M
>> > On Thursday, October 25, 2012 2:21:06 PM UTC-7, Josiah Carlson wrote:
>> >> When Redis is using a lot of memory, can you send us the output of
>> >> INFO?
>> >> Regards,
>> >> - Josiah
>> >> On Thu, Oct 25, 2012 at 1:47 PM, c.z <cz...@rocketfuelinc.com> wrote:
>> >> > I have an application where multiple clients that
>> >> > a) tries to get from redis;
>> >> > b) if it's not there, then it get the value from db and set it to
>> >> > expire
>> >> > in
>> >> > 10 seconds;
>> >> > the clients run the above logic every 11 seconds. There are only two
>> >> > keys
>> >> > that are used, each with less than 200k bytes value.
>> >> > there are about 50 clients. However, after running the clients for 30
>> >> > mins,
>> >> > Redis memory usage went way up to exceed the memory limit.
>> >> > It seems that redis doesn't handle expiry gracefully -- is it because
>> >> > it
>> >> > doesn't do de-fragment aggressively or is there a bug?
>> >> > thanks.
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "Redis DB" group.
>> >> > To view this discussion on the web visit
>> >> > https://groups.google.com/d/msg/redis-db/-/mcOTtRjZDwoJ.
>> >> > To post to this group, send email to redi...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to
>> >> > redis-db+u...@googlegroups.com.
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/redis-db?hl=en.
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Redis DB" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/redis-db/-/tnABAQuCl60J.
>> > To post to this group, send email to redi...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > redis-db+u...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/redis-db?hl=en.
> To post to this group, send email to redis-db@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
On Thursday, October 25, 2012 11:05:37 PM UTC-7, Josiah Carlson wrote:
> If you have 6 keys/values and a 4.6M dump, that means that each > key/value pair averages about 760k compressed. Your original claim > that each key/value is less than 200k is incorrect.
> Could you do us another favor and call STRLEN on each value, and also > calculate the length of the related key? I suspect that you will find > that your <200kbyte key/value pairs are actually substantially larger > than what you expect, as no one has previously experienced the bug you > seem to be experiencing.
> Regards, > - Josiah
> On Thu, Oct 25, 2012 at 9:41 PM, c.z <cz...@rocketfuelinc.com<javascript:>> > wrote: > > dump size is 4658467. Redis has some serious problems.
> > On Thursday, October 25, 2012 8:00:48 PM UTC-7, Josiah Carlson wrote:
> >> That is odd. If you were to execute a BGSAVE, how large is the > resulting > >> .rdb?
> >> - Josiah
> >> On Thu, Oct 25, 2012 at 7:01 PM, c.z <cz...@rocketfuelinc.com> wrote:
> >> > here is the info output. in this case, we stopped using expiry, just > 6 > >> > keys, > >> > 4 of which gets updated every 10 seconds with new value, the max > memory > >> > is > >> > 256M
> >> > On Thursday, October 25, 2012 2:21:06 PM UTC-7, Josiah Carlson wrote:
> >> >> When Redis is using a lot of memory, can you send us the output of > >> >> INFO?
> >> >> Regards, > >> >> - Josiah
> >> >> On Thu, Oct 25, 2012 at 1:47 PM, c.z <cz...@rocketfuelinc.com> > wrote: > >> >> > I have an application where multiple clients that > >> >> > a) tries to get from redis; > >> >> > b) if it's not there, then it get the value from db and set it to > >> >> > expire > >> >> > in > >> >> > 10 seconds;
> >> >> > the clients run the above logic every 11 seconds. There are only > two > >> >> > keys > >> >> > that are used, each with less than 200k bytes value. > >> >> > there are about 50 clients. However, after running the clients for > 30 > >> >> > mins, > >> >> > Redis memory usage went way up to exceed the memory limit.
> >> >> > It seems that redis doesn't handle expiry gracefully -- is it > because > >> >> > it > >> >> > doesn't do de-fragment aggressively or is there a bug? > >> >> > thanks.
> >> >> > -- > >> >> > You received this message because you are subscribed to the Google > >> >> > Groups > >> >> > "Redis DB" group. > >> >> > To view this discussion on the web visit > >> >> > https://groups.google.com/d/msg/redis-db/-/mcOTtRjZDwoJ. > >> >> > To post to this group, send email to redi...@googlegroups.com. > >> >> > To unsubscribe from this group, send email to > >> >> > redis-db+u...@googlegroups.com. > >> >> > For more options, visit this group at > >> >> > http://groups.google.com/group/redis-db?hl=en.
> >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Redis DB" group. > >> > To view this discussion on the web visit > >> > https://groups.google.com/d/msg/redis-db/-/tnABAQuCl60J.
> >> > To post to this group, send email to redi...@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > redis-db+u...@googlegroups.com. > >> > For more options, visit this group at > >> > http://groups.google.com/group/redis-db?hl=en.
> > To post to this group, send email to redi...@googlegroups.com<javascript:>.
> > To unsubscribe from this group, send email to > > redis-db+u...@googlegroups.com <javascript:>. > > For more options, visit this group at > > http://groups.google.com/group/redis-db?hl=en.
