client-output-buffer-limit behavior for periodic large messages

[Bob Gordon]

In my config: client-output-buffer-limit pubsub 32mb 8mb 60

Suppose my client periodically receives 8MB messages, but rarely anything smaller. It looks like the soft limit is enforced by checkClientOutputBufferLimits, which is only called when publishing data to my client. Even if I read each message in well under 60 seconds, the next time a message gets published Redis still has obuf_soft_limit_reached_time != 0 and thinks the soft limit's been exceeded ever since the first one was published.

My buffer is not "continuously" larger than 8MB. It's only larger for short periods before and after each call to check the buffer.

Is this intended? If so, I can simply disable soft limits in my config. Or should the soft limit get checked when removing data from the buffer, too? (Maybe sendReplyToClient or something?)

Apologies if I've misunderstood this code somehow. I'm a newbie to the Redis code!

[Josiah Carlson]

On Tue, Oct 7, 2014 at 4:31 PM, Bob Gordon <bobgo...@gmail.com> wrote:

In my config: client-output-buffer-limit pubsub 32mb 8mb 60

Suppose my client periodically receives 8MB messages, but rarely anything smaller. It looks like the soft limit is enforced by checkClientOutputBufferLimits, which is only called when publishing data to my client. Even if I read each message in well under 60 seconds, the next time a message gets published Redis still has obuf_soft_limit_reached_time != 0 and thinks the soft limit's been exceeded ever since the first one was published.

My buffer is not "continuously" larger than 8MB. It's only larger for short periods before and after each call to check the buffer.

Is this intended?

This was probably not intended. Looking at the code, it does seem as though your analysis is correct.

 

If so, I can simply disable soft limits in my config. Or should the soft limit get checked when removing data from the buffer, too? (Maybe sendReplyToClient or something?)

That looks like a good spot. You should submit a pull request with the changes at Github.

Apologies if I've misunderstood this code somehow. I'm a newbie to the Redis code!

Everyone starts somewhere. :)

 - Josiah

-- 

You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Bob Gordon]

On Friday, October 10, 2014 at 4:29:44 PM UTC-4, Josiah Carlson wrote:

On Tue, Oct 7, 2014 at 4:31 PM, Bob Gordon <bobgo...@gmail.com> wrote:

In my config: client-output-buffer-limit pubsub 32mb 8mb 60

Suppose my client periodically receives 8MB messages, but rarely anything smaller. It looks like the soft limit is enforced by checkClientOutputBufferLimits, which is only called when publishing data to my client. Even if I read each message in well under 60 seconds, the next time a message gets published Redis still has obuf_soft_limit_reached_time != 0 and thinks the soft limit's been exceeded ever since the first one was published.

My buffer is not "continuously" larger than 8MB. It's only larger for short periods before and after each call to check the buffer.

Is this intended?

This was probably not intended. Looking at the code, it does seem as though your analysis is correct.

 

Sounds good, thanks. Can you please look at my pull request https://github.com/antirez/redis/pull/2063 ?

[Josiah Carlson]

The code looks fine to me, though you can/should move the checkClientOutputBufferLimits() prototype to networking.c, around the existing setProtocolError() prototype.

Until this gets merged (if it does get merged), you can obviously increase your soft limit to (hopefully) address your disconnection problem (which I suspect you've already done ~4 months ago now).

 - Josiah

--

[Bob Gordon]

On Friday, February 6, 2015 at 7:06:11 PM UTC-5, Josiah Carlson wrote:

The code looks fine to me, though you can/should move the checkClientOutputBufferLimits() prototype to networking.c, around the existing setProtocolError() prototype.

Oops! Yes, that would be better. I missed the existing one.
 

Until this gets merged (if it does get merged), you can obviously increase your soft limit to (hopefully) address your disconnection problem (which I suspect you've already done ~4 months ago now).

We actually set it to 0 but it's annoying to propagate that to certain hosted Redis products where we don't have direct access to either the config file or CONFIG GET commands.
 

[Bob Gordon]

Pull request updated.