[RateLimit] Help with Latency debugging

[Raman Sharma]

Hi All,

I'm debugging a latency issue with RateLimit Service. We use REDIS and suddenly we saw latency increase from 35ms to 150ms. There was no traffic increase that could have caused this.

Found that the metric ratelimit_redis_pool_cx_local_close is erratic from the day the latency increased.



What can be the reasons for this increase. Have looked that our Redis cluster is stable.
I have also increased size of connection pool but it hasn't yielded any result.

I have used REDIS_PIPELINE_WINDOW and REDIS_PIPELINE_LIMIT but i found it counter productive and it increased the latency further. I was expecting it will actually reduce some latency instead. What can be the reason for this?

What are the other metric I should look at to debug this further? What are other settings that can help?

FYI I'm using XDS server to load the configurations. But that is not a recent change.

Thanks in advance,

Raman Sharma