Redis-Sentinel Tilt
1,842 views
Skip to first unread message
Zach Newman
Jun 3, 2015, 1:58:35 AM6/3/15
to redi...@googlegroups.com
We are currently running redis 2.8.19 in our staging environment, evaluating for production. Throughout the day sentinel has been entering tilt mode briefly without explanation. We have not updated this system for over two months and haven't experienced this before.
Redis and Sentinel run on the same host, so the time should not differ. Our storage team has confirmed that the iops are well below the expected maximum so disk iIO should not be blocking. The storage is on an enterprise level SAN with SAS disks. Memory and CPU are constantly under 10% utilization. I am at a loss to why tilt mode is entered and how to debug this. Any suggestions?
Jan-Erik Rediger
Jun 3, 2015, 3:51:11 AM6/3/15
to redi...@googlegroups.com
TILT mode is a local problem to a Sentinel node.
See http://redis.io/topics/sentinel#tilt-mode
It is entered if Sentinel detects a unexpected change in the system
time.
Are you're servers running with NTP sync?
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.
See http://redis.io/topics/sentinel#tilt-mode
It is entered if Sentinel detects a unexpected change in the system
time.
Are you're servers running with NTP sync?
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.
Zach Newman
Jun 3, 2015, 8:04:35 AM6/3/15
to redi...@googlegroups.com, jan...@fnordig.de
We disabled ntpd to debug, and sentinel is still entering tilt mode.
Zach Newman
Jun 3, 2015, 8:05:37 AM6/3/15
to redi...@googlegroups.com, jan...@fnordig.de
[24123] 03 Jun 06:51:38.725 # +tilt #tilt mode entered
[24123] 03 Jun 06:52:08.733 # -tilt #tilt mode exited
Salvatore Sanfilippo
Jun 3, 2015, 8:10:27 AM6/3/15
to Redis DB, Jan-Erik Rediger
If Sentinel enters tilt mode without any time jump, it is likely due
to it being blocked for multiple seconds syncing the configuration on
disk, so when it is able again to enter the event loop, it discovers
an unusual time jump that can't be explained. IMHO the right fix is:
1) Check if my hypothesis is true with strace or alike.
2) Assuming 1 passes, to understand why the environment where you are
running has so poor disk performances.
Otherwise you could modify the following define:
#define SENTINEL_TILT_TRIGGER 2000
From 2 seconds (2000 milliseconds) to an higher value. However I would
not run Sentinels in an environment where to fsync from time to time
uses multiple seconds.
--
Salvatore 'antirez' Sanfilippo
open source developer - Pivotal http://pivotal.io
"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.
to it being blocked for multiple seconds syncing the configuration on
disk, so when it is able again to enter the event loop, it discovers
an unusual time jump that can't be explained. IMHO the right fix is:
1) Check if my hypothesis is true with strace or alike.
2) Assuming 1 passes, to understand why the environment where you are
running has so poor disk performances.
Otherwise you could modify the following define:
#define SENTINEL_TILT_TRIGGER 2000
From 2 seconds (2000 milliseconds) to an higher value. However I would
not run Sentinels in an environment where to fsync from time to time
uses multiple seconds.
Salvatore 'antirez' Sanfilippo
open source developer - Pivotal http://pivotal.io
"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.
Jan-Erik Rediger
Jun 2, 2016, 6:56:30 AM6/2/16
to Pascal Oliver, Redis DB
It's not dynamically configurable.
On Thu, Jun 02, 2016 at 01:18:44AM -0700, Pascal Oliver wrote:
> Hey,
>
> how can I modify the following define:
>
> #define SENTINEL_TILT_TRIGGER 2000
>
> In which config file is this define?
>
> Regards,
> Pascal
> > > email to redis-db+u...@googlegroups.com <javascript:>.
On Thu, Jun 02, 2016 at 01:18:44AM -0700, Pascal Oliver wrote:
> Hey,
>
> how can I modify the following define:
>
> #define SENTINEL_TILT_TRIGGER 2000
>
> In which config file is this define?
>
> Regards,
> Pascal
> > > To post to this group, send email to redi...@googlegroups.com
> > <javascript:>.
Pascal Oliver
Jun 2, 2016, 7:27:20 AM6/2/16
to Redis DB, jan...@fnordig.de
Hey,
how can I modify the following define:
#define SENTINEL_TILT_TRIGGER 2000
how can I modify the following define:
#define SENTINEL_TILT_TRIGGER 2000
In which config file is this define?
Regards,
Pascal
Am Mittwoch, 3. Juni 2015 14:10:27 UTC+2 schrieb Salvatore Sanfilippo:
AlexanderB
Jun 3, 2016, 6:25:32 PM6/3/16
to Redis DB, jan...@fnordig.de
You would likely need to change that in source and rebuild sentinel from source, and redeploy your built from source version.
Reply all
Reply to author
Forward
0 new messages