Hello Everyone,
I did not get much information on this issue anywhere. Hence posting the issue here.
I am running apscheduler in EKS which stores the jobs in Redis which runs with Sentinel for HA Setup.
While Creating the scheduler I am doing it like below.
sentinel = Sentinel([redis_host,redis_port], socket_timeout=redis_socket_timeout)
master_host, master_port = sentinel.discover_master(redis_master_name)
jobstores = {
'redis': RedisJobStore(
jobs_key=redis_job_key,
run_times_key=redis_run_times_key,
host=master_host, port=master_port )
}
When the redis master node changes during the scheduler execution, I get below error.
jobstore.update_job(job)
....
redis.exceptions.ReadOnlyError: Command # 1 (HSET .....)
....
pipeline caused error: You can't write against a read replica
I could understand that the Redis instance it connected got converted to slave and not accepting writes. How can we update the scheduler's job store's redis host to point to latest master hostname?