Orchestrator node issue

[Dipesh Acharya]

Hi we have 3 orchestrator nodes setup. it has been working fine for year. Suddenly we are encountering errors. Not sure what is wrong but the configuration/ firewall and connections all works fine to backend and mysql nodes

 Any help will be appreciated.

Here is the error log :-

 6 11:38:39pl01 orchestrator: 2021-12-06 11:38:39 DEBUG raft leader is 11.530.111.113:10008; state: Follower

Dec  6 11:38:39 pl01 orchestrator: [martini] Started GET /api/leader-check for 10.137.111.169:56745

Dec  6 11:38:39  pl01 orchestrator: [martini] Completed 404 Not Found in 435.066µs

Dec  6 11:38:39 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.131:43303

Dec  6 11:38:39 pl01 orchestrator: [martini] Completed 404 Not Found in 526.394µs

Dec  6 11:38:40 pl01 orchestrator: 2021-12-06 11:38:40 ERROR Heath test is failing for over 5 seconds. raft yielding

Dec  6 11:38:40 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.131:43325

Dec  6 11:38:40 pl01 orchestrator: [martini] Completed 404 Not Found in 525.625µs

Dec  6 11:38:40 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.169:56781

Dec  6 11:38:40 pl01 orchestrator: [martini] Completed 404 Not Found in 379.643µs

Dec  6 11:38:41 pl01 orchestrator: 2021-12-06 11:38:41 ERROR Heath test is failing for over 5 seconds. raft yielding

Dec  6 11:38:41 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.170:59870

Dec  6 11:38:41 pl01 orchestrator: [martini] Completed 404 Not Found in 490.34µs

Dec  6 11:38:41pl01 orchestrator: [martini] Started GET /api/leader-check for 11.337.00.170:41599

Dec  6 11:38:41 pl01 orchestrator: [martini] Completed 404 Not Found in 615.029µs

Dec  6 11:38:41 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.168:60231

Dec  6 11:38:41pl01 orchestrator: [martini] Completed 404 Not Found in 562.267µs

Dec  6 11:38:41 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.170:59898

Dec  6 11:38:41pl01 orchestrator: [martini] Completed 404 Not Found in 558.492µs

Dec  6 11:38:42 pl01 orchestrator: 2021-12-06 11:38:42 ERROR Heath test is failing for over 5 seconds. raft yielding

Dec  6 11:38:42 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.337.00.170:41615

Dec  6 11:38:42 pl01 orchestrator: [martini] Completed 404 Not Found in 379.921µs

Dec  6 11:38:42 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.168:60255

Dec  6 11:38:42 pl01 orchestrator: [martini] Completed 404 Not Found in 462.882µs

Dec  6 11:38:42 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.337.00.169:51060

Dec  6 11:38:42pl01 orchestrator: [martini] Completed 404 Not Found in 722.2µs

Dec  6 11:38:42 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.337.00.168:37900

Dec  6 11:38:42 pl01 orchestrator: [martini] Completed 404 Not Found in 498.131µs

Dec  6 11:38:43 pl01 orchestrator: 2021-12-06 11:38:43 ERROR Heath test is failing for over 5 seconds. raft yielding

Dec  6 11:38:43 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.337.00.169:51076

Dec  6 11:38:43 pl01 orchestrator: [martini] Completed 404 Not Found in 464.895µs

Dec  6 11:38:43 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.337.00.168:37916

Dec  6 11:38:43 pl01 orchestrator: [martini] Completed 404 Not Found in 387.678µs

Dec  6 11:38:44 pl01 orchestrator: 2021-12-06 11:38:44 ERROR Heath test is failing for over 5 seconds. raft yielding

Dec  6 11:38:44 pl01 orchestrator: 2021-12-06 11:38:44 DEBUG raft leader is 11.530.111.113:10008; state: Follower

Dec  6 11:38:44pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.169:56853

Dec  6 11:38:44 pl01 orchestrator: [martini] Completed 404 Not Found in 1.45892ms

Dec  6 11:38:45pl01 orchestrator: 2021-12-06 11:38:45 ERROR Heath test is failing for over 5 seconds. raft yielding

Dec  6 11:38:45 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.131:43371

Dec  6 11:38:45pl01 orchestrator: [martini] Completed 404 Not Found in 435.234µs

Dec  6 11:38:45 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.169:56881

Dec  6 11:38:45 pl01 orchestrator: [martini] Completed 404 Not Found in 411.118µs

Dec  6 11:38:45 pl01 orchestrator: [martini] Started GET /api/leader-check for 11.122.87.131:43393

Dec  6 11:38:45  pl01 orchestrator: [martini] Completed 404 Not Found in 574.359µs

Dec  6 11:38:46 pl01 orchestrator: 2021-12-06 11:38:46 ERROR Heath test is failing for over 5 seconds. raft yielding

Dec  6 11:38:46  pl01 orchestrator: 2021-12-06 11:38:46 FATAL Node is unable to register health. Please check database connnectivity.

Dec  6 11:41:16 pl01 systemd: orchestrator.service: main process exited, code=exited, status=1/FAILURE