Segment fails to start

[Vunene Maboya]

Good day,

I hope someone can help us with this issue that we are currently having.

Environment: 1 master, 1 master standby, 8 segment hosts
Greenplum version: 6.11
Segments: 32 primary & 32 mirror
Server version: RHEL 7.6
Problematic segment: Segment 19

Problem:
We are facing an issue in which 1 of the mirror segments fails to start even after performing a full segment recovery (gprecoverseg -F).

Background:
Over the past 3 years, we have experienced 3 or 4 segment failures, and it has always been the same segment (SEG19) that fails and its mirror assumes the primary role. We have always been able to recover the failed segment and return it to its primary role.

Current issue:
The primary problematic segment has failed again and its mirror assumed the primary role. We performed a full segment recovery which seems to have run to 100% but then failed to start. Please see the logs observed in the primary segment 19 logs.

We attempted another full segment recovery and segment rebalance which eventually ran well and managed to bring up the primary segment and return the segments to their preferred roles but now the mirror segment is failing to start and below are the log entries we have observed in the mirror segment logs.

The "FATAL","57P03","the database system is starting up","last replayed record at 0/0",,,,,,0,,"postmaster.c",2539 fatal errors were observed in the segment logs until the database was restarted.