[CRITICAL]:-gprecoverseg failed. (Reason='Unable to connect to database and start transaction

1,987 views
Skip to first unread message

Ronald Olavidez

unread,
Oct 20, 2016, 1:15:36 AM10/20/16
to Greenplum Users
Hi Gurus,

  Need someone's help in starting up the DB. 16 primary segments are online but i can't get 16 failed mirrror segments to be online. I've done "gprecoverseg -F" but no to avail.

CURRENT STATUS:

gpstop -M fast

0161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:---------------------------------------------
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   Host             Datadir                  Port    Status
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox11   /data1/primary/gpseg0    40000   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox12   /data1/mirror/gpseg0     50000   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox11   /data1/primary/gpseg1    40001   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox12   /data1/mirror/gpseg1     50001   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox11   /data1/primary/gpseg2    40002   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox12   /data1/mirror/gpseg2     50002   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox11   /data1/primary/gpseg3    40003   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox12   /data1/mirror/gpseg3     50003   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox12   /data1/primary/gpseg4    40000   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox13   /data1/mirror/gpseg4     50000   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox12   /data1/primary/gpseg5    40001   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox13   /data1/mirror/gpseg5     50001   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox12   /data1/primary/gpseg6    40002   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox13   /data1/mirror/gpseg6     50002   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox12   /data1/primary/gpseg7    40003   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox13   /data1/mirror/gpseg7     50003   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox13   /data1/primary/gpseg8    40000   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox14   /data1/mirror/gpseg8     50000   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox13   /data1/primary/gpseg9    40001   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox14   /data1/mirror/gpseg9     50001   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox13   /data1/primary/gpseg10   40002   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox14   /data1/mirror/gpseg10    50002   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox13   /data1/primary/gpseg11   40003   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox14   /data1/mirror/gpseg11    50003   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox14   /data1/primary/gpseg12   40000   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox11   /data1/mirror/gpseg12    50000   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox14   /data1/primary/gpseg13   40001   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox11   /data1/mirror/gpseg13    50001   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox14   /data1/primary/gpseg14   40002   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox11   /data1/mirror/gpseg14    50002   d
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox14   /data1/primary/gpseg15   40003   u
20161020:13:36:06:007779 gpstop:devbox10:gpadmin-[INFO]:-   devbox11   /data1/mirror/gpseg15    50003   d



gpstart -a output

20161020:13:41:19:008218 gpstart:devbox10:gpadmin-[INFO]:-Starting gpstart with args: -a
20161020:13:41:19:008218 gpstart:devbox10:gpadmin-[INFO]:-Gathering information and validating the environment...
20161020:13:41:19:008218 gpstart:devbox10:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.6.2 build 1'
20161020:13:41:19:008218 gpstart:devbox10:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150'
20161020:13:41:19:008218 gpstart:devbox10:gpadmin-[INFO]:-Starting Master instance in admin mode
20161020:13:41:20:008218 gpstart:devbox10:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20161020:13:41:20:008218 gpstart:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161020:13:41:20:008218 gpstart:devbox10:gpadmin-[INFO]:-Setting new master era
20161020:13:41:20:008218 gpstart:devbox10:gpadmin-[INFO]:-Master Started...
20161020:13:41:20:008218 gpstart:devbox10:gpadmin-[INFO]:-Shutting down master
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox12 directory /data1/mirror/gpseg0 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox12 directory /data1/mirror/gpseg1 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox12 directory /data1/mirror/gpseg2 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox12 directory /data1/mirror/gpseg3 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox13 directory /data1/mirror/gpseg4 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox13 directory /data1/mirror/gpseg5 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox13 directory /data1/mirror/gpseg6 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox13 directory /data1/mirror/gpseg7 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox14 directory /data1/mirror/gpseg8 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox14 directory /data1/mirror/gpseg9 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox14 directory /data1/mirror/gpseg10 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox14 directory /data1/mirror/gpseg11 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox11 directory /data1/mirror/gpseg12 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox11 directory /data1/mirror/gpseg13 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox11 directory /data1/mirror/gpseg14 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox11 directory /data1/mirror/gpseg15 <<<<<
20161020:13:41:22:008218 gpstart:devbox10:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
..
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[INFO]:-Process results...
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[INFO]:-----------------------------------------------------
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[INFO]:-   Successful segment starts                                            = 16
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[INFO]:-   Failed segment starts                                                = 0
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration)   = 16   <<<<<<<<
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[INFO]:-----------------------------------------------------
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[INFO]:-
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[INFO]:-Successfully started 16 of 16 segment instances, skipped 16 other segments
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[INFO]:-----------------------------------------------------
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[WARNING]:-****************************************************************************
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[WARNING]:-There are 16 segment(s) marked down in the database
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases.
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[WARNING]:-****************************************************************************
20161020:13:41:24:008218 gpstart:devbox10:gpadmin-[INFO]:-Starting Master instance devbox10 directory /data1/master/gpseg-1
20161020:13:41:25:008218 gpstart:devbox10:gpadmin-[INFO]:-Command pg_ctl reports Master devbox10 instance active
20161020:13:41:25:008218 gpstart:devbox10:gpadmin-[INFO]:-Starting standby master
20161020:13:41:25:008218 gpstart:devbox10:gpadmin-[INFO]:-Checking if standby master is running on host: devbox11  in directory: /data1/master/gpseg-1
20161020:13:41:25:008218 gpstart:devbox10:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'CentOS release 6.7 (Final)
Kernel \r on an \m
'
20161020:13:41:26:008218 gpstart:devbox10:gpadmin-[WARNING]:-Number of segments not attempted to start: 16
20161020:13:41:26:008218 gpstart:devbox10:gpadmin-[INFO]:-Check status of database with gpstate utility



20161020:13:41:25:008218 gpstart:devbox10:gpadmin-[INFO]:-Checking if standby master is running on host: devbox11  in directory: /data1/master/gpseg-1
20161020:13:41:25:008218 gpstart:devbox10:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'CentOS release 6.7 (Final)
Kernel \r on an \m
'
20161020:13:41:26:008218 gpstart:devbox10:gpadmin-[WARNING]:-Number of segments not attempted to start: 16
20161020:13:41:26:008218 gpstart:devbox10:gpadmin-[INFO]:-Check status of database with gpstate utility
[gpadmin@devbox10 pg_distributedlog]$ gprecoverseg -F
20161020:13:46:16:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Starting gprecoverseg with args: -F
20161020:13:46:16:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.6.2 build 1'
20161020:13:46:16:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.6.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Nov 12 2015 23:50:28'
20161020:13:46:16:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161020:13:46:16:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161020:13:46:19:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 1
20161020:13:46:24:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161020:13:46:24:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161020:13:46:24:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 2
20161020:13:46:29:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161020:13:46:29:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161020:13:46:30:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 3
20161020:13:46:35:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161020:13:46:35:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161020:13:46:35:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 4
20161020:13:46:40:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161020:13:46:40:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161020:13:46:41:009150 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 5
20161020:13:46:46:009150 gprecoverseg:devbox10:gpadmin-[CRITICAL]:-gprecoverseg failed. (Reason='Unable to connect to database and start transaction') exiting...
[gpadmin@devbox10 pg_distributedlog]$

Stephen

unread,
Oct 20, 2016, 8:54:00 AM10/20/16
to Ronald Olavidez, Greenplum Users
Hi Ronald,

This error generally occurs when the change tracking logs on a primary segment have become corrupted somehow.
Running a full recovery is the correct action but unfortunately this issue prevents the gprecoverseg command from running.

To resolve...

Run "gprecoverseg -v"

This will display the verbose information on screen.

Most segments will show:
mode: PrimarySegment
segmentState: Ready
dataState: InSync
faultType: NotInitialized
mode: PrimarySegment
segmentState: Ready
dataState: InSync
faultType: NotInitialized
A segment with bad changetracking logs will show:
mode: PrimarySegment
segmentState: ChangeTrackingDisabled
dataState: InChangeTracking
faultType: NotInitialized
mode: PrimarySegment
segmentState: ChangeTrackingDisabled
dataState: InChangeTracking
faultType: NotInitialized
Once you have determined which primary segments have bad pg_changetracking logs you can:
  1. gpstop -af
  2. ssh <segment host IP>
  3. cd /path_to_data_dir/gpsegX/
  4. mkdir pg_changetracking_backup
  5. mv pg_changetracking pg_changetracking_backup
  6. gpstart -a
  7. Run gprecoverseg:
    • Full recovery on segments where you cleared the pg_changetracking
    • Incremental recovery on other segments
    • (to be sure you could just run full recovery on all segments)

Kind regards,
Stephen

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+unsubscribe@greenplum.org.
To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

Ronald Olavidez

unread,
Oct 20, 2016, 7:03:52 PM10/20/16
to Stephen, Greenplum Users
[gpadmin@devbox10 ~]$ gpstop -af
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Starting gpstop with args: -af
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Gathering information and validating the environment...
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.3.6.2 build 1'
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-There are 1 connections to the database
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='fast'
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Master host=devbox10
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Detected 1 connections to database
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Switching to WAIT mode
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Will wait for shutdown to complete, this may take some time if
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-there are a large number of active complex transactions, please wait...
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=fast
20161021:07:47:46:025663 gpstop:devbox10:gpadmin-[INFO]:-Master segment instance directory=/data1/master/gpseg-1
20161021:07:47:47:025663 gpstop:devbox10:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process
20161021:07:47:47:025663 gpstop:devbox10:gpadmin-[INFO]:-Terminating processes for segment /data1/master/gpseg-1
20161021:07:47:47:025663 gpstop:devbox10:gpadmin-[INFO]:-Stopping master standby host devbox11 mode=fast
20161021:07:47:49:025663 gpstop:devbox10:gpadmin-[INFO]:-Successfully shutdown standby process on devbox11
20161021:07:47:49:025663 gpstop:devbox10:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20161021:07:47:49:025663 gpstop:devbox10:gpadmin-[INFO]:-0.00% of jobs completed
20161021:07:47:59:025663 gpstop:devbox10:gpadmin-[INFO]:-100.00% of jobs completed
20161021:07:47:59:025663 gpstop:devbox10:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20161021:07:47:59:025663 gpstop:devbox10:gpadmin-[INFO]:-0.00% of jobs completed
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-100.00% of jobs completed
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-----------------------------------------------------
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-   Segments stopped successfully                              = 32
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-   Segments with errors during stop                           = 0
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[WARNING]:-Segments that are currently marked down in configuration   = 16   <<<<<<<<
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-            (stop was still attempted on these segments)
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-----------------------------------------------------
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-Successfully shutdown 32 of 32 segment instances
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-Cleaning up leftover gpmmon process
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-No leftover gpmmon process found
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20161021:07:48:09:025663 gpstop:devbox10:gpadmin-[INFO]:-Cleaning up leftover shared memory

Cleanup all down segments /data1/mirrorx/gpsegx/pg_changetracking/*


gpstart -a

0161021:07:57:42:026251 gpstart:devbox10:gpadmin-[INFO]:-Starting gpstart with args: -a
20161021:07:57:42:026251 gpstart:devbox10:gpadmin-[INFO]:-Gathering information and validating the environment...
20161021:07:57:42:026251 gpstart:devbox10:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.6.2 build 1'
20161021:07:57:42:026251 gpstart:devbox10:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150'
20161021:07:57:42:026251 gpstart:devbox10:gpadmin-[INFO]:-Starting Master instance in admin mode
20161021:07:57:43:026251 gpstart:devbox10:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20161021:07:57:43:026251 gpstart:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161021:07:57:43:026251 gpstart:devbox10:gpadmin-[INFO]:-Setting new master era
20161021:07:57:43:026251 gpstart:devbox10:gpadmin-[INFO]:-Master Started...
20161021:07:57:43:026251 gpstart:devbox10:gpadmin-[INFO]:-Shutting down master
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox12 directory /data1/mirror/gpseg0 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox12 directory /data1/mirror/gpseg1 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox12 directory /data1/mirror/gpseg2 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox12 directory /data1/mirror/gpseg3 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox13 directory /data1/mirror/gpseg4 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox13 directory /data1/mirror/gpseg5 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox13 directory /data1/mirror/gpseg6 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox13 directory /data1/mirror/gpseg7 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox14 directory /data1/mirror/gpseg8 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox14 directory /data1/mirror/gpseg9 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox14 directory /data1/mirror/gpseg10 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox14 directory /data1/mirror/gpseg11 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox11 directory /data1/mirror/gpseg12 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox11 directory /data1/mirror/gpseg13 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox11 directory /data1/mirror/gpseg14 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on devbox11 directory /data1/mirror/gpseg15 <<<<<
20161021:07:57:45:026251 gpstart:devbox10:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
...
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[INFO]:-Process results...
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[INFO]:-----------------------------------------------------
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[INFO]:-   Successful segment starts                                            = 16
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[INFO]:-   Failed segment starts                                                = 0
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration)   = 16   <<<<<<<<
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[INFO]:-----------------------------------------------------
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[INFO]:-
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[INFO]:-Successfully started 16 of 16 segment instances, skipped 16 other segments
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[INFO]:-----------------------------------------------------
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[WARNING]:-****************************************************************************
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[WARNING]:-There are 16 segment(s) marked down in the database
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases.
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[WARNING]:-****************************************************************************
20161021:07:57:48:026251 gpstart:devbox10:gpadmin-[INFO]:-Starting Master instance devbox10 directory /data1/master/gpseg-1
20161021:07:57:49:026251 gpstart:devbox10:gpadmin-[INFO]:-Command pg_ctl reports Master devbox10 instance active
20161021:07:57:49:026251 gpstart:devbox10:gpadmin-[INFO]:-Starting standby master
20161021:07:57:49:026251 gpstart:devbox10:gpadmin-[INFO]:-Checking if standby master is running on host: devbox11  in directory: /data1/master/gpseg-1
20161021:07:57:49:026251 gpstart:devbox10:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'CentOS release 6.7 (Final)
Kernel \r on an \m
'
20161021:07:57:50:026251 gpstart:devbox10:gpadmin-[WARNING]:-Number of segments not attempted to start: 16
20161021:07:57:50:026251 gpstart:devbox10:gpadmin-[INFO]:-Check status of database with gpstate utility

[gpadmin@devbox10 ~]$ gprecoverseg -F
20161021:07:59:21:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Starting gprecoverseg with args: -F
20161021:07:59:21:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.6.2 build 1'
20161021:07:59:21:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.6.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Nov 12 2015 23:50:28'
20161021:07:59:21:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161021:07:59:21:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161021:07:59:22:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 1
20161021:07:59:27:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161021:07:59:27:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161021:07:59:27:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 2
20161021:07:59:32:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161021:07:59:32:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161021:07:59:33:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 3
20161021:07:59:38:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161021:07:59:38:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161021:07:59:38:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 4
20161021:07:59:43:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Checking if segments are ready
20161021:07:59:43:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Obtaining Segment details from master...
20161021:07:59:44:026740 gprecoverseg:devbox10:gpadmin-[INFO]:-Unable to connect to database. Retrying 5
20161021:07:59:49:026740 gprecoverseg:devbox10:gpadmin-[CRITICAL]:-gprecoverseg failed. (Reason='Unable to connect to database and start transaction') exiting...

------------------------------------------------------------
I've removed all the files in /data1/mirrorxx/segmentxx/pg_changetracking on all the down segments but the problem of "unable to connect to database" still there.


Deepika Pandhi

unread,
Oct 20, 2016, 7:18:38 PM10/20/16
to Ronald Olavidez, Stephen, Greenplum Users
Can you check firewall with below, if it is on disable it  and then run recovery again

service iptables status

Thanks
Deepika

Liang Chen

unread,
Oct 20, 2016, 8:04:28 PM10/20/16
to Deepika Pandhi, Ronald Olavidez, Stephen, Greenplum Users
Hi Ronald,

The pg_changetracking files need to be removed from primary segments instead of mirror. In your case, you need remove pg_changetracking from gpseg[0-15] primary.

After that, reboot the db and try the full recovery again.

Thanks,
Liang Chen


To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

Ronald Olavidez

unread,
Oct 20, 2016, 8:31:09 PM10/20/16
to Liang Chen, Deepika Pandhi, Stephen, Greenplum Users
thanks..

  1. the firewall is not running, i've checked it days now.
  2. i'm trying to do move the files from the primary/segment/pg_changetracking to pg_changetracking_backup

will keep you posted if there's any progress.

Ronald Olavidez

unread,
Oct 20, 2016, 8:55:32 PM10/20/16
to Liang Chen, Deepika Pandhi, Stephen, Greenplum Users
Looking good. It's in resync mode at this moment. 

Thank you for the advise Liang, Stephen and others... i've been resolving this issue last 3 days now.

Sad to say, can't create a ticket with pivotal yet. but this is a good exercise for me.

:(



hua beatls

unread,
Jan 25, 2018, 3:30:02 PM1/25/18
to Greenplum Users, lic...@pivotal.io, dpa...@pivotal.io, stephe...@gmail.com, ronald....@gmail.com
HI, 
   Seems not work to me.
   I have the same problem, " Unable to connect to database. Retrying .." when "gprecoverseg -F",   after move pg_changetracking to pg_changetracking_backup, get more error, please see below:

      gpadmin@gp1 ~]$ gprecoverseg -F
20180125:12:08:59:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Starting gprecoverseg with args: -F
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May  9 2016 22:28:40'
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Checking if segments are ready
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Obtaining Segment details from master...
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Obtaining Segment details from master...
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Greenplum instance recovery parameters
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Recovery type              = Standard
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Recovery 1 of 4
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Synchronization mode                        = Full
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance host                        = gp2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance address                     = gp2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance directory                   = /gpdata1/mirror/gpseg0
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance port                        = 41000
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance replication port            = 42000
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance host               = gp1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance address            = gp1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance directory          = /gpdata1/primary/gpseg0
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance port               = 40000
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance replication port   = 43000
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Target                             = in-place
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Recovery 2 of 4
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Synchronization mode                        = Full
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance host                        = gp2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance address                     = gp2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance directory                   = /gpdata2/mirror/gpseg1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance port                        = 41001
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance replication port            = 42001
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance host               = gp1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance address            = gp1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance directory          = /gpdata2/primary/gpseg1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance port               = 40001
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance replication port   = 43001
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Target                             = in-place
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Recovery 3 of 4
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Synchronization mode                        = Full
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance host                        = gp1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance address                     = gp1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance directory                   = /gpdata1/mirror/gpseg2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance port                        = 41000
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance replication port            = 42000
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance host               = gp2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance address            = gp2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance directory          = /gpdata1/primary/gpseg2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance port               = 40000
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance replication port   = 43000
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Target                             = in-place
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Recovery 4 of 4
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Synchronization mode                        = Full
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance host                        = gp1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance address                     = gp1
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance directory                   = /gpdata2/mirror/gpseg3
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance port                        = 41001
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Failed instance replication port            = 42001
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance host               = gp2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance address            = gp2
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance directory          = /gpdata2/primary/gpseg3
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance port               = 40001
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Source instance replication port   = 43001
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:-   Recovery Target                             = in-place
20180125:12:09:00:025280 gprecoverseg:gp1:gpadmin-[INFO]:----------------------------------------------------------

Continue with segment recovery procedure Yy|Nn (default=N):
> y
20180125:12:09:02:025280 gprecoverseg:gp1:gpadmin-[INFO]:-4 segment(s) to recover
20180125:12:09:02:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Ensuring 4 failed segment(s) are stopped
 
20180125:12:09:03:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
20180125:12:09:09:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Cleaning files from 4 segment(s)
. 
20180125:12:09:10:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Building template directory
20180125:12:09:10:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Validating remote directories
. 
20180125:12:09:11:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Copying template directory file
. 
20180125:12:09:12:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Configuring new segments
. 
20180125:12:09:13:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Cleaning files
. 
20180125:12:09:14:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Starting file move procedure for gp2:/gpdata1/mirror/gpseg0:content=0:dbid=6:mode=r:status=d
20180125:12:09:14:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Starting file move procedure for gp2:/gpdata2/mirror/gpseg1:content=1:dbid=7:mode=r:status=d
20180125:12:09:14:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Starting file move procedure for gp1:/gpdata1/mirror/gpseg2:content=2:dbid=8:mode=s:status=d
20180125:12:09:14:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Starting file move procedure for gp1:/gpdata2/mirror/gpseg3:content=3:dbid=9:mode=s:status=d
updating flat files
20180125:12:09:14:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Updating configuration with new mirrors
20180125:12:09:15:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Updating mirrors
. 
20180125:12:09:16:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Starting mirrors
20180125:12:09:16:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
................ 
20180125:12:09:32:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Process results...
20180125:12:09:32:025280 gprecoverseg:gp1:gpadmin-[WARNING]:-Failed to start segment.  The fault prober will shortly mark it as down. Segment: gp2:/gpdata1/mirror/gpseg0:content=0:dbid=6:mode=r:status=d: REASON: PG_CTL failed.
20180125:12:09:32:025280 gprecoverseg:gp1:gpadmin-[WARNING]:-Failed to start segment.  The fault prober will shortly mark it as down. Segment: gp2:/gpdata2/mirror/gpseg1:content=1:dbid=7:mode=r:status=d: REASON: PG_CTL failed.
20180125:12:09:32:025280 gprecoverseg:gp1:gpadmin-[WARNING]:-Failed to start segment.  The fault prober will shortly mark it as down. Segment: gp1:/gpdata2/mirror/gpseg3:content=3:dbid=9:mode=r:status=d: REASON: PG_CTL failed.
20180125:12:09:32:025280 gprecoverseg:gp1:gpadmin-[WARNING]:-Failed to start segment.  The fault prober will shortly mark it as down. Segment: gp1:/gpdata1/mirror/gpseg2:content=2:dbid=8:mode=r:status=d: REASON: PG_CTL failed.
20180125:12:09:32:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Updating configuration to mark mirrors up
20180125:12:09:33:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Updating primaries
20180125:12:09:33:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Commencing parallel primary conversion of 4 segments, please wait...
...............................................................................................................................................................................................................................................................................................................................................................................................................................^[[A.^[[B............................................................................................................................................................................................... 
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Process results...
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[WARNING]:-Failed to inform primary segment of updated mirroring state.  Segment: gp2:/gpdata2/primary/gpseg3:content=3:dbid=5:mode=r:status=u: REASON: Conversion failed.  stdout:""  stderr:"failure: Error: MirroringFailure failure: Error: MirroringFailure "
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[WARNING]:-Failed to inform primary segment of updated mirroring state.  Segment: gp2:/gpdata1/primary/gpseg2:content=2:dbid=4:mode=r:status=u: REASON: Conversion failed.  stdout:""  stderr:"failure: Error: MirroringFailure failure: Error: MirroringFailure "
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[WARNING]:-Failed to inform primary segment of updated mirroring state.  Segment: gp1:/gpdata2/primary/gpseg1:content=1:dbid=3:mode=r:status=u: REASON: Conversion failed.  stdout:""  stderr:"failure: Error: MirroringFailure failure: Error: MirroringFailure "
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[WARNING]:-Failed to inform primary segment of updated mirroring state.  Segment: gp1:/gpdata1/primary/gpseg0:content=0:dbid=2:mode=r:status=u: REASON: Conversion failed.  stdout:""  stderr:"failure: Error: MirroringFailure failure: Error: MirroringFailure "
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Done updating primaries
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[INFO]:-******************************************************************
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Updating segments for resynchronization is completed.
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background.
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[INFO]:-
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[INFO]:-Use  gpstate -s  to check the resynchronization progress.
20180125:12:19:41:025280 gprecoverseg:gp1:gpadmin-[INFO]:-


   and the mirror segments all failed, please help , is there any other problem?

Thanks a lot!

bealts.


在 2016年10月21日星期五 UTC+8上午8:55:32，Ronald Olavidez写道：

Thanks,
Liang Chen



Thanks
Deepika


Run "gprecoverseg -v"

Kind regards,
Stephen

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

hua beatls

unread,
Jan 26, 2018, 12:06:38 AM1/26/18
to Greenplum Users, lic...@pivotal.io, dpa...@pivotal.io, stephe...@gmail.com, ronald....@gmail.com


在 2016年10月21日星期五 UTC+8上午8:31:09，Ronald Olavidez写道：

Thanks,
Liang Chen



Thanks
Deepika


Run "gprecoverseg -v"

Kind regards,
Stephen

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

hua beatls

unread,
Jan 26, 2018, 12:10:25 AM1/26/18
to Greenplum Users, ronald....@gmail.com


在 2016年10月20日星期四 UTC+8下午1:15:36，Ronald Olavidez写道：
Reply all
Reply to author
Forward
0 new messages