Error initialize process greenplum major upgrade 5.29.12 to 6.25.2 on centos 7

[Dodi Prastyo]

hi all, we have a problem when Upgrade main greenplum version 5.29.12 to 6.25.2 on centos 7, when our initialization process has failed in substep saving_source_cluster_config. you can see the cli.log for detail process

cli.log :
[INFO]: Verifying source and target cluster versions... [SKIPPED]
[INFO]: -----------------------------------------------------------------------------
[INFO]: Saving source cluster configuration... [IN PROGRESS]
[INFO]: Executing: "/opt/greenplum-db-5.29.12/bin/postgres --gp-version"
[INFO]: connecting to source cluster with: "postgresql://localhost:5432/template1?search_path="
[INFO]: Executing: "/opt/greenplum-db-5.29.12/bin/postgres --gp-version"
[INFO]: Saving source cluster configuration... [FAILED]
[INFO]: Saving source cluster configuration... [5m1s]
[INFO]: -----------------------------------------------------------------------------
[INFO]: Initialize [5m1s]
[INFO]: substep "saving_source_cluster_config": 5m0s timeout exceeded waiting for all segments to be up, in their preferred roles, and synchronized.

We've tried to add time out in postgres.conf dan we check the status all segments already up

Environment (please complete the following information):

• OS version: centos 7
• greenplum version 5.29.12 to 6.25.2

[Kalen Krempely]

For others following along there is further context with queries to use in debugging in the Github issue that was already opened here: 

https://github.com/greenplum-db/gpupgrade/issues/901

The above error states what is going on:

> [INFO]: Initialize [5m1s]
> [INFO]: substep "saving_source_cluster_config": 5m0s timeout exceeded waiting for all segments to be up, in their preferred roles, and synchronized.

During the initialize step of gpupgrade the substep "saving_source_cluster_config" failed when the 5 minute timeout was reached waiting for all segments to be up, in their preferred roles, and synchronized. During the upgrade process especially at the start gpupgrade checks that all segments are up, in their preferred roles, and synchronized before proceeding. It does so with two queries: one for the segments and one for the standby (if there is one).

Gpupgrade executes the following query to check the segments and should return the number of segments excluding the coordinator or standby. If the number is off then some of the segments are not up, in their preferred roles, and synchronized. Inspecting gp_segment_configuration can tell you which segments.
SELECT COUNT(*) FROM gp_segment_configuration  WHERE content > -1 AND status = 'u' AND (role = preferred_role) AND mode = 's';

Gpupgrade executes this query to check if the standy is ready which should return 1 since there is only 1 standby. If it does not return 1 then inspecting pg_stat_replication can tell you what is going on.
SELECT COUNT(*) FROM pg_stat_replication WHERE state = 'streaming' AND sent_location = flush_location;

The exact code can be found here:
https://github.com/greenplum-db/gpupgrade/blob/8c78006504d306744c7518f7e2f1a4e7c97f4c27/greenplum/wait_for_segments.go#L45

These queries return quickly. However, to be extra safe gpupgrade has a very long 5 minute timeout. Increasing the timeout will very likely not solve the issue since more than likely some of the segments are not up, in their preferred roles, and synchronized. Before increasing the timeout its best to inspect gp_segment_configuration and pg_stat_replication when the issue is reproduced.

Kalen

gpupgrade team

--
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-dev+u...@greenplum.org.

This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.