Node Rejoin not working after standby server Disconnect

21 views
Skip to first unread message

Ankit Sanghavi

unread,
Apr 29, 2021, 3:54:43 AMApr 29
to repmgr
Hi All,

I had implement replication with repmgr with 1 primary and 2 standby and 1 witness server.

 ID  | Name              | Role    | Status    | Upstream            | Location | Prio. | TLI
-----+-------------------+---------+-----------+---------------------+----------+-------+-----
 100 | WitnessDB_AFCS    | witness | * running | PrimaryDB_AFCS_DC   | default  | 0     | n/a
 301 | Standby_AFCS_HA   | standby |   running | PrimaryDB_AFCS_DC   | default  | 0     | 12
 701 | PrimaryDB_AFCS_DC | primary | * running |                     | default  | 100   | 12
 801 | Standby_AFCS_DR   | standby |   running | ! PrimaryDB_AFCS_DC | default  | 100   | 12


After disconnecting the standby server I could not rejoin the standby server.
due to Big database size not feasible to me, standby clone from scratch

/usr/pgsql-12/bin/repmgr node rejoin -f /var/lib/pgsql/repmgr.conf -d '$primary_conninfo' --force-rewind --config-files=pg_hba.conf,postgresql.conf --verbose
NOTICE: using provided configuration file "/var/lib/pgsql/repmgr.conf"
INFO: timelines are same, this server is not ahead
DETAIL: local node lsn is 1C/A005650, rejoin target lsn is 1C/16005E30
INFO: prerequisites for using pg_rewind are met
INFO: 2 files copied to "/tmp/repmgr-config-archive-Standby_AFCS_DR"
NOTICE: executing pg_rewind
DETAIL: pg_rewind command is "/usr/pgsql-12/bin/pg_rewind -D '/data/PG_DATA' --source-server=' $primary_conninfo'"
NOTICE: 2 files copied to /data/PG_DATA
INFO: directory "/tmp/repmgr-config-archive-Standby_AFCS_DR" deleted
NOTICE: setting node 801's upstream to node 701
WARNING: unable to ping "$primary_conninfo"
DETAIL: PQping() returned "PQPING_NO_RESPONSE"
NOTICE: starting server using "sudo /usr/bin/systemctl start postgresql-12.service"
INFO: node "Standby_AFCS_DR" (ID: 801) is pingable
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
INFO: waiting for node "Standby_AFCS_DR" (ID: 801) to connect to new primary; 1 of max 60 attempts (parameter "node_rejoin_timeout")
DETAIL: checking for record in node "PrimaryDB_AFCS_DC"'s "pg_stat_replication" table where "application_name" is "Standby_AFCS_DR"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
INFO: waiting for node "Standby_AFCS_DR" (ID: 801) to connect to new primary; 6 of max 60 attempts (parameter "node_rejoin_timeout")
DETAIL: checking for record in node "PrimaryDB_AFCS_DC"'s "pg_stat_replication" table where "application_name" is "Standby_AFCS_DR"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
WARNING: node "Standby_AFCS_DR" not found in "pg_stat_replication"
ERROR: NODE REJOIN failed
DETAIL: no active record for local node "Standby_AFCS_DR" found in node "PrimaryDB_AFCS_DC"'s "pg_stat_replication" table
HINT: check the PostgreSQL log on the local node

Please help us to solve this issue in an efficient way.

Ian Barwick

unread,
May 26, 2021, 7:57:09 PMMay 26
to rep...@googlegroups.com
On 29/04/2021 16:54, Ankit Sanghavi wrote:
> Hi All,

(...)
> ERROR: NODE REJOIN failed
> DETAIL: no active record for local node "Standby_AFCS_DR" found in node "PrimaryDB_AFCS_DC"'s "pg_stat_replication" table
> **HINT: check the PostgreSQL log on the local node**
>
> Please help us to solve this issue in an efficient way.

Did you check the PostgreSQL log on the local node?


Regards

Ian Barwick

--
Ian Barwick https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Reply all
Reply to author
Forward
0 new messages