Unable to bring up failed segment after gprecoverseg

[Henk Vorster]

Good day,

I have 1 failed segment, after performing a gprecoverseg (which appeared to be fine), it was still unable to start the segment, although all other segments on the same server was still up and synced.

Any idea where to start looking.

Much appreciated.

Henk

[Maciej Wawrzyniak]

Hi,

try to search ERROR/FATAL in the current log file (pg_log in the failed segment directory). 

This will explain why the segment could not start.

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.
To view this discussion on the web visit https://groups.google.com/a/greenplum.org/d/msgid/gpdb-users/081284dc-883d-4bc5-ac9c-1a0861ce4aadn%40greenplum.org.

--

Pozdrawiam/Regards,

 

Maciej Wawrzyniak

Senior Solutions Architect

tel. +48 600 050 583

Linux Polska Sp. z o.o.

Al. Jerozolimskie 100, 00-807 Warszawa

tel. +48 22 213 95 71, fax +48 22 213 96 71

KRS 00000326158, Sąd Rejonowy dla M. St. Warszawy w Warszawie, XII Wydział Gospodarczy KRS

Kapitał zakładowy 1 000 500 PLN wpłacony w całości, NIP 7010181018, REGON 141791601

 

www.linuxpolska.pl |    

_________________________________________________________________________

  

This message may contain confidential information that is covered by legal privilege. If you are not the intended recipient or if you have received this message by mistake, please notify the sender immediately and delete this e-mail and its attachments from your system. Any unauthorized copying, disclosure or distribution of the material in this e-mail and its attachments is strictly forbidden.

[Henk Vorster]

Good morning,

Here are some of the distinct messages I'm seeing in the primary and mirror logs for that segment:

Please note that all other segments are online on the same server.

2022-05-14 17:14:21.237105 CAT,"gpadmin","postgres",p36245,th-1262794624,"[local]",,2022-05-14 17:14:21 CAT,0,,,seg19,,,,,"FATAL","57P03","the database system is in recovery mode","last replayed record at 0/0",,,,,,0,,"postmaster.c",2553,

2022-05-14 17:09:45.209549 CAT,,,p32479,th-1262794624,,,,0,,,seg19,,,,,"LOG","00000","database system was interrupted while in recovery at log time 2022-05-14 02:05:34 CAT",,"If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.",,,,,0,,"xlog.c",6439,


2022-05-14 17:14:20.831201 CAT,,,p36045,th-1262794624,,,,0,,,seg19,,,,,"PANIC","58P01","could not fsync file ""base/16384/55950431.1"" (is_ao: 1): No such file or directory",,,,,,,0,,"md.c",1321,"Stack trace:
1    0xbefdbc postgres errstart (elog.c:557)
2    0xa80747 postgres mdsync (md.c:1318)
3    0xa505a8 postgres CheckPointBuffers (bufmgr.c:2008)
4    0x745837 postgres CreateRestartPoint (xlog.c:9199)
5    0xa012d0 postgres CheckpointerMain (checkpointer.c:527)
6    0x7884be postgres AuxiliaryProcessMain (bootstrap.c:443)
7    0xa0ce35 postgres <symbol not found> (postmaster.c:5832)
8    0xa0ee4c postgres <symbol not found> (postmaster.c:5502)
9    0x7f77b21b9630 libpthread.so.0 <symbol not found> + 0xb21b9630
10   0x7f77b1632983 libc.so.6 __select + 0x13
11   0x6b0e98 postgres <symbol not found> (postmaster.c:1894)
12   0xa103d2 postgres PostmasterMain (postmaster.c:1523)
13   0x6b5861 postgres main (main.c:205)
14   0x7f77b155f555 libc.so.6 __libc_start_main + 0xf5
15   0x6c14cc postgres <symbol not found> + 0x6c14cc
"


2022-05-14 17:14:20.990482 CAT,,,p32471,th-1262794624,,,,0,,,seg19,,,,,"WARNING","01000","Better set max_prepared_transactions greater than max_connections",,,,,,,0,,"cdbtm.c",1064,


2022-05-14 17:19:01.753659 CAT,,,p32471,th-1262794624,,,,0,,,seg19,,,,,"LOG","00000","checkpointer process (PID 39908) was terminated by signal 6: Aborted",,,,,,,0,,"postmaster.c",3970,
2022-05-14 17:19:01.753687 CAT,,,p32471,th-1262794624,,,,0,,,seg19,,,,,"LOG","00000","terminating any other active server processes",,,,,,,0,,"postmaster.c",3687,
2022-05-14 17:19:01.803171 CAT,"gpadmin","postgres",p40093,th-1262794624,"[local]",,2022-05-14 17:19:01 CAT,0,,,seg19,,,,,"FATAL","57P03","the database system is in recovery mode","last replayed record at 4E7B/8450EEA0",,,,,,0,,"postmaster.c",2553,
2022-05-14 17:19:01.811872 CAT,,,p32471,th-1262794624,,,,0,,,seg19,,,,,"LOG","00000","all server processes terminated; reinitializing",,,,,,,0,,"postmaster.c",4222,
2022-05-14 17:19:01.910938 CAT,,,p32471,th-1262794624,,,,0,,,seg19,,,,,"WARNING","01000","Better set max_prepared_transactions greater than max_connections",,,,,,,0,,"cdbtm.c",1064,
2022-05-14 17:19:02.217840 CAT,,,p40094,th-1262794624,,,,0,,,seg19,,,,,"LOG","00000","database system was interrupted while in recovery at log time 2022-05-14 02:05:34 CAT",,"If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.",,,,,0,,"xlog.c",6439,


2022-05-14 11:32:38.821741 CAT,"gpmon","gpperfmon",p32027,th1199687808,"10.1.1.11","52344",2022-05-14 10:23:59 CAT,0,con245034954,cmd3503,seg19,,dx83355601,,sx1,"LOG","00000","An exception was encountered during the execution of statement: INSERT INTO gpmetrics.gpcc_pg_log_history SELECT * FROM gpmetrics._gpcc_pg_log_increment WHERE logseverity IN ('WARNING', 'ERROR', 'FATAL', 'PANIC');",,,,,,,0,,,,

2022-05-14 11:33:12.376556 CAT,,,p54229,th1199687808,,,,0,,,seg19,,,,sx1,"FATAL","57P03","connections to primary segments are not allowed","This database instance is running as a primary segment in a Greenplum cluster and does not permit direct connections.","To force a connection anyway (dangerous!), use utility mode.",,,,,0,,"postinit.c",1151,

[Luis Filipe de Macedo]

Henk,

 

Which GPDB version are you using?

 

Usually when regular recovery does not work we do a full recovery. Full recovery will rewrite the whole segment for you. Take a look at the docs and see if this is an option for you.

 

Rgds,

 

Luis F R Macedo

Advisory Data Engineer & Business Development for Latam

VMware Tanzu Data

Call Me @ +55 11 98860 8596 (new)

Take care of the customers and the rest takes care of itself

 

 

De: Henk Vorster <vorst...@gmail.com>
Data: terça-feira, 17 de maio de 2022 03:37
Para: Greenplum Users <gpdb-...@greenplum.org>
Cc: maciej.w...@linuxpolska.pl <maciej.w...@linuxpolska.pl>, Greenplum Users <gpdb-...@greenplum.org>, Henk Vorster <vorst...@gmail.com>
Assunto: Re: [gpdb-users] Unable to bring up failed segment after gprecoverseg

⚠ External Email

Linux Polska Sp. z o.o.

Al. Jerozolimskie 100, 00-807 Warszawa

tel. +48 22 213 95 71, fax +48 22 213 96 71

KRS 00000326158, Sąd Rejonowy dla M. St. Warszawy w Warszawie, XII Wydział Gospodarczy KRS

Kapitał zakładowy 1 000 500 PLN wpłacony w całości, NIP 7010181018, REGON 141791601

 

www.linuxpolska.pl |    

_________________________________________________________________________

  

This message may contain confidential information that is covered by legal privilege. If you are not the intended recipient or if you have received this message by mistake, please notify the sender immediately and delete this e-mail and its attachments from your system. Any unauthorized copying, disclosure or distribution of the material in this e-mail and its attachments is strictly forbidden.

--

You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

To view this discussion on the web visit https://groups.google.com/a/greenplum.org/d/msgid/gpdb-users/2a3fa57f-8c9e-4e61-88f7-5a569a49346cn%40greenplum.org.

 

---

⚠ External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.

[Henk Vorster]

Good day,

Apologies for the late response, GPDB version is  6.11.0

I also had a look at SEG19 (primary & mirror) logs and here are the distinct warnings:

MIRROR (ONLINE/UP)

Better set max_prepared_transactions greater than max_connections

canceling MPP operation

connections to primary segments are not allowed

extra data after last expected column
ic-proxy-server: received signal 3

the database system is in recovery mode

PRIMARY (OFFLINE/DOWN)

could not fsync file "base/16384/55950431.1" (is_ao: 1): No such file or directory

Better set max_prepared_transactions greater than max_connections

Hope this helps in some way.

[Henk Vorster]

Adding an image of SEG19 status in GCC

[Luis Filipe de Macedo]

Did you try to do a full recovery? If it failed I suggest that you open a ticket with support in case you have a subscription as that usually solves the problem.

 

Also, the newer versions have many improvements on the replication mechanism, you should think in upgrading. I would wait for 6.21 release then upgrade as it will come with many improvements.

[Henk Vorster]

Thank you for the information. Yes, I tried the full recovery (gprecoverseg) a couple of times (even restarted the database and servers and tried gprecoverseg again). But that one primary just doesn't want to come up.