gpbackup failing

[Vunene Maboya]

Hi all, 

We are facing a problem where the backups performed with gpbackup fail with error:  [CRITICAL]:-ERROR: relation "pg_aoseg.pg_aocsseg_16535704" does not exist (SQLSTATE 42P01)

The gpbackup command we use: gpbackup --dbname edwdb --include-table-file /home/gpadmin/backup/scripts/get_2022_Q4_backup.txt --leaf-partition-data --jobs 32 --compression-level 9 --verbose --backup-dir /data/san/backups]

The relation mentioned in the error is not in the list of tables to be backed up.

GP environment setup:
6 node - 1 master, 1 master standby, 4 segment hosts
Greenplum version: 6.17.1
Server version: Centos 7.8

Any assistance will be appreciated.

Thanks & Regards
Vunene

[Ashwin Agrawal]

Well the reported table is meta-data table for AppendOptimized Columnar table (which should correspond to one of the tables mentioned in the list of tables to be backed-up).

Is this persistent problem or onetime occurrence?

I wish the error message was more elaborate to provide baae table name along to easily know the affected table. Given lack of it try to run SELECT count(*) on the tables listed for backup to identify which table has problem. Or even can run SELECT relid::regclass, segrelid::regclass FROM pg_appendonly to find the association between this table to CO table.

Once table is identified exclude it from list. we may dig more if required to trace the reason for problem with current info not much can be traced for reason.

-Ashwin

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.
To view this discussion on the web visit https://groups.google.com/a/greenplum.org/d/msgid/gpdb-users/a9b8806d-c958-4e75-ac55-a81298c07f74n%40greenplum.org.

[Vunene Maboya]

Thanks Ashwin. I missed your response.

This specific problem/error is not persistent but it occurs time and again a few minutes after the backup is kicked off.

Here are the full backup logs.

Start of Backup:  Sun Mar 26 00:58:10 UTC 2023
20230326:00:58:19 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Backup Command: [gpbackup --dbname edwdb --include-table-file /home/gpadmin/backup/scripts/get_2022_Q4_to_backup.txt --leaf-partition-data --jobs 32 --compression-level 9 --verbose --backup-dir /data/san/backups]
20230326:00:58:19 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-gpbackup version = 1.20.4
20230326:00:58:21 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Greenplum Database Version = 6.17.1 build commit:3633e368b674f35eb9609717eed42569ab138858
20230326:00:58:21 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Starting backup of database edwdb
20230326:00:58:21 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Validating Tables and Schemas exist in Database
20230326:00:59:41 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Creating backup directories
20230326:00:59:41 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Backup Timestamp = 20230326005819
20230326:00:59:41 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Backup Database = edwdb
20230326:00:59:41 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Backup Parameters: {compression: gzip, plugin executable: None, backup section: All Sections, object filtering: Include Table Filter, includes statistics: No, data file format: Multiple Data Files Per Segment, incremental: False}
20230326:00:59:41 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Gathering table state information
20230326:01:00:23 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Acquiring ACCESS SHARE locks on tables
20230326:01:01:11 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Gathering additional table metadata
20230326:01:01:11 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Getting column definitions
20230326:01:01:22 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Getting distribution policies
20230326:01:01:24 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Getting partition definitions
20230326:01:01:55 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Getting storage information
20230326:01:02:02 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Retrieving external table information
20230326:01:02:05 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-Getting child partitions with altered schema
20230326:01:02:06 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Constructing table definition map
20230326:01:02:06 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Querying table row mod counts
20230326:01:02:15 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[CRITICAL]:-ERROR: relation "pg_aoseg.pg_aocsseg_16535704" does not exist (SQLSTATE 42P01)
github.com/greenplum-db/gp-common-go-libs/gplog.FatalOnError
/tmp/build/3e49593f/go/pkg/mod/github.com/greenplum-db/gp-commo...@v1.0.5-0.20201005232358-ee3f0135881b/gplog/gplog.go:310
github.com/greenplum-db/gpbackup/backup.getModCount
/tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/backup/queries_incremental.go:83
github.com/greenplum-db/gpbackup/backup.getAllModCounts
/tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/backup/queries_incremental.go:31
github.com/greenplum-db/gpbackup/backup.GetAOIncrementalMetadata
/tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/backup/queries_incremental.go:13
github.com/greenplum-db/gpbackup/backup.backupIncrementalMetadata
/tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/backup/wrappers.go:660
github.com/greenplum-db/gpbackup/backup.DoBackup
/tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/backup/backup.go:123
main.main.func1
/tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/gpbackup.go:23
github.com/spf13/cobra.(*Command).execute
/tmp/build/3e49593f/go/pkg/mod/github.com/spf13/co...@v0.0.5/command.go:830
github.com/spf13/cobra.(*Command).ExecuteC
/tmp/build/3e49593f/go/pkg/mod/github.com/spf13/co...@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/tmp/build/3e49593f/go/pkg/mod/github.com/spf13/co...@v0.0.5/command.go:864
main.main
/tmp/build/3e49593f/go/src/github.com/greenplum-db/gpbackup/gpbackup.go:27
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
20230326:01:02:17 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[INFO]:-/home/gpadmin/gp_email_contacts.yaml list found, /data/san/backups/gpseg-1/backups/20230326/20230326005819/gpbackup_20230326005819_report will be sent
20230326:01:02:17 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[WARNING]:-Unable to send email report: Error reading email contacts file.
20230326:01:02:17 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[WARNING]:-Please ensure that the email contacts file is in valid YAML format.
20230326:01:02:17 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Beginning cleanup
20230326:01:02:17 gpbackup:gpadmin:mdw-01.dwh.gocom.za:022697-[DEBUG]:-Cleanup complete
End of Backup:  Sun Mar 26 01:02:17 UTC 2023

In this case, there are about 20000 tables selected for backup, and about half of the tables are Column-oriented. I cannot identify the base table associated with the metadata table referenced in the logs as it also does not exist. I ran this query SELECT relid::regclass, segrelid::regclass FROM pg_appendonly where segrelid = 'pg_aoseg.pg_aocsseg_16535704'::regclass;

[Ashwin Agrawal]

On Tue, Apr 11, 2023 at 6:23 AM Vunene Maboya <vune...@gmail.com> wrote:

In this case, there are about 20000 tables selected for backup, and about half of the tables are Column-oriented. I cannot identify the base table associated with the metadata table referenced in the logs as it also does not exist. I ran this query SELECT relid::regclass, segrelid::regclass FROM pg_appendonly where segrelid = 'pg_aoseg.pg_aocsseg_16535704'::regclass;

Can you give try using the support/developer/troubleshooting GUC gp_select_invisible and see if can trace the table

set gp_select_invisible to on; SELECT relid::regclass, segrelid::regclass FROM pg_appendonly where segrelid = 'pg_aoseg.pg_aocsseg_16535704'::regclass;

Reads like some race condition is happening. I will try to look from a backup code perspective also if I can hunt down the problem.

--

Ashwin Agrawal (VMware)

[Vunene Maboya]

Hi Ashwin,

I have set gp_select_invisible to on and ran the select query as advised but I am getting the relation does not exist error.

ERROR:  relation "pg_aoseg.pg_aocsseg_16535704" does not exist

[Ashwin Agrawal]

On Wed, Apr 12, 2023 at 3:37 AM Vunene Maboya <vune...@gmail.com> wrote:

Hi Ashwin,

I have set gp_select_invisible to on and ran the select query as advised but I am getting the relation does not exist error.

ERROR:  relation "pg_aoseg.pg_aocsseg_16535704" does not exist

Opps my bad, had tried the query on a visible table and not dropped the table (regclass is going to fail after dropping the table). Here is modified query. Let's see if this can help to locate the table....

set gp_select_invisible to on; select oid,relname,relnamespace from pg_class, pg_appendonly where oid = relid and segrelid in (select oid from pg_class where relname='pg_aocsseg_16535704'); set gp_select_invisible to off;

--

Ashwin Agrawal (VMware)

[Jimmy Yih]

You can also try running the actual metadata query that gpbackup runs to help debug the issue as well:

SELECT seg.aotablefqn,
        'pg_aoseg.' || quote_ident(aoseg_c.relname) AS aosegtablefqn
FROM pg_class aoseg_c
        JOIN (SELECT pg_ao.relid AS aooid,
                        pg_ao.segrelid,
                        aotables.aotablefqn
                FROM pg_appendonly pg_ao
                        JOIN (SELECT c.oid,
                                        quote_ident(n.nspname)|| '.' || quote_ident(c.relname) AS aotablefqn
                                FROM pg_class c
                                        JOIN pg_namespace n ON c.relnamespace = n.oid
                                WHERE relstorage IN ( 'ao', 'co' )
                        ) aotables ON pg_ao.relid = aotables.oid
        ) seg ON aoseg_c.oid = seg.segrelid
WHERE aoseg_c.relname = 'pg_aocsseg_16535704';

- Jimmy

[Vunene Maboya]

Thank you Ashwin and Jimmy.

I have run both queries as suggested, and they did not return any rows, nor were any error messages raised.