Catalog is missing attributes.

[Vishal Patel]

We are getting this error trying to read/delete from a table:

ERROR:  catalog is missing 10 attribute(s) for relid 15890665 (relcache.c:1012)  (seg61 slice1 gpseg1:40001 pid=30490) (cdbdisp.c:1320)

We had a long running vacuum command that was killed producing this error:

WARNING:  canceling wait for synchronous replication due to user request

DETAIL:  The transaction has already committed locally, but might not have been replicated to the standby.

WARNING:  The distributed transaction 'Abort [Prepared]' broadcast failed to one or more segments for gid = 1492614208-0002631738.

NOTICE:  Releasing segworker groups to retry broadcast.

NOTICE:  Retry of the distributed transaction 'Abort Prepared' broadcast succeeded to the segments for gid = 1492614208-0002631738.

ERROR:  The distributed transaction 'Prepare' broadcast failed to one or more segments for gid = 1492614208-0002631738. (cdbtm.c:689)

We believe this error seems to have possibly caused this. Is there a way to force drop that table or clean up the catalog? 

I am currently running gpcheckcat.

Thank you. 

[Vishal Patel]

Just an update, my approach to recover was to stop that segment where the catalog error was coming from. Deleted the table successfully after that and recovered the segment. I'm still curious to hear from others if they have ever experienced something like this. 

I also realized there isn't an easy way to stop a single segment. I had to kill the pid or reboot the server. Maybe there is a better way to do this? 

[Asim Praveen]

Hi Vishal,

What version of Greenplum are you running?

On Sat, Jun 24, 2017 at 11:46 AM, Vishal Patel <vishp...@gmail.com> wrote:
>

> We had a long running vacuum command that was killed producing this error:
>
> WARNING:  canceling wait for synchronous replication due to user request
> DETAIL:  The transaction has already committed locally, but might not have been replicated to the standby.

Can you please look for details for this error in mater logs and share what you find?

>
> WARNING:  The distributed transaction 'Abort [Prepared]' broadcast failed to one or more segments for gid = 1492614208-0002631738.
> NOTICE:  Releasing segworker groups to retry broadcast.
> NOTICE:  Retry of the distributed transaction 'Abort Prepared' broadcast succeeded to the segments for gid = 1492614208-0002631738.
> ERROR:  The distributed transaction 'Prepare' broadcast failed to one or more segments for gid = 1492614208-0002631738. (cdbtm.c:689)
>

This error should have details in master as well as segment logs,  Can you please share them?  Grep master and segment logs for "dx2631738".  This is the last part of "gid" from the error message.

One way to stop a specific segment is to run "pg_ctl -D <path-to-segment-data-directory> stop -m immediate".  The command must be run from the host where the segment is running. 

Asim

[Vishal Patel]

Hi Asim,

Thanks for the stop segment command, that makes sense. 

We are running greenplum-db-4.3.11.1.

I couldn't find dx26317 in any segment logs, however here is the information from the master:

https://gist.github.com/vishalrp/2beefcfbd0df9f7dcd87865b832370ac

Thanks,

Vishal

[Jon Roberts]

Try "drop table if exists <tablename>"

Jon Roberts

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+unsubscribe@greenplum.org.
To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.