Intermittent Error--> FATAL: (08006) connection to client lost

[suraj birla]

For past 3 weeks we are having issue that client via ODBC driver is not able to connect to GPDB database .. this happen only 4-5 times a day and afterward we are able to connect. each occurrence last for 2-3 minutes.

ODBC driver : DataDirect   

Version 7.1

from GPDB below is the alert message.

ssrsappuser logged on from host 192.X.Y.Z connected to database lytxbi

FATAL: (08006) connection to client lost

Also here is the log from master.  Since we are receiving alert, that's mean client was able to connect to GPDB but GPDB didn't respond back  to the client in that 30 second, client is getting disconnected and  receive an error message

[DataDirect][ODB Greenplum Wire Protocal driver] timeout expired.

even though the logon timeout setting is set to 300 second, the timeout error is received after 30 second.. that's driver issue .

My question is why does GPDB takes more than 30 second to respond back to client...And this happens Intermittent...  we never faced this issue for past 3 years .. what might cause this issue?

We do Vacuum full on system catalog table every weekend.

Reindex system table.

Analyzedb -s pg_catalog

I'm seeing that the pg_attribute table size has grown to 2 GB  ..  Sometime I do vacuum full on PG_attribute table on weekdays and then the next day we don;t see any 'connection to client lost' error... but after a day we start seeing this error.

CPU utilization at the time of error occurrence is  less than 10 %. No batch job or heavy query running at that time.

has anyone faced this issue?

>>>> Log from master

2017-12-06 22:24:52.873558 UTC,"ssrsappuser","lytxbi",p236722,th-1565403360,"172.x.x.x","59869",2017-12-06 22:23:52 UTC,0,con4352697,,seg-1,,,,,"LOG","08006","could not send data to client: Connection reset by peer",,,,,,,0,,"pqcomm.c",1432,

2017-12-06 22:24:52.873592 UTC,"ssrsappuser","lytxbi",p236722,th-1565403360,"172.x.x.x","59869",2017-12-06 22:23:52 UTC,0,con4352697,,seg-1,,,,,"FATAL","08006","connection to client lost",,,,,,,0,,"postgres.c",3599,

2017-12-06 22:25:00.723519 UTC,"gpmon","gpperfmon",p237109,th-1565403360,"[local]",,2017-12-06 22:25:00 UTC,131198695,con4352732,cmd1,seg-1,,dx6563615,x131198695,sx1,"LOG","00000","statement: SELECT sess_id, current_query FROM pg_stat_activity;",,,,,,"SELECT sess_id, current_query FROM pg_stat_activity;",0,,"postgres.c",1619,

2017-12-06 22:25:06.152856 UTC,"ssrsappuser","lytxbi",p236752,th-1565403360,"172.x.x.x","59934",2017-12-06 22:24:06 UTC,0,con4352699,,seg-1,,,,,"LOG","08006","could not send data to client: Connection reset by peer",,,,,,,0,,"pqcomm.c",1432,

2017-12-06 22:25:06.152928 UTC,"ssrsappuser","lytxbi",p236752,th-1565403360,"172.x.x.x","59934",2017-12-06 22:24:06 UTC,0,con4352699,,seg-1,,,,,"FATAL","08006","connection to client lost",,,,,,,0,,"postgres.c",3599,

2017-12-06 22:25:07.887214 UTC,"ssrsappuser","lytxbi",p236754,th-1565403360,"172.x.x.x","59935",2017-12-06 22:24:07 UTC,0,con4352700,,seg-1,,,,,"LOG","08006","could not send data to client: Connection reset by peer",,,,,,,0,,"pqcomm.c",1432,

2017-12-06 22:25:07.887317 UTC,"ssrsappuser","lytxbi",p236754,th-1565403360,"172.x.x.x","59935",2017-12-06 22:24:07 UTC,0,con4352700,,seg-1,,,,,"FATAL","08006","connection to client lost",,,,,,,0,,"postgres.c",3599,

2017-12-06 22:25:10.813814 UTC,"gpadmin","postgres",p237130,th-1565403360,"172.x.y.Z","24129",2017-12-06 22:25:10 UTC,131198698,con4352734,cmd1,seg-1,,dx6563616,x131198698,sx1,"LOG","00000","statement: select * from gp_segment_configuration",,,,,,"select * from gp_segment_configuration",0,,"postgres.c",1619,

<<<<

Thanks

Suraj

[Ming Li]

From the log, it seems that master also cannot connect to client. So maybe some network issue occurs during this time. Can you check the Router or DNS working status? Or maybe you can write some scripts to ping some other machine in the same network.

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+unsubscribe@greenplum.org.
To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

[Danilo Fortunato]

I would also investigate on the client side.
I don't know which are your clients, but query and reporting tools (Cognos BI, MicroStrategy, QlikView, etc.) generally speaking have several processes that connect in parallel to the source database.
If one of these processes crashes or hangs, then you could have an issue like yours.
If this is the case, you should find a correspondence in the log files of the client.

Regards,
Danilo Fortunato

---

Da: Ming Li (m...@pivotal.io)
Inviato: Venerdì 8 Dicembre 2017 04:38
A: Suraj Birla (suraj...@gmail.com)
Cc: Greenplum Users (gpdb-...@greenplum.org)
Oggetto: Re: [gpdb-users] Intermittent Error--> FATAL: (08006) connection to client lost

 

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

[Luis Macedo]

Looks like network to me...

--- Sent from my Google Pixel