Client hanging on idle connection

[jac...@gmail.com]

We are having a somewhat common occurrence where the ruby process will hang for days (indefinitely). I have been trying to debug it using strace and gdb but my skills are limited. Here is an example of what the gdb trace looks like for one of the processes:

0x00007fb507606653 in select () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) backtrace

#0  0x00007fb507606653 in select () from /lib/x86_64-linux-gnu/libc.so.6

#1  0x00007fb507a99129 in native_fd_select (timeout=0x0, exceptfds=0x0, 

    writefds=0x0, readfds=0x7fff32a84000, n=32, th=<optimized out>)

    at thread_pthread.c:1044

#2  do_select (timeout=0x0, except=0x0, write=0x0, read=0x7fff32a84000, n=32)

    at thread.c:3455

#3  rb_thread_fd_select (max=32, read=0x7fff32a84000, write=0x0, except=0x0, 

    timeout=<optimized out>) at thread.c:3601

#4  0x00007fb503a2ca9a in wait_socket_readable (conn=0x1c03e4f0, ptimeout=0x0, 

    is_readable=0x7fb503a2cb50 <get_result_readable>) at pg_connection.c:2347

#5  0x00007fb503a315ab in pgconn_block (self=409897440, argv=0x0, argc=0)

    at pg_connection.c:2973

#6  pgconn_async_exec (argc=<optimized out>, argv=<optimized out>, 

    self=409897440) at pg_connection.c:3043

I am using version 0.17.1, so this is the line where it is stuck:

https://github.com/ged/ruby-pg/blob/6ba1d233449fb7918a27c6d160936ff73c8e01cf/ext/pg_connection.c#L2973

When I use strace on the process it shows that is hanging out on the system call:

select(32, [31], NULL, NULL, NULL

When I select from pg_stat_activity on the postgres database, it looks like this:

{"datid"=>"16385",

  "datname"=>"<redacted>",

  "pid"=>"26377",

  "usesysid"=>"16387",

  "usename"=>"<redacted>",

  "application_name"=>

   "resque-1.25.2: Processing low ...0967 [FullContactUpdateSignup]",

  "client_addr"=>"<redacted>",

  "client_hostname"=>nil,

  "client_port"=>"45497",

  "backend_start"=>"2015-12-30 04:50:14.7293+00",

  "xact_start"=>nil,

  "query_start"=>"2015-12-30 04:50:29.98868+00",

  "state_change"=>"2015-12-30 04:50:29.99737+00",

  "waiting"=>"f",

  "state"=>"idle",

  "query"=>

   "EXPLAIN             SELECT a.attname, format_type(a.atttypid, a.atttypmod),\n                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod\n              FROM pg_attribute a LEFT JOIN pg_attrdef d\n                ON a.attrelid = d.adrelid AND a.attnum = d.adnum\n             WHERE a.attrelid = '\"activity_datas\"'::regclass\n               AND a.attnum > 0 AND NOT a.attisdropped\n             ORDER BY a.attnum\n"}

The client is running on a Linux machine:

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=12.04

DISTRIB_CODENAME=precise

DISTRIB_DESCRIPTION="Ubuntu 12.04.4 LTS"

NAME="Ubuntu"

VERSION="12.04.4 LTS, Precise Pangolin"

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME="Ubuntu precise (12.04.4 LTS)"

VERSION_ID="12.04"

We are running this version of Ruby:

ruby 2.1.6p336 (2015-04-13 revision 50298) [x86_64-linux]

Any guidance would be appreciated.

Thanks,

Jacob