Postgres Connection Issues causing workers to freeze

[Nick DeSteffen]

We recently upgraded to Ruby 2.2.2 (from 2.0-p481) and have started getting strange Postgres connection errors periodically.  These errors cause our Sidekiq workers to just hang and not finish their jobs or pickup any new ones.  

Some of the errors we have encountered are:

• PG::UnableToSend: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.
  
• PG::ConnectionBad: PQsocket() can't get socket descriptor
  
• PG::ConnectionBad: ERROR: no more connections allowed (max_client_conn)
  
• PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

We are running Postgres 9.3.6, Sidekiq 3.4.1 (concurrency set to 32 in production), Ruby 2.2.2, Rails 4.1.12, pg gem (0.18.2).  We had sidekiq-limit_fetch (2.4.2) running but removed that in case it was the root of the problem, but it isn't.  Our application is connecting to the database via PgBouncer, however we had these problems prior to connecting through PgBouncer. 

Restarting Sidekiq causes the jobs to clear out and finish.

Any help would be appreciated.

Regards,

Nick

[Mike Perham]

That looks like a connection corruption error that people have reported before.  I suspect the problem is somewhere in MRI/openssl/pg because no one with mysql reports the same issue.  Perhaps it's a threading issue; you can bump concurrency down to 1 and see if the problem disappears too.  Sadly I don't have any idea how to solve it.

Mike

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/bda7c3d2-ae03-4bc5-a4ae-35e3a7bd9203%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mike Perham – CEO, Contributed Systems

Smart, effective open source infrastructure for your apps.

[Nick DeSteffen]

Mike,

Just an update here, we configured activerecord's reaper in our application and the exceptions seem to have subsided.  So far we don't see any side effects.  If anybody else has this problem they can try adding the configuration setting to their database.yml

reaping_frequency: 10

10 being the number of seconds between each reaper run.

Thanks,

Nick

[Mike Perham]

Nick, that's awesome info.  I will add that to the troubleshooting wiki page, thanks for following up.

To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/dc04a63c-e2a9-46bb-a243-6358b1b7fc01%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.