segmentation faults following upgrade

[Graham Seaman]

Hi

I have a service which has been running on ubuntu for a couple of years without problems. It has a backend API which runs on Sinatra and uses PG to read from Postgres (1.9.1).

I just upgraded the server to Ubuntu 14.04.1. This included upgrading Ruby to 1.9.1. Following the upgrade, I now get intermittent but fairly regular segfaults (every fifteen minutes or so when the server is busy). They  appear to only be happening on attempts to access a socket which has been open for some time. The error is possibly also related to number of accesses: I have a test server with an identical setup upgraded at the same time, which has not so far shown the error (it is firewalled so only gets traffic from me).

The typical error message that kicks it off each time is (with varying non-ASCII message types, and varying, usually negative, lengths):

PG::UnableToSend - lost synchronization with server: got message type "<90>", length 2126233537

I've uploaded a copy of the log with ruby versions etc to http://pastebin.ca/2860925

I don't really have the knowledge to debug this in any kind of depth but will happily follow instructions if anyone can suggest anything to try!

Graham

[Perry Smith]

I think (but not sure) that Ruby 1.9.1 is considered yucky by the Ruby folks.  I may be getting my versions mixed up.  Is there any chance you could move to a different Ruby (or move back to the old Ruby)?

--
You received this message because you are subscribed to the Google Groups "ruby-pg" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-pg+u...@googlegroups.com.
To post to this group, send email to rub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-pg/1878b733-7567-4457-b2c3-e4ce63c4ed94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lars Kanis]

Am 19.10.2014 um 16:40 schrieb Graham Seaman:
> The typical error message that kicks it off each time is (with varying
> non-ASCII message types, and varying, usually negative, lengths):
>
> PG::UnableToSend - lost synchronization with server: got message type
> "<90>", length 2126233537

This is typical when a single connection is used by multiple threads.
Ruby-1.8 does not use native threads, so that it has less concurrency
and is therefor less prone to race conditions.

Is this the code that is crashing?
https://bitbucket.org/m25consortium/api25/src/b1947057b98d987e6b039a6bedc760176c9330f4/src/search25.rb?at=master

I'm not familiar with Sinatra, but it looks to me as that the
PG::Connection is shared between all the worker threads. This is not good!

--
Regards,
Lars

[Graham Seaman]

Hi Lars

Yes, that is the code. It has been working without complaint for so long that I'd almost forgotten it.

So, it sounds like the alternatives are to try to revert the server to Ruby 1.8, which doesn't sound easy and will likely cause problems with updating the system in future, or to rewrite the code to handle Connections on a per-thread basis. At the moment I have no idea how to do that, so I'll go off and start looking...

Thanks for the info

Graham

[Lars Kanis]

Am 19.10.2014 um 18:33 schrieb Graham Seaman:
> So, it sounds like the alternatives are to try to revert the server to
> Ruby 1.8, which doesn't sound easy and will likely cause problems with
> updating the system in future, or to rewrite the code to handle
> Connections on a per-thread basis. At the moment I have no idea how to
> do that, so I'll go off and start looking...

One connection per thread can easily be achieved by using thread local
variables ( something like Thread.current[:conn] ||= PG::Connection.new
). Maybe there is also some logic in Sinatra for these kind of issues.

--
Regards,
Lars

[Robert Klemme]

That might still hold on to the connection for too long. Usually a
connection is borrowed from a pool only for the time that it is
needed. IMHO that is a much more robust design. With Ruby's blocks
you can even implement a robust checkout and return mechanism
including automated commit or rollback.

Kind regards

robert

--
[guy, jim].each {|him| remember.him do |as, often| as.you_can - without end}
http://blog.rubybestpractices.com/

[Graham Seaman]

My ruby and my knowledge of Sinatra are both very rusty now. I've fixed it with a hack that tests the connection each time before use, and if dead creates a new one. I know it's not ideal, and may eventually run out of connections, but it's been running for a day now without problems. I can get back to it later to rewrite properly when I have more time (ha, like that's going to happen..)

Graham

[Robert Klemme]

I bet it's not.

Cheers