On Monday, October 22, 2012 11:47:29 AM UTC-7, Rodrigo Rosenfeld Rosas wrote:
> I had to restart the PostgreSQL service in our database server and the > Rails apps using Sequel stopped working.
> Is there a way to instruct Sequel to try to reconnect on such cases? One > that works both for MRI and JRuby?
Sequel should be removing the connections from the pool, reraising the exceptions, and creating new connections as needed in the future. So it's expected that some DatabaseDisconnectErrors will be raised, but Sequel should recover, assuming that it correctly recognizes the disconnections (which it should on the postgres and jdbc/postgres adapters). If that isn't what you experienced, please try to put together a reproducible test case and I'll take a look.
> On Monday, October 22, 2012 11:47:29 AM UTC-7, Rodrigo Rosenfeld Rosas > wrote:
>> I had to restart the PostgreSQL service in our database server and the >> Rails apps using Sequel stopped working.
>> Is there a way to instruct Sequel to try to reconnect on such cases? One >> that works both for MRI and JRuby?
> Sequel should be removing the connections from the pool, reraising the > exceptions, and creating new connections as needed in the future. So it's > expected that some DatabaseDisconnectErrors will be raised, but Sequel > should recover, assuming that it correctly recognizes the disconnections > (which it should on the postgres and jdbc/postgres adapters). If that > isn't what you experienced, please try to put together a reproducible test > case and I'll take a look.
But this is a problem for our setup as we have a cluster with a dozen servers and there will be as many reported errors as there are servers in the cluster and all of them end up in our SLA errors log which affects our contract :(
Is there a way of just instructing Sequel to query something like "select 1" before attempting to use the connection? Hibernate has an option to do so and it works. Our application is a mix of a Grails and a Rails app. We only had to restart the Rails servers of the cluster as the Grails ones that used Hibernate with this option enabled didn't experience any issues.
We'd like to be able to do something like with Sequel as well.
On Monday, October 22, 2012 6:20:41 PM UTC-7, Rodrigo Rosenfeld Rosas wrote:
> Em segunda-feira, 22 de outubro de 2012 19h34min26s UTC-2, Jeremy Evans > escreveu:
>> On Monday, October 22, 2012 11:47:29 AM UTC-7, Rodrigo Rosenfeld Rosas >> wrote:
>>> I had to restart the PostgreSQL service in our database server and the >>> Rails apps using Sequel stopped working.
>>> Is there a way to instruct Sequel to try to reconnect on such cases? One >>> that works both for MRI and JRuby?
>> Sequel should be removing the connections from the pool, reraising the >> exceptions, and creating new connections as needed in the future. So it's >> expected that some DatabaseDisconnectErrors will be raised, but Sequel >> should recover, assuming that it correctly recognizes the disconnections >> (which it should on the postgres and jdbc/postgres adapters). If that >> isn't what you experienced, please try to put together a reproducible test >> case and I'll take a look.
> But this is a problem for our setup as we have a cluster with a dozen > servers and there will be as many reported errors as there are servers in > the cluster and all of them end up in our SLA errors log which affects our > contract :(
> Is there a way of just instructing Sequel to query something like "select > 1" before attempting to use the connection? Hibernate has an option to do > so and it works. Our application is a mix of a Grails and a Rails app. We > only had to restart the Rails servers of the cluster as the Grails ones > that used Hibernate with this option enabled didn't experience any issues.
> We'd like to be able to do something like with Sequel as well.
> Would that be possible?
It's actually something I'm planning to implement as a extension very soon. The recently committed refactoring of the database <-> connection pool interface is the first step, I need to add code to check connections for validity and then an extension that automatically checks connections for validity when they are checked out from the pool.
Note that restarting your app servers whenever you restart your database server has been the recommended way for a long time. The extension I'm planning to implement is mostly to handle inactive connections in the pool that get timed out and disconnected by the database. If you have control over when your database server restarts, it shouldn't be difficult to make sure your app servers are restarted at the same time.
> Em segunda-feira, 22 de outubro de 2012 19h34min26s UTC-2, Jeremy Evans > escreveu:
>> On Monday, October 22, 2012 11:47:29 AM UTC-7, Rodrigo Rosenfeld Rosas >> wrote:
>>> I had to restart the PostgreSQL service in our database server and the >>> Rails apps using Sequel stopped working.
>>> Is there a way to instruct Sequel to try to reconnect on such cases? One >>> that works both for MRI and JRuby?
>> Sequel should be removing the connections from the pool, reraising the >> exceptions, and creating new connections as needed in the future. So it's >> expected that some DatabaseDisconnectErrors will be raised, but Sequel >> should recover, assuming that it correctly recognizes the disconnections >> (which it should on the postgres and jdbc/postgres adapters). If that >> isn't what you experienced, please try to put together a reproducible test >> case and I'll take a look.
> But this is a problem for our setup as we have a cluster with a dozen > servers and there will be as many reported errors as there are servers in > the cluster and all of them end up in our SLA errors log which affects our > contract :(
> Is there a way of just instructing Sequel to query something like "select > 1" before attempting to use the connection? Hibernate has an option to do > so and it works. Our application is a mix of a Grails and a Rails app. We > only had to restart the Rails servers of the cluster as the Grails ones > that used Hibernate with this option enabled didn't experience any issues.
> We'd like to be able to do something like with Sequel as well.
By the way, Java libraries usually refer to this feature as "test on borrow":
> On Monday, October 22, 2012 6:20:41 PM UTC-7, Rodrigo Rosenfeld Rosas > wrote:
> Em segunda-feira, 22 de outubro de 2012 19h34min26s UTC-2, Jeremy
> Evans escreveu:
> On Monday, October 22, 2012 11:47:29 AM UTC-7, Rodrigo
> Rosenfeld Rosas wrote:
> I had to restart the PostgreSQL service in our database
> server and the
> Rails apps using Sequel stopped working.
> Is there a way to instruct Sequel to try to reconnect on
> such cases? One
> that works both for MRI and JRuby?
> Sequel should be removing the connections from the pool,
> reraising the exceptions, and creating new connections as
> needed in the future. So it's expected that some
> DatabaseDisconnectErrors will be raised, but Sequel should
> recover, assuming that it correctly recognizes the
> disconnections (which it should on the postgres and
> jdbc/postgres adapters). If that isn't what you experienced,
> please try to put together a reproducible test case and I'll
> take a look.
> But this is a problem for our setup as we have a cluster with a
> dozen servers and there will be as many reported errors as there
> are servers in the cluster and all of them end up in our SLA
> errors log which affects our contract :(
> Is there a way of just instructing Sequel to query something like
> "select 1" before attempting to use the connection? Hibernate has
> an option to do so and it works. Our application is a mix of a
> Grails and a Rails app. We only had to restart the Rails servers
> of the cluster as the Grails ones that used Hibernate with this
> option enabled didn't experience any issues.
> We'd like to be able to do something like with Sequel as well.
> Would that be possible?
> It's actually something I'm planning to implement as a extension very > soon. The recently committed refactoring of the database <-> > connection pool interface is the first step, I need to add code to > check connections for validity and then an extension that > automatically checks connections for validity when they are checked > out from the pool.
> Note that restarting your app servers whenever you restart your > database server has been the recommended way for a long time. The > extension I'm planning to implement is mostly to handle inactive > connections in the pool that get timed out and disconnected by the > database. If you have control over when your database server > restarts, it shouldn't be difficult to make sure your app servers are > restarted at the same time.
It is a bit tricky actually because I need some tool to ssh over all servers in the cluster and ask them to restart. Also, using it this way causes some downtime and loading the Rails app is much slower than restarting the PostgreSQL database (which happens in within a second).
No downtime was reported when we restarted the database from the Grails app, but we had issues with the Rails app.
Anyway, it is good to know that you're working on such an extension. Looking forward to it.
On Tuesday, October 23, 2012 9:09:25 AM UTC-7, Jeremy Evans wrote:
> On Monday, October 22, 2012 6:20:41 PM UTC-7, Rodrigo Rosenfeld Rosas > wrote:
>> Em segunda-feira, 22 de outubro de 2012 19h34min26s UTC-2, Jeremy Evans >> escreveu:
>>> On Monday, October 22, 2012 11:47:29 AM UTC-7, Rodrigo Rosenfeld Rosas >>> wrote:
>>>> I had to restart the PostgreSQL service in our database server and the >>>> Rails apps using Sequel stopped working.
>>>> Is there a way to instruct Sequel to try to reconnect on such cases? >>>> One >>>> that works both for MRI and JRuby?
>>> Sequel should be removing the connections from the pool, reraising the >>> exceptions, and creating new connections as needed in the future. So it's >>> expected that some DatabaseDisconnectErrors will be raised, but Sequel >>> should recover, assuming that it correctly recognizes the disconnections >>> (which it should on the postgres and jdbc/postgres adapters). If that >>> isn't what you experienced, please try to put together a reproducible test >>> case and I'll take a look.
>> But this is a problem for our setup as we have a cluster with a dozen >> servers and there will be as many reported errors as there are servers in >> the cluster and all of them end up in our SLA errors log which affects our >> contract :(
>> Is there a way of just instructing Sequel to query something like "select >> 1" before attempting to use the connection? Hibernate has an option to do >> so and it works. Our application is a mix of a Grails and a Rails app. We >> only had to restart the Rails servers of the cluster as the Grails ones >> that used Hibernate with this option enabled didn't experience any issues.
>> We'd like to be able to do something like with Sequel as well.
>> Would that be possible?
> It's actually something I'm planning to implement as a extension very > soon. The recently committed refactoring of the database <-> connection > pool interface is the first step, I need to add code to check connections > for validity and then an extension that automatically checks connections > for validity when they are checked out from the pool.
The code to check connections for validity has been pushed. I have the extension that automatically checks connections for validity on pool checkout written and apparently working based on the integration-level tests, but I still need to add documentation and extension-level tests. I expect that the extension will be pushed to the master branch sometime tomorrow.
> On Tuesday, October 23, 2012 9:09:25 AM UTC-7, Jeremy Evans wrote:
> > On Monday, October 22, 2012 6:20:41 PM UTC-7, Rodrigo Rosenfeld Rosas
> > wrote:
> >> Em segunda-feira, 22 de outubro de 2012 19h34min26s UTC-2, Jeremy Evans
> >> escreveu:
> >>> On Monday, October 22, 2012 11:47:29 AM UTC-7, Rodrigo Rosenfeld Rosas
> >>> wrote:
> >>>> I had to restart the PostgreSQL service in our database server and the
> >>>> Rails apps using Sequel stopped working.
> >>>> Is there a way to instruct Sequel to try to reconnect on such cases?
> >>>> One
> >>>> that works both for MRI and JRuby?
> >>> Sequel should be removing the connections from the pool, reraising the
> >>> exceptions, and creating new connections as needed in the future. So
> >>> it's expected that some DatabaseDisconnectErrors will be raised, but
> >>> Sequel should recover, assuming that it correctly recognizes the
> >>> disconnections (which it should on the postgres and jdbc/postgres
> >>> adapters). If that isn't what you experienced, please try to put
> >>> together a reproducible test case and I'll take a look.
> >> But this is a problem for our setup as we have a cluster with a dozen
> >> servers and there will be as many reported errors as there are servers
> >> in the cluster and all of them end up in our SLA errors log which
> >> affects our contract :(
> >> Is there a way of just instructing Sequel to query something like
> >> "select 1" before attempting to use the connection? Hibernate has an
> >> option to do so and it works. Our application is a mix of a Grails and
> >> a Rails app. We only had to restart the Rails servers of the cluster as
> >> the Grails ones that used Hibernate with this option enabled didn't
> >> experience any issues.
> >> We'd like to be able to do something like with Sequel as well.
> >> Would that be possible?
> > It's actually something I'm planning to implement as a extension very
> > soon. The recently committed refactoring of the database <-> connection
> > pool interface is the first step, I need to add code to check connections
> > for validity and then an extension that automatically checks connections
> > for validity when they are checked out from the pool.
> The code to check connections for validity has been pushed. I have the
> extension that automatically checks connections for validity on pool
> checkout written and apparently working based on the integration-
level
> tests, but I still need to add documentation and extension-level tests. I
> expect that the extension will be pushed to the master branch sometime
> tomorrow.
Wow! This is really great news! Thank you so much!
> Em terça-feira 23 outubro 2012, às 20:28:47, Jeremy Evans escreveu: > > On Tuesday, October 23, 2012 9:09:25 AM UTC-7, Jeremy Evans > wrote: > > > On Monday, October 22, 2012 6:20:41 PM UTC-7, Rodrigo > Rosenfeld Rosas
> > > wrote: > > >> Em segunda-feira, 22 de outubro de 2012 19h34min26s UTC-2, > Jeremy Evans
> > >> escreveu: > > >>> On Monday, October 22, 2012 11:47:29 AM UTC-7, Rodrigo > Rosenfeld Rosas
> > >>> wrote: > > >>>> I had to restart the PostgreSQL service in our database > server and the > > >>>> Rails apps using Sequel stopped working.
> > >>>> Is there a way to instruct Sequel to try to reconnect on such > cases? > > >>>> One > > >>>> that works both for MRI and JRuby?
> > >>> Sequel should be removing the connections from the pool, > reraising the > > >>> exceptions, and creating new connections as needed in the > future. So > > >>> it's expected that some DatabaseDisconnectErrors will be > raised, but > > >>> Sequel should recover, assuming that it correctly recognizes > the > > >>> disconnections (which it should on the postgres and > jdbc/postgres > > >>> adapters). If that isn't what you experienced, please try to put > > >>> together a reproducible test case and I'll take a look.
> > >> But this is a problem for our setup as we have a cluster with a > dozen > > >> servers and there will be as many reported errors as there are > servers > > >> in the cluster and all of them end up in our SLA errors log which > > >> affects our contract :(
> > >> Is there a way of just instructing Sequel to query something like > > >> "select 1" before attempting to use the connection? Hibernate > has an > > >> option to do so and it works. Our application is a mix of a Grails > and > > >> a Rails app. We only had to restart the Rails servers of the > cluster as > > >> the Grails ones that used Hibernate with this option enabled > didn't > > >> experience any issues.
> > >> We'd like to be able to do something like with Sequel as well.
> > >> Would that be possible?
> > > It's actually something I'm planning to implement as a extension > very > > > soon. The recently committed refactoring of the database <-> > connection > > > pool interface is the first step, I need to add code to check > connections > > > for validity and then an extension that automatically checks > connections > > > for validity when they are checked out from the pool.
> > The code to check connections for validity has been pushed. I have > the > > extension that automatically checks connections for validity on pool > > checkout written and apparently working based on the integration- > level > > tests, but I still need to add documentation and extension-level > tests. I > > expect that the extension will be pushed to the master branch > sometime > > tomorrow.
> Wow! This is really great news! Thank you so much!
By the way, I hope this will help me with another issue I'm having. Yesterday I had to restart the Rails apps because of EBADF (Bad File Descriptor) errors in a Sequel line like "User[id]". For some reason we're having some troubles with our connection to the PG server even when we don't restart the server. :(
> Em terça-feira, 23 de outubro de 2012 22h04min13s UTC-2, Rodrigo Rosenfeld > Rosas escreveu:
>> Em terça-feira 23 outubro 2012, às 20:28:47, Jeremy Evans escreveu: >> > On Tuesday, October 23, 2012 9:09:25 AM UTC-7, Jeremy Evans >> wrote: >> > > On Monday, October 22, 2012 6:20:41 PM UTC-7, Rodrigo >> Rosenfeld Rosas
>> > > wrote: >> > >> Em segunda-feira, 22 de outubro de 2012 19h34min26s UTC-2, >> Jeremy Evans
>> > >> escreveu: >> > >>> On Monday, October 22, 2012 11:47:29 AM UTC-7, Rodrigo >> Rosenfeld Rosas
>> > >>> wrote: >> > >>>> I had to restart the PostgreSQL service in our database >> server and the >> > >>>> Rails apps using Sequel stopped working.
>> > >>>> Is there a way to instruct Sequel to try to reconnect on such >> cases? >> > >>>> One >> > >>>> that works both for MRI and JRuby?
>> > >>> Sequel should be removing the connections from the pool, >> reraising the >> > >>> exceptions, and creating new connections as needed in the >> future. So >> > >>> it's expected that some DatabaseDisconnectErrors will be >> raised, but >> > >>> Sequel should recover, assuming that it correctly recognizes >> the >> > >>> disconnections (which it should on the postgres and >> jdbc/postgres >> > >>> adapters). If that isn't what you experienced, please try to put >> > >>> together a reproducible test case and I'll take a look.
>> > >> But this is a problem for our setup as we have a cluster with a >> dozen >> > >> servers and there will be as many reported errors as there are >> servers >> > >> in the cluster and all of them end up in our SLA errors log which >> > >> affects our contract :(
>> > >> Is there a way of just instructing Sequel to query something like >> > >> "select 1" before attempting to use the connection? Hibernate >> has an >> > >> option to do so and it works. Our application is a mix of a Grails >> and >> > >> a Rails app. We only had to restart the Rails servers of the >> cluster as >> > >> the Grails ones that used Hibernate with this option enabled >> didn't >> > >> experience any issues.
>> > >> We'd like to be able to do something like with Sequel as well.
>> > >> Would that be possible?
>> > > It's actually something I'm planning to implement as a extension >> very >> > > soon. The recently committed refactoring of the database <-> >> connection >> > > pool interface is the first step, I need to add code to check >> connections >> > > for validity and then an extension that automatically checks >> connections >> > > for validity when they are checked out from the pool.
>> > The code to check connections for validity has been pushed. I have >> the >> > extension that automatically checks connections for validity on pool >> > checkout written and apparently working based on the integration- >> level >> > tests, but I still need to add documentation and extension-level >> tests. I >> > expect that the extension will be pushed to the master branch >> sometime >> > tomorrow.
>> Wow! This is really great news! Thank you so much!
> By the way, I hope this will help me with another issue I'm having. > Yesterday I had to restart the Rails apps because of EBADF (Bad File > Descriptor) errors in a Sequel line like "User[id]". For some reason we're > having some troubles with our connection to the PG server even when we > don't restart the server. :(
Another thing to consider is removing all connections from the pool when a connection error is detected in one of them. Most likely it will happen with the other ones as well. For instance if you restart the PG server, all connections will become invalid. But still Sequel calls will fail 50 times consecutively if there are 50 connections in the pool. Maybe there should be an option to clear the pool once it has been detected some problem in any of the connections. Maybe that could be the default behavior. Does it make sense?
For example, suppose you have a 10 servers cluster and each one has a pool size of 50. There will be 500 failed requests after restarting a PG database or some similar issue that could cause the connections to be killed.
On Wednesday, October 24, 2012 6:07:58 AM UTC-7, Rodrigo Rosenfeld Rosas wrote:
> Another thing to consider is removing all connections from the pool when a > connection error is detected in one of them. Most likely it will happen > with the other ones as well. For instance if you restart the PG server, all > connections will become invalid. But still Sequel calls will fail 50 times > consecutively if there are 50 connections in the pool. Maybe there should > be an option to clear the pool once it has been detected some problem in > any of the connections. Maybe that could be the default behavior. Does it > make sense?
It makes sense as an option, but not as the default behavior (I'll consider patches, but won't work on it myself). I don't think very many people are doing what you are doing (restarting the database while the application is running). Personally, I consider that a bad idea. The problem that far more people have (I'm guessing based on reported issues) is that idle connections in the connection pool get disconnected by the database after a period of time. In that case, removing all connections from the pool just because a single one idled out is definitely the wrong approach.
Note that with the connection verification extension I'm working on, you can verify connections on pool checkout before use, so your only chance of issues there is a disconnect while a connection is checked out (in which case I don't think anything could help). Of course, this can result in a performance hit, especially if you aren't attempting to manage connection checkouts manually in a coarse manner (e.g. in a rack middleware). Using coarse connection checkouts hurts multithreaded use, basically limiting the number of concurrent web requests to the number of database connections, so there is a definite tradeoff in that area.
> On Wednesday, October 24, 2012 6:07:58 AM UTC-7, Rodrigo Rosenfeld Rosas > wrote:
>> Another thing to consider is removing all connections from the pool when >> a connection error is detected in one of them. Most likely it will happen >> with the other ones as well. For instance if you restart the PG server, all >> connections will become invalid. But still Sequel calls will fail 50 times >> consecutively if there are 50 connections in the pool. Maybe there should >> be an option to clear the pool once it has been detected some problem in >> any of the connections. Maybe that could be the default behavior. Does it >> make sense?
> It makes sense as an option, but not as the default behavior (I'll > consider patches, but won't work on it myself). I don't think very many > people are doing what you are doing (restarting the database while the > application is running). Personally, I consider that a bad idea. The > problem that far more people have (I'm guessing based on reported issues) > is that idle connections in the connection pool get disconnected by the > database after a period of time. In that case, removing all connections > from the pool just because a single one idled out is definitely the wrong > approach.
> Note that with the connection verification extension I'm working on, you > can verify connections on pool checkout before use, so your only chance of > issues there is a disconnect while a connection is checked out (in which > case I don't think anything could help). Of course, this can result in a > performance hit, especially if you aren't attempting to manage connection > checkouts manually in a coarse manner (e.g. in a rack middleware). Using > coarse connection checkouts hurts multithreaded use, basically limiting the > number of concurrent web requests to the number of database connections, so > there is a definite tradeoff in that area.
In the meanwhile I'm just monkey patching Sequel 3.35.0:
# Patch Sequel to try the connection first before using it class Sequel::ThreadedConnectionPool < Sequel::ConnectionPool private def available begin (@available_connections.pop || make_new(DEFAULT_SERVER)).tap{|c| c.execute 'select 1' } rescue # reconnects on invalid connection (disconnected for some reason) DB.disconnect DB.pool.all_connections{} @available_connections.pop end end end
This works for me, but as soon as you get your extension released I'll change to it :)
There are other reasons for the connections to be closed. I don't often restart my database and only did that once because I was restoring a dump from another server into this one and I needed to rename the database to database-old and database-new to database but I had to kill all connections to the database first before being able to rename them. I didn't have much time to investigate how to do that in that time so I just restarted the pg service. Now I'd probably use this instead:
But I had this disconnection issue yesterday again even not restarting the database. I don't know the cause yet, but while researching the subject, one possible reason could be that a switch is killing idle connections after a while. So, suppose PostgreSQL keep alive is set to 2h and the switch is set to kill TCP connections idle for 15 minutes. All connections could be invalid after a while where no one is using the database (at night, for instance). I don't know if that is our case though and I'm still investigating what could be causing the connections to become invalid sometimes. I'm just saying that the testOnBorrow option available in most Java libraries proved to be pretty useful in cases like this. So I'm willing to accept a minimum overhead (select 1) to avoid such kind of problems in my Ruby applications as well. For now the monkey patch above is working for me, but I'd prefer to use your extension when it is ready.
I'm just saying that I can think of more reasons for a connection to become invalid even when not restarting the database. In most cases I think it would affect all connections instead of a single one. Another option would be someone replacing a switch with another one...
> Em quarta-feira, 24 de outubro de 2012 13h38min36s UTC-2, Jeremy Evans > escreveu:
>> On Wednesday, October 24, 2012 6:07:58 AM UTC-7, Rodrigo Rosenfeld Rosas >> wrote:
>>> Another thing to consider is removing all connections from the pool when >>> a connection error is detected in one of them. Most likely it will happen >>> with the other ones as well. For instance if you restart the PG server, all >>> connections will become invalid. But still Sequel calls will fail 50 times >>> consecutively if there are 50 connections in the pool. Maybe there should >>> be an option to clear the pool once it has been detected some problem in >>> any of the connections. Maybe that could be the default behavior. Does it >>> make sense?
>> It makes sense as an option, but not as the default behavior (I'll >> consider patches, but won't work on it myself). I don't think very many >> people are doing what you are doing (restarting the database while the >> application is running). Personally, I consider that a bad idea. The >> problem that far more people have (I'm guessing based on reported issues) >> is that idle connections in the connection pool get disconnected by the >> database after a period of time. In that case, removing all connections >> from the pool just because a single one idled out is definitely the wrong >> approach.
>> Note that with the connection verification extension I'm working on, you >> can verify connections on pool checkout before use, so your only chance of >> issues there is a disconnect while a connection is checked out (in which >> case I don't think anything could help). Of course, this can result in a >> performance hit, especially if you aren't attempting to manage connection >> checkouts manually in a coarse manner (e.g. in a rack middleware). Using >> coarse connection checkouts hurts multithreaded use, basically limiting the >> number of concurrent web requests to the number of database connections, so >> there is a definite tradeoff in that area.
> In the meanwhile I'm just monkey patching Sequel 3.35.0:
> # Patch Sequel to try the connection first before using it > class Sequel::ThreadedConnectionPool < Sequel::ConnectionPool > private > def available > begin > (@available_connections.pop || make_new(DEFAULT_SERVER)).tap{|c| > c.execute 'select 1' } > rescue > # reconnects on invalid connection (disconnected for some reason) > DB.disconnect > DB.pool.all_connections{} > @available_connections.pop > end > end > end
> This works for me, but as soon as you get your extension released I'll > change to it :)
Actually it doesn't. Just realized that after trying to deploy it on JRuby/GlassFish. Thread reentrant locking related issues on this method :(
On Tuesday, October 23, 2012 3:28:47 PM UTC-7, Jeremy Evans wrote: > The code to check connections for validity has been pushed. I have the > extension that automatically checks connections for validity on pool > checkout written and apparently working based on the integration-level > tests, but I still need to add documentation and extension-level tests. I > expect that the extension will be pushed to the master branch sometime > tomorrow.
> On Tuesday, October 23, 2012 3:28:47 PM UTC-7, Jeremy Evans wrote:
>> The code to check connections for validity has been pushed. I have the >> extension that automatically checks connections for validity on pool >> checkout written and apparently working based on the integration-level >> tests, but I still need to add documentation and extension-level tests. I >> expect that the extension will be pushed to the master branch sometime >> tomorrow.
Awesome! Thanks! I have a tight deadline for delivering some features for tomorrow afternoon and I'll be working all this night on that. After the demo happening tomorrow I'll be able to check this extension. Fortunately we didn't experience any issues today like we did yesterday but we still don't know what caused them and it is really bad to us when they happen. That is why I'd like to get this extension set up soon before we get more issues like Yesterday's.
I'll let you know tomorrow how it works for us. Thank you very much!
Em quarta-feira, 24 de outubro de 2012 21h14min42s UTC-2, Jeremy Evans escreveu:
> On Tuesday, October 23, 2012 3:28:47 PM UTC-7, Jeremy Evans wrote:
>> The code to check connections for validity has been pushed. I have the >> extension that automatically checks connections for validity on pool >> checkout written and apparently working based on the integration-level >> tests, but I still need to add documentation and extension-level tests. I >> expect that the extension will be pushed to the master branch sometime >> tomorrow.
With the new 9.2 JDBC4 driver I didn't have any issues and your extension is working great!
Amazing job. Thank you very much! We had issues again this morning and I'm hoping the new version we're going to release with your extension should stop the errors from happening. Our partner is really sad about this issue and your work on this extension may save our asses :)
> Em quarta-feira, 24 de outubro de 2012 21h14min42s UTC-2, Jeremy Evans > escreveu:
>> On Tuesday, October 23, 2012 3:28:47 PM UTC-7, Jeremy Evans wrote:
>>> The code to check connections for validity has been pushed. I have the >>> extension that automatically checks connections for validity on pool >>> checkout written and apparently working based on the integration-level >>> tests, but I still need to add documentation and extension-level tests. I >>> expect that the extension will be pushed to the master branch sometime >>> tomorrow.
> With the new 9.2 JDBC4 driver I didn't have any issues and your extension > is working great!
> Amazing job. Thank you very much! We had issues again this morning and I'm > hoping the new version we're going to release with your extension should > stop the errors from happening. Our partner is really sad about this issue > and your work on this extension may save our asses :)
> Thank you very much!
Unfortunately I'm still having troubles when deploying on JRuby :(
When I call Devise's authenticate_user from my controller I get this:
ThreadError Exception: Mutex relocking by same thread
I'm trying to debug into Devise's internal now to understand what is happening but it seems that ruby-debug is a bit broken too on JRuby 1.7:
> Em sexta-feira, 26 de outubro de 2012 14h44min06s UTC-2, Rodrigo Rosenfeld > Rosas escreveu:
>> Em quarta-feira, 24 de outubro de 2012 21h14min42s UTC-2, Jeremy Evans >> escreveu:
>>> On Tuesday, October 23, 2012 3:28:47 PM UTC-7, Jeremy Evans wrote:
>>>> The code to check connections for validity has been pushed. I have the >>>> extension that automatically checks connections for validity on pool >>>> checkout written and apparently working based on the integration-level >>>> tests, but I still need to add documentation and extension-level tests. I >>>> expect that the extension will be pushed to the master branch sometime >>>> tomorrow.
>> With the new 9.2 JDBC4 driver I didn't have any issues and your extension >> is working great!
>> Amazing job. Thank you very much! We had issues again this morning and >> I'm hoping the new version we're going to release with your extension >> should stop the errors from happening. Our partner is really sad about this >> issue and your work on this extension may save our asses :)
>> Thank you very much!
> Unfortunately I'm still having troubles when deploying on JRuby :(
> When I call Devise's authenticate_user from my controller I get this:
> ThreadError Exception: Mutex relocking by same thread
> I'm trying to debug into Devise's internal now to understand what is > happening but it seems that ruby-debug is a bit broken too on JRuby 1.7:
I'm aware of the difference, Mutex is being used on purpose for performance (Monitor is about 2.5x slower).
One other limitation with the current code is that it does the connection validation while holding the pool mutex, which means that no other threads can checkout a connection while this connection is being validated. That's very bad. I've modified things so the checking is done while not holding the mutex (in addition to precaching the SQL).
I'm just running the changes through a final set of tests, hopefully it will be pushed within 15 minutes.
> I'm aware of the difference, Mutex is being used on purpose for > performance (Monitor is about 2.5x slower).
> One other limitation with the current code is that it does the > connection validation while holding the pool mutex, which means that > no other threads can checkout a connection while this connection is > being validated. That's very bad. I've modified things so the > checking is done while not holding the mutex (in addition to > precaching the SQL).
> I'm just running the changes through a final set of tests, hopefully > it will be pushed within 15 minutes.
Great, specially because I tried using a Monitor instead of Mutex and now "rails server" segfaults on JRuby 1.7 :P
> I'm aware of the difference, Mutex is being used on purpose for > performance (Monitor is about 2.5x slower).
> One other limitation with the current code is that it does the connection > validation while holding the pool mutex, which means that no other threads > can checkout a connection while this connection is being validated. That's > very bad. I've modified things so the checking is done while not holding > the mutex (in addition to precaching the SQL).
> I'm just running the changes through a final set of tests, hopefully it > will be pushed within 15 minutes.
> I'm aware of the difference, Mutex is being used on purpose for
> performance (Monitor is about 2.5x slower).
> One other limitation with the current code is that it does the
> connection validation while holding the pool mutex, which means
> that no other threads can checkout a connection while this
> connection is being validated. That's very bad. I've modified
> things so the checking is done while not holding the mutex (in
> addition to precaching the SQL).
> I'm just running the changes through a final set of tests,
> hopefully it will be pushed within 15 minutes.
