How do I reconnect when the Cassandra service restarts?

[Clark Bremer]

If our Cassandra service is restarted, our app gets this exception on the next request:

Cql::NotConnected Error 

I could rescue this exception and try to reconnect, but don't know the right incantation to do so.  

I found this post: https://github.com/cequel/cequel/issues/115

So I tried:

 Cequel::Record.connection.clear_active_connections!

with no success.

[Matthew A. Brown]

Hi Clark,

Are you using the latest version of Cequel? As of fairly recently, there's an auto-retry mechanism built in; here's the pull request. By default, it should attempt to reconnect three times if the connection is dropped.

Mat

--
You received this message because you are subscribed to the Google Groups "Cequel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cequel+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Clark Bremer]

Hi Mat.  We're using 1.4.2, which should have that change.  But it's not reconnecting.  If I simply do sudo service cassandra restart,  all subsequent attempts to hit the database fail with 

Cql::NotConnectedError.  Using netstat on the app server, I can see the tcp connection is lost.  And I don't see a new connection being established when I retry the request.  

[Matthew A. Brown]

What version of cql-rb do you have installed? I think the retry code in Cequel may only catch the exception raised by newer versions (this is a bug in cequel if so)

[Clark Bremer]

cql-rb (2.0.3)

      ione (~> 1)

To unsubscribe from this group and stop receiving emails from it, send an email to cequel+unsubscribe@googlegroups.com.

[Matthew A. Brown]

OK, this seems like a bug in the retry code—mind filing an issue on the repo?

[Clark Bremer]

You bet.

[Clark Bremer]

Meanwhile, I've updated to the latest versions of all the gems:

   cequel (1.4.4)

    cql-rb (2.0.3)

   ione (1.1.4)

And I still have the problem.  I put some debug code into the cequel gem source, and clear_active_connections! does not seem to be getting triggered.  

I then added my own exception handler to call clear_active_connections!, and I do see that getting executed.  I see the instance variable @client getting removed after the first error, but not on subsequent attempts.  Is that supposed to automatically get created again?  

[Clark Bremer]

I found that if you also remove the instance variable @raw_client in clear_active_connections!, it works properly.  It fails on the first attempt, but recovers for subsequent access.  Although it's still not triggering on it's own. I'm calling clear_active_connections! myself in the exception handler.

[Clark Bremer]

I submitted a PR with a fix and updated tests.  The test was not actually generating an exception, and thus was passing when it shouldn't have.

[Matthew A. Brown]

Thanks for the great investigation and fix, Clark—I've released 1.4.5 with your solution.

[Clark Bremer]

You bet.  Thanks for the quick merge.  I already see one thing I should have done differently :-).   I'll submit another PR that does not need an urgent release.

Look for a similar fix for newrelic coming from my coworker Jeff later today.

[Matthew A. Brown]

Sounds great! It might be worth taking a look at pezra's pull request to integrate ActiveSupport notifications at the record level—I'm not super up to date on how NewRelic's integration works but that seems like the sort of thing that might make integration easier.