Fault tolerance in case of slave unavailability

[Vincent Boisard]

Hello,

I'm running a Rails 4 app, and I'm using Octopus to perform specific queries (mostly stats) on a slave, and for the most part it works really great.

However, I noticed that in case the slave isn't available (network issue, maintenance, etc...), nothing works anymore, even on pages that do not query the slave.

Is there a way to allow automatic failover to the master in case of slave unavailability? Or at least, a way to contain the errors to the pages that perform queries on the slave?

Thank you very much in advance,

[G. Sobrinho]

Did you get an exception?

--
You received this message because you are subscribed to the Google Groups "Octopus - Database sharding for ActiveRecord" group.
To unsubscribe from this group and stop receiving emails from it, send an email to octopus-activere...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

Gabriel Sobrinho

http://gabrielsobrinho.com/

[Vincent Boisard]

Indeed I did. I simulated the behaviour by starting my app, then configuring the slave to refuse connections. 

I get the following stacktrace

Mysql2::Error - Can't connect to MySQL server on '192.168.10.15' (61 "Connection refused"):
  mysql2 (0.3.15) lib/mysql2/client.rb:67:in `initialize'
  activerecord (4.1.9) lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `mysql2_connection'
  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:435:in `new_connection'
  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:445:in `checkout_new_connection'
  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `acquire_connection'
  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `block in checkout'
  /Users/elhu/.rbenv/versions/2.1.5/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:350:in `checkout'
  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
  /Users/elhu/.rbenv/versions/2.1.5/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
  ar-octopus (0.8.4) lib/octopus/proxy.rb:205:in `safe_connection'
  ar-octopus (0.8.4) lib/octopus/proxy.rb:292:in `block in clear_query_cache'
  ar-octopus (0.8.4) lib/octopus/proxy.rb:292:in `clear_query_cache'
  activerecord (4.1.9) lib/active_record/query_cache.rb:51:in `restore_query_cache_settings'
  activerecord (4.1.9) lib/active_record/query_cache.rb:43:in `rescue in call'
  activerecord (4.1.9) lib/active_record/query_cache.rb:32:in `call'
  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in `call'
  activerecord (4.1.9) lib/active_record/migration.rb:380:in `call'
...

Thank you very much for taking the time to look into this.

On Monday, March 16, 2015 at 9:27:08 PM UTC+1, Gabriel Sobrinho wrote:

Did you get an exception?

On Wed, Mar 11, 2015 at 7:14 AM, Vincent Boisard <bois...@gmail.com> wrote:

Hello,

I'm running a Rails 4 app, and I'm using Octopus to perform specific queries (mostly stats) on a slave, and for the most part it works really great.

However, I noticed that in case the slave isn't available (network issue, maintenance, etc...), nothing works anymore, even on pages that do not query the slave.

Is there a way to allow automatic failover to the master in case of slave unavailability? Or at least, a way to contain the errors to the pages that perform queries on the slave?

Thank you very much in advance,

--
You received this message because you are subscribed to the Google Groups "Octopus - Database sharding for ActiveRecord" group.

To unsubscribe from this group and stop receiving emails from it, send an email to octopus-activerecord+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gabriel Sobrinho]

Sorry for the delay, pretty busy weeks here.

As far I remember we don’t support down connections, which may makes sense but I’m not really sure about that.

What was you expecting octopus to do in that case? Ignore the slave and use another one? Which one?

On Mar 27, 2015, at 10:18, Vincent Boisard <bois...@gmail.com> wrote:

Indeed I did. I simulated the behaviour by starting my app, then configuring the slave to refuse connections. 

I get the following stacktrace

Mysql2::Error - Can't connect to MySQL server on '192.168.10.15' (61 "Connection refused"):
  mysql2 (0.3.15) lib/mysql2/client.rb:67:in `initialize'


  activerecord (4.1.9)lib/active_record/connection_adapters/mysql2_adapter.rb:18:in`mysql2_connection'


  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:435:in `

new_connection'


  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:445:in `checkout_new_connection'


  activerecord (4.1.9)lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in`acquire_connection'


  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `

block in checkout'
  /Users/elhu/.rbenv/versions/2.1.5/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.1.9)lib/active_record/connection_adapters/abstract/connection_pool.rb:350:in`checkout'


  activerecord (4.1.9) lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `

block in connection'
  /Users/elhu/.rbenv/versions/2.1.5/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  activerecord (4.1.9)lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in`connection'

safe_connection'

To unsubscribe from this group and stop receiving emails from it, send an email to octopus-activere...@googlegroups.com.

[Vincent Boisard]

No problem.

Ideally, I would expect octopus to fallback to another, working, connection. However, I would settle with Octopus at least not failing even if it has no reason to use the "slave" collection.

My app is not configured to be fully replicated, only a handful of requests use the slave connection. However, if the connection to the slave fails, every subsequent requests, even to the master, will fail.

Best regards,

To unsubscribe from this group and stop receiving emails from it, send an email to octopus-activere...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 

Cheers,

Gabriel Sobrinho

http://gabrielsobrinho.com/

-- 
You received this message because you are subscribed to the Google Groups "Octopus - Database sharding for ActiveRecord" group.
To unsubscribe from this group and stop receiving emails from it, send an email to octopus-activere...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Octopus - Database sharding for ActiveRecord" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/octopus-activerecord/Ue8kXriWR8U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to octopus-activere...@googlegroups.com.