Handling connection timeouts gracefully with a replica set

[Justin Dossey]

Hi all,

I have a pretty robust setup in production, but occasionally the write load on the primary server is high enough that a query generates a Mongo::ConnectionTimeoutError.  Handling this can be a bit challenging, so I wanted to share one approach to resolution to encourage discussion.

Original query:

MyStuff.all(:key => val, :key2 => val)

Updated query:

MyStuff.all_with_fallback(:key => val, :key2 => val)

Implementation

class MyStuff

...(mongomapper stuff)...

def self.all_with_fallback(conditions)

  read_preference = :primary

  begin

    records = []

    collection.find(conditions, {:read => read_preference, :timeout => 10}) do |cur|

       cur.each {|hsh| records << load(hsh) }

    end

    return records

   rescue Mongo::ConnectionTimeoutError => e

     raise e if read_preference == :secondary

     read_preference == :secondary

     retry

    end

  end

end        

Justin Dossey