Pausing a subscription when external service is down

[Keenan Brock]

Hello all,

I have a basic consumer interface that reads from a rabbit queue and republishes to an external service (Apple APNS in my case).

I would like to handle the use case where the external service is down.

Is there a special trick to suspending/blocking/pausing the queue#subscribe block?

My library to call the external service is not written in event machine. But I don't think that is pertinent.

I have tried rejecting the message and closing the rabbit connection, but wasn't able to get the messages back. And it seemed worse than just having a sleep. The sleep could timeout the server heartbeat and kill the connection. So this is not optimal.

How have others solved this?

Thanks for any insight

Keenan

EventMachine.run do

  send_apns = SendApnsContext.new

  AMQP.connect(@rabbit_uri) do |connection|

    #connection.on_tcp_connection_loss do |conn, settings| ...

    AMQP::Channel.new(connection, auto_recovery: true) do |channel|

      #channel.on_error do |ch, channel_close| ...

      channel.direct(@apns_channel, durable: true) do |exchange|

        AMQP::Queue.new(channel, @apns_queue, durable: true) do |queue|

          queue.bind(exchange, routing_key: @apns_routing_key)

          queue.subscribe do |metadata, payload| #(ack: true)

            begin

              send_apns.send_notification(payload) #retries 2 times - regular sockets

            rescue Errno::ECONNREFUSED

              Rails.logger.error "ApnsService trouble sending #{payload} retrying in 5 seconds"

              sleep(5) #BAD

              retry

            end

          end

        end

      end

    end

  end

end

[Michael Klishin]

2013/5/9 Keenan Brock <kee...@thebrocks.net>

Is there a special trick to suspending/blocking/pausing the queue#subscribe block?

My library to call the external service is not written in event machine. But I don't think that is pertinent.

EventMachine.defer which will execute a block in a thread pool. However, if your problem is that external services may be down, this is likely to not

be enough. Eventually you'll have to keep track of what has been delivered in a data store of some kind.

 

I have tried rejecting the message and closing the rabbit connection, but wasn't able to get the messages back. And it seemed worse than just having a sleep. The sleep could timeout the server heartbeat and kill the connection. So this is not optimal.

If redeliveries do not happen, this means you use automatic acknowledgement mode. See the docs. Closing the channel

is sufficient, no need to close the connection.
--
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin