enotconn (socket is not connected)

[Haseeb]

we are having intermittent errors around 20-40 everyday on an existing system which was working fine previously. 

System was running fine a while but recently we started seeing a lot of errors, every time same message as follows (from server log files), 

ourdomain.dblayer.com rabbitmq200: rabbitmq.24  | Error on AMQP connection <0.14678.2788>: enotconn (socket is not connected)

On the other hand, the client side we are using python (pika) and we get the following error

"ProbableAuthenticationError pika.adapters.base_connection in _check_state_on_disconnect" 

We are using RMQ ( 3.6.14; Erlang 20.1) hosted in cloud (ourdomain.composedb.com). 

Following is the stack trace

ProbableAuthenticationError: null
  File "pika/adapters/blocking_connection.py", line 339, in __init__
    self._process_io_for_connection_setup()
  File "pika/adapters/blocking_connection.py", line 374, in _process_io_for_connection_setup
    self._open_error_result.is_ready)
  File "pika/adapters/blocking_connection.py", line 410, in _flush_output
    self._impl.ioloop.poll()
  File "pika/adapters/select_connection.py", line 602, in poll
    self._process_fd_events(fd_event_map, write_only)
  File "pika/adapters/select_connection.py", line 443, in _process_fd_events
    handler(fileno, events, write_only=write_only)
  File "pika/adapters/base_connection.py", line 364, in _handle_events
    self._handle_read()
  File "pika/adapters/base_connection.py", line 412, in _handle_read
    return self._handle_disconnect()
  File "pika/adapters/base_connection.py", line 288, in _handle_disconnect
    self._adapter_disconnect()
  File "pika/adapters/select_connection.py", line 95, in _adapter_disconnect
    super(SelectConnection, self)._adapter_disconnect()
  File "pika/adapters/base_connection.py", line 154, in _adapter_disconnect
    self._check_state_on_disconnect()
  File "pika/adapters/base_connection.py", line 173, in _check_state_on_disconnect
    raise exceptions.ProbableAuthenticationError

[Luke Bakken]

Hello,

Without more information my first guess is that there is a firewall or load balancer that is closing the TCP socket unexpectedly between RabbitMQ and your Pika application.

Luke

[Haseeb]

Regarding unexpected connection closing, I will say that we have application that consume data from the queue and it can fall asleep as long as for an hour (based on certain logic). It be due to inactivity that connection is getting closed, if so how we can avoid the exception to happen e.g. by running an additional check or setting some property? Thanks

[Haseeb]

[Luke Bakken]

Hello,

Do you have a firewall or load balancer between your application and RabbitMQ?

Heartbeats should keep your connection open. Are you using any custom heartbeat configuration in your Pika application or RabbitMQ?

If your system was running "fine for a while" and then started exhibiting this issue you must try to determine what changed. Were there software upgrades? Configuration changes?

Can you provide more than just one log line from RabbitMQ?

Thanks,

Luke

[Haseeb]

Thanks for the response and patience. :-)

Do you have a firewall or load balancer between your application and RabbitMQ?

We are using the compose db and it does use haproxy but there isn't any firewall.

Heartbeats should keep your connection open. Are you using any custom heartbeat configuration in your Pika application or RabbitMQ?

No, it's just calling methods from the lib. 

If your system was running "fine for a while" and then started exhibiting this issue you must try to determine what changed. Were there software upgrades? Configuration changes?

There wasn't any code changes or version upgrades in the libraries. Server is hosted on compose db, will check there if something happened

Can you provide more than just one log line from RabbitMQ?

It's a basic python application which fetches data from the online web API and send it on the queue, then downstream application process the data and put it into the database. 

It's version are 3.6.14; Erlang 20.1. Python/pika application sleeps and reads the data from the webapi every 60 mins. Similarly downstream application runs, reads the queue, sleeps, then read the queue. While reading and sleeping, reading sometime it throws the error. Can you please let me know what are some more details you need?

[Luke Bakken]

Hello,

Python/pika application sleeps and reads the data from the webapi every 60 mins.

I bet that the sleep prevents heartbeats because you're blocking Pika's I/O loop. Can you please provide all of your code that uses Pika?

Thanks,

Luke