Timeout during AMQP handshake and _socket.getaddrinfo() error

[Seung-Jin Sul]

Hi, 

I am using pika 1.1.0 with Rabbitmq 3.8.2. 

We are using the blocking connection and running. a long task (~6-7hr) on the server-side. 

Recently we are seeing 

1) pika.adapters.utils.connection_workflow.AMQPConnectorStackTimeout

and 

2) socket.gaierror: [Errno -3] Temporary failure in name resolution.

We are using this connection setup now (We tested with blocked_connection_timeout=600 but now we are running it without it.

def __init__(self, hb=0): # return a pika connection
    if not self.__connection:
        creds = pika.PlainCredentials(RMQ_USER, RMQ_PASS)
        params = pika.ConnectionParameters(credentials=creds,
                                           host=RMQ_HOST,
                                           virtual_host=RMQ_VHOST,
                                           port=RMQ_PORT,
                                           heartbeat=hb,
                                           #blocked_connection_timeout=600,
                                           )
        self.__connection = pika.BlockingConnection(params)

And on_request is like

#-------------------------------------------------------------------------------
def on_request(ch, method, props, body):
#-------------------------------------------------------------------------------
    msgUnzipped = zloads(body)

    ##############################
    response = run_something(body)
    ##############################

    try:
        logger.info("    Sent the result back to the client via '%s' queue.", props.reply_to)
        logger.debug("   Tag id: %s ", str(props.correlation_id))

        ch.basic_publish(exchange='', ## nameless exchange
                         routing_key=props.reply_to, ## use the queue which the client created
                         properties=pika.BasicProperties(
                            delivery_mode=2, ## make message persistent
                            correlation_id=props.correlation_id),
                         body=response)

        ch.basic_ack(delivery_tag=method.delivery_tag)

    except Exception as e:
        logger.critical("Something wrong in on_request(): %s", e)

Is there anyone who has any clue?

Traceback (most recent call last):
File "/global/projectb/software/pkanot/jgi-taskfarmermq/tfmq-worker", line 637, in <module>
sys.exit(main(sys.argv))
File "/global/projectb/software/pkanot/jgi-taskfarmermq/tfmq-worker", line 559, in main
rabbitConnection = RmqConnectionHB()
File "/global/projectb/software/pkanot/jgi-taskfarmermq/Utils/RabbitmqConnection.py", line 227, in __init__
self.__connection = pika.BlockingConnection(params)
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 359, in __init__
self._impl = self._create_connection(parameters, _impl_class)
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 450, in _create_connection
raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError
Process Process-1:
Traceback (most recent call last):
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/global/projectb/software/pkanot/jgi-taskfarmermq/tfmq-worker", line 306, in send_hb_to_client_thread
rabbitConnection = RmqConnectionHB()
File "/global/projectb/software/pkanot/jgi-taskfarmermq/Utils/RabbitmqConnection.py", line 227, in __init__
self.__connection = pika.BlockingConnection(params)
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 359, in __init__
self._impl = self._create_connection(parameters, _impl_class)
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 450, in _create_connection
raise self._reap_last_connection_workflow_error(error)
pika.adapters.utils.connection_workflow.AMQPConnectorStackTimeout: Timeout during AMQP handshake'rmq.nersc.gov'/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('128.55.206.27', 5672)); ssl=False.

Traceback (most recent call last):
File "/global/projectb/software/pkanot/jgi-taskfarmermq/tfmq-worker", line 637, in <module>
sys.exit(main(sys.argv))
File "/global/projectb/software/pkanot/jgi-taskfarmermq/tfmq-worker", line 559, in main
rabbitConnection = RmqConnectionHB()
File "/global/projectb/software/pkanot/jgi-taskfarmermq/Utils/RabbitmqConnection.py", line 227, in __init__
self.__connection = pika.BlockingConnection(params)
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 359, in __init__
self._impl = self._create_connection(parameters, _impl_class)
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 450, in _create_connection
raise self._reap_last_connection_workflow_error(error)
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/site-packages/pika/adapters/utils/selector_ioloop_adapter.py", line 562, in _resolve
result = socket.getaddrinfo(self._host, self._port, self._family,
File "/global/projectb/sandbox/omics/gbp/conda/envs/tfmq_client_env/lib/python3.8/socket.py", line 914, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution. 

[Luke Bakken]

Hello -

Here is the relevant error when trying to start a connection:

socket.gaierror: [Errno -3] Temporary failure in name resolution.

It appears that your DNS resolution isn't working correctly.

Luke