Connection confusion (e.g. DNS requests went to webservers tcp port 9200)

[Manu Thoen]

Hi there!

We're having an index worker. Its consuming RabbitMQ messages, querying against

MongoDB and HTTP POSTing to our elasticsearch cluster. Nothing special so far.

A few months ago we discovered conspicuous HTTP 400 log entries in nginx, which is the

frontend to our elasticsearch cluster. They appear a few times a day (~10 times)

and containing the hostname of the HTTP endpoint (search.srv.company.tld).

1.2.3.4 - - [26/Jun/2014:12:56:19 +0200] "\xF7U\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x06 **search\x03srv\x0Ccompany\x03tld** \x00\x00\x1C\x00\x01" 400 166 "-" "-"

We were totally confused, but found out whats happening: What you see there is

a DNS request, which should - of course - sent via UDP to port 53 to a DNS server. Instead

it arrived at our webserver via TCP on port 9200.

Even more strange:

Sometimes the misled DNS request is **concatenated** with the original HTTP request:

1.2.3.4 - - [25/Jun/2014:07:37:17 +0200] "\x9EI\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x06 **search\x03srv\x0Ccompany\x03tld** \x00\x00\x1C\x00\x01 **POST /_bulk HTTP/1.1** " 400 166 "-" "-"

We proved this behaviour by recording network traffic. We also saw DNS requests

going through our AMQP connection (non-http, TCP 5672), which led to break it.

Because of this "connection confusion" the process is throwing "Error: getaddrinfo ENOTFOUND",

but does not trigger the error callbacks.

In general this doesnt look like an expected or wanted behaviour. Could this be a nodejs bug?

> cf-worker@worker02:~$ node -v

> v0.10.22

> cf-worker@worker02:~$ uname -r

> 3.2.0-4-amd64

We're running Debian 7.2

Regards,

Manu