Problem with RabbitMQ & Pika - Connection Reset by Peer

1,200 views
Skip to first unread message

Matt Shields

unread,
Apr 27, 2018, 10:13:51 AM4/27/18
to rabbitmq-users
I have four servers (old puppet, old rabbitmq, new puppet, and old rabbitmq).  In my old environment I have puppet setup the rabbitmq server and I have a test python/pika script that I use to test to make sure everything is ready for production use.  I created the new puppet and rabbitmq servers so that I could use updated versions of everything.  I've been able to get the rabbitmq server built, but I'm having an issue trying to connect to it.  I can log into the web admin using all the accounts that Puppet creates for rabbit, but I keep getting "Connection Reset by Peer" error.

I've uploaded everything to a gist to keep things clean.  I've listed all the configs, software versions, log outputs, and the script that's run with it's output. Again, this works on the old puppet and rabbitmq servers, the only difference is the new environment uses newer versions of all the software.  Any help would be greatly appreciated.


Matt

Michael Klishin

unread,
Apr 27, 2018, 10:35:08 AM4/27/18
to rabbitm...@googlegroups.com
Thank you for starting with providing quite a bint of information.

According to the log [1] there was a successful connection.
I'd use Wireshark (we have a doc guide on rabbitmq.com) to dig deeper.

Also note that Erlang versions up to 19.3.6.5 have known bugs that
can prevent nodes from accepting connections [2]. RabbitMQ 3.3.x cannot run on 19.x
so consider upgrading [3][4].

With a Wireshark/tcpdump/libpcap dump there will be more information to work with.
There's also [5] which probably won't reveal much new to you but I thought I should mention it.


--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Matt Shields

unread,
Apr 27, 2018, 11:23:44 AM4/27/18
to rabbitmq-users
Excellent.  Thank you.  I bet it is Erlang.  I haven't updated my Puppet code to use a newer version of Erlang.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michael Klishin

unread,
Apr 27, 2018, 9:07:04 PM4/27/18
to rabbitm...@googlegroups.com
To be honest I wouldn't be so sure. Networking and distributed systems can fail in really obscure ways.
But upgrading in this case would eliminate a few years worth of resolved issues, so if you can do it at all, give it a try first.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Matt Shields

unread,
May 1, 2018, 11:01:23 AM5/1/18
to rabbitmq-users
It ended up being an Erlang problem.  The Amazon Linux ami only had Erlang R14B available.  Even the puppet forge modules for RabbitMQ and Erlang didn't take into account that there may be different versions available/needed.  I've had to hack something together in puppet, but it's working now.  Thanks


On Friday, April 27, 2018 at 9:07:04 PM UTC-4, Michael Klishin wrote:
To be honest I wouldn't be so sure. Networking and distributed systems can fail in really obscure ways.
But upgrading in this case would eliminate a few years worth of resolved issues, so if you can do it at all, give it a try first.
On Fri, Apr 27, 2018 at 9:34 AM, Michael Klishin <mkli...@pivotal.io> wrote:
Thank you for starting with providing quite a bint of information.

According to the log [1] there was a successful connection.
I'd use Wireshark (we have a doc guide on rabbitmq.com) to dig deeper.

Also note that Erlang versions up to 19.3.6.5 have known bugs that
can prevent nodes from accepting connections [2]. RabbitMQ 3.3.x cannot run on 19.x
so consider upgrading [3][4].

With a Wireshark/tcpdump/libpcap dump there will be more information to work with.
There's also [5] which probably won't reveal much new to you but I thought I should mention it.

On Fri, Apr 27, 2018 at 2:13 PM, Matt Shields <ma...@shields.tv> wrote:
I have four servers (old puppet, old rabbitmq, new puppet, and old rabbitmq).  In my old environment I have puppet setup the rabbitmq server and I have a test python/pika script that I use to test to make sure everything is ready for production use.  I created the new puppet and rabbitmq servers so that I could use updated versions of everything.  I've been able to get the rabbitmq server built, but I'm having an issue trying to connect to it.  I can log into the web admin using all the accounts that Puppet creates for rabbit, but I keep getting "Connection Reset by Peer" error.

I've uploaded everything to a gist to keep things clean.  I've listed all the configs, software versions, log outputs, and the script that's run with it's output. Again, this works on the old puppet and rabbitmq servers, the only difference is the new environment uses newer versions of all the software.  Any help would be greatly appreciated.


Matt

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

Michael Klishin

unread,
May 1, 2018, 11:19:48 AM5/1/18
to rabbitm...@googlegroups.com
Interesting. What package type did your Puppet module use? At least with Debian and RPM packages
you shouldn't be able to install an incompatible version. That said, if Erlang is compiled from source and added to PATH
or similar, or with generic UNIX packages, there isn't anything other than Rabbit itself to do the checking.

Our code tends to do feature checking not version checking (although we do some of that during cluster formation, too).
It's a good question whether making RabbitMQ be aggressive around Erlang/OTP version checking would be a net win.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages