RabbitMQ 3.10.5 , Erlang 24.3.4.1 & IIS 10, Windows Server 2019

[__]

Hello Everyone,

Before I file a bug report, it asked me to check here first.  I installed Erlang 24.3.4.1 and then installed RabbitMQ 3.10.5.

Erlang : 24.3.4.1

RabbitMQ : 3.10.5

C# RabbitMQ Nuget Package : 6.4.0

Windows Server : Windows Server 2019, Version 1809, OS Build 17763.3406

IIS : 10.0.1.17763.1

Problem : I created a WCF service and after it posts the data to the MySQL server, it then submits the data to RabbitMQ. The data gets posted to the MySQL server just fine, the next few lines of code is about posting to RabbitMQ and it consistently comes back "Destination Host was Unreachable". RabbitMQ is on the same server as MySQL.

I have narrowed down the problem to be with IIS.  I have done the following:

• Used the Exact same code in a Windows Forms app, ran it on the IIS server and it connected to RabbitMQ with no problem.
• I can run the WCF code in Visual Studio 2017 and it will send the data across the web and my router will forward it on to RabbitMQ just fine and I get the Consumer grabbing the data instantly.
• I have tried on my Dev environment, and I have tried using a group of servers on Microsoft Azure. Same exact problem. My Dev environment is using Windows Server 2012 with IIS 8.#.
• I have enabled the RabbitMQ Management.
• I have created a new user and new password. Again no I am not using guest, guest. I have given the new user permission to default vhost ("/") and "/Testing" and it hasn't seemed to make a difference.
• I have used the new user, and I have used many passwords. My passwords started with an @, removed @ and replaced with a, and then my last password is "1".
• I have tried using a ConnectionFactory() defining the hostname, username, password, port, etc.  Still nothing.
• I have tried using a ConnectionFactory() defining a URI().
• I have tried the IP Address, Server Name, and Full Qualified Machine Name on the network.
• I have made sure the firewall is open.
  
• I tested with Microsofts powershell Test-Netconnection 10.1.0.6 -port 5672, it comes back as able to connect. 
• I then moved the Port thinking IIS is blocking that port for some reason thinking that IIS is blocking 5672, and changed it to 63672 by creating a rabbitmq.conf file.
• Again used Test-Netconnection 10.1.0.6 -port 63672 and it passes. I can ping the server, I can telnet to the server:port, and still nothing.
• I have setup RabbitMQ on the WebServer (IIS) and repeated on the steps above using 127.0.0.1, 10.1.0.5, and localhost. Get all the same responses back. 
• I gave the website, and AppPool "Domain Administrator" rights, so now it can play GOD, still the exact same issue.
• Set "Preload User Profile" = true.

Microsoft says I should be able to connect on either of my networks and network wise everything is accurate and should work.  I get the exact same results on both networks. The only common element in everything is that IIS has to be the issue. It works when the code is in the software. It works when I run the code in Visual Studio with IIS Express. But IIS Express is a very trimmed down version of IIS, and just because it works on your machine doesn't mean it will deploy fine, thus is why we have staging environments. At this point this one issue is the only thing keeping us from deploying our software.

Anyone have any ideas on what could possibly be wrong. If there are no suggestions I will move to report this as a BUG. I noticed that the directions on how to setup a rabbitmq.conf file are a little out of date on their website. It would be nice if this was direct out of the box solution. We are currently working on a workaround, but I don't like the workaround, but we have to do what we go to do to make this crap work as we intended it.

-Mike

[Luke Bakken]

Hi Mike,

As you've determined, this isn't an issue with RabbitMQ. It might be an issue with the .NET client that I would like to be able to solve.

Here is another user who is having issues with the .NET client and IIS - https://groups.google.com/g/rabbitmq-users/c/ql35PcVAWjE

Some questions:

• What is the full exception you see, including the inner exception, when  "Destination Host was Unreachable" is thrown?
  
• Does RabbitMQ log anything at that same point in time?
• Can you provide a complete project that I can run via IIS to see this issue? A GitHub repository that I can clone is preferred, as well as instructions how to run the code using IIS. Providing code and instructions to run it is important because then I can focus on your issue rather than trying to guess how to reproduce it. I'm downloading the evaluation VHD for Windows Server 2019 and should be able to run it via hyper-v.

Thanks,

Luke

[Luke Bakken]

Hi Mike,

I have a Windows Server 2019 environment ready with IIS installed.

[__]

Luke,

Sorry about that. I am extremely busy with my day job and starting my company, and dealing with my own employees.  Anyhow, I can't provide a complete project because it has all of my code (IP) for my entire business. Umm I can possibly build you a new WCF service.  Or, I will add the code I used, updated, etc. that I got from RabbitMQ website.  You will see many functions that I used. Each function is slightly different because I was trying many variances of the connection as I could each time I copied the WCF code to the server.

As for RabbitMQ logs, nothing that I could see. I reviewed the logs and it looks like only the stuff from when I rebooted the service thinking maybe if I make a change I need to reboot the service for the change to take affect.

Point being, the full exception is :

Exception Message :

"None of the specified endpoints were reachable"

Exception Inner Message :

@"""System.IO.FileLoadException: Could not load file or assembly 'System.Threading.Tasks.Extensions, Version=4.2.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)\r\n   at System.Threading.Channels.SingleConsumerUnboundedChannel`1..ctor(Boolean runContinuationsAsynchronously)\r\n   at System.Threading.Channels.Channel.CreateUnbounded[T](UnboundedChannelOptions options)\r\n   at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection..ctor(ConnectionFactory factory, String clientProvidedName)\r\n   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)"""

Exception Stack Trace :

@"""   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)\r\n   at NeXtRabbitMQ_Push.NeXtRabbitMqPushService..ctor(String hostName, String userName, String password, ExchangeDetailModel[] exchanges)\r\n   at PosService.Controllers.ControllersBaseClass.<SendRabbitmqCustomerOrders>d__0.MoveNext() in E:\\NeXt\\repos\\_NextCode\\WCF\\WCF_POS\\Controllers\\ControllersBaseClass.cs:line 33\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()\r\n   at PosService.Controllers.customer_ordersController.<Postcustomer_orders>d__10.MoveNext() in E:\\NeXt\\repos\\_NextCode\\WCF\\WCF_POS\\Controllers\\customer_ordersController.cs:line 435"""

CODE : 

public async Task<string> PublishRabbitMessage(string UserName = "NeXtClient", string Password = "1")
        {
            try
            {
                var factory = new ConnectionFactory()
                {
                    //Uri = new Uri("amqp://NeXtClient:1...@10.1.0.6:5672/"),
                    HostName = "MySQL.isPlatform.local",
                    Port = 63672,
                    UserName = UserName,
                    Password = Password,
                    VirtualHost = "/%2f",
                    RequestedHeartbeat = DateTime.Now.AddSeconds(60) - DateTime.Now,
                    //Ssl =
                    //{
                    //    ServerName = rabbitMqHostName,
                    //    Enabled = false
                    //}
                };
                using (var connection = factory.CreateConnection())
                using (var channel = connection.CreateModel())
                {
                    channel.ExchangeDeclare(exchange: "logs", type: ExchangeType.Fanout);

                    string message = "Hello World!";
                    var body = Encoding.UTF8.GetBytes(message);
                    channel.BasicPublish(exchange: "logs",
                                         routingKey: "",
                                         basicProperties: null,
                                         body: body);
                   
                    return $" [x] Sent {message}";
                }
            }
            catch(Exception e)
            {
                return $"Error : {e.Message}";
            }
           
        }

        public async Task<string> PublishRabbitMessageVirtualHost(string UserName = "NeXtClient", string Password = "1")
        {
            try
            {
                var factory = new ConnectionFactory()
                {
                    //Uri = new Uri("amqp://NeXtClient:1...@10.1.0.6:5672/"),
                    HostName = "MySQL.isPlatform.local",
                    Port = 63672,
                    UserName = UserName,
                    Password = Password,
                    VirtualHost = "/NeXt",
                    RequestedHeartbeat = DateTime.Now.AddSeconds(60) - DateTime.Now,
                    //Ssl =
                    //{
                    //    ServerName = rabbitMqHostName,
                    //    Enabled = false
                    //}
                };
                using (var connection = factory.CreateConnection())
                using (var channel = connection.CreateModel())
                {
                    channel.ExchangeDeclare(exchange: "logs", type: ExchangeType.Fanout);

                    string message = "Hello World!";
                    var body = Encoding.UTF8.GetBytes(message);
                    channel.BasicPublish(exchange: "logs",
                                         routingKey: "",
                                         basicProperties: null,
                                         body: body);

                    return $" [x] Sent {message}";
                }
            }
            catch (Exception e)
            {
                return $"Error : {e.Message}";
            }
        }

        public async Task<string> PublishRabbitMessageURI(string UserName = "NeXtClient", string Password = "1")
        {
            try
            {
                var factory = new ConnectionFactory()
                {
                    Uri = new Uri("amqp://NeXtClient:1...@MySQL.isPlatform.local:63672/%2f"),
                    //HostName = "10.1.0.5",
                    //Port = 5672,
                    //UserName = UserName,
                    //Password = Password,
                    //VirtualHost = "/NeXt",
                    //RequestedHeartbeat = DateTime.Now.AddSeconds(60) - DateTime.Now,
                    //Ssl =
                    //{
                    //    ServerName = rabbitMqHostName,
                    //    Enabled = false
                    //}
                };
                using (var connection = factory.CreateConnection())
                using (var channel = connection.CreateModel())
                {
                    channel.ExchangeDeclare(exchange: "logs", type: ExchangeType.Fanout);

                    string message = "Hello World!";
                    var body = Encoding.UTF8.GetBytes(message);
                    channel.BasicPublish(exchange: "logs",
                                         routingKey: "",
                                         basicProperties: null,
                                         body: body);

                    return $" [x] Sent {message}";
                }
            }
            catch (Exception e)
            {
                return $"Error : {e.Message}";
            }
        }

        public async Task<string> PublishRabbitMessageURI_2(string UserName = "NeXtClient", string Password = "11")
        {
            try
            {
                var factory = new ConnectionFactory()
                {
                    Uri = new Uri("amqp://NeXtClient:1...@MySQL.isPlatform.local:63672/NeXt"),
                    //HostName = "10.1.0.5",
                    //Port = 5672,
                    //UserName = UserName,
                    //Password = Password,
                    //VirtualHost = "/NeXt",
                    //RequestedHeartbeat = DateTime.Now.AddSeconds(60) - DateTime.Now,
                    //Ssl =
                    //{
                    //    ServerName = rabbitMqHostName,
                    //    Enabled = false
                    //}
                };
                using (var connection = factory.CreateConnection())
                using (var channel = connection.CreateModel())
                {
                    channel.ExchangeDeclare(exchange: "logs", type: ExchangeType.Fanout);

                    string message = "Hello World!";
                    var body = Encoding.UTF8.GetBytes(message);
                    channel.BasicPublish(exchange: "logs",
                                         routingKey: "",
                                         basicProperties: null,
                                         body: body);

                    return $" [x] Sent {message}";
                }
            }
            catch (Exception e)
            {
                return $"Error : {e.Message}";
            }
}

[Luke Bakken]

Hi Mike,

Based on the inner exception, it seems like you're running into an issue like this - https://github.com/JoshClose/CsvHelper/issues/1488

You may be able to fix the issue by adding configuration like this to your WCF project:

 https://github.com/JoshClose/CsvHelper/issues/1488#issuecomment-722294003

Without being able to reproduce this issue I can't try out the fix.

Let me know how that goes.

[__]

Luke,

Thanks for the update.  I will certainly update this thread at some point.  I will look into your suggested post, it might be a possible fix, but by time you sent me back this as a potential solution, I had created a work around.  What I ended up doing, is creating a Self-Hosted WebAPI as a Windows Service and all it does is send messages to RabbitMQ.  I did this to take IIS and WCF completely out of the equation.  Then instead I took the function in the baseclass in the WCF services for sending data to RabbitMQ and redirected it to the Windows Service.  That worked 100% fine for me.  It's not ideal, it's Ugly in some ways.  But I have multiple WCF services, and they can now all communicate to that 1 Windows Service for communication to Rabbit MQ. In a way that is a bit cleaner.  Just means I have an extra piece of code to maintain out there and resolve errors on and a potential hacking point.  Soon as my software is launched, I will come back and try your suggestion.  As for right now, I would recommend anyone running an old standard like me and having these problems just to create a new service and move on.  Time cost money, and that was the deciding factor for just moving on and thinking outside of the box.

-Mike