NetMQ holding onto open bindings after application exit - How can we forcefully close those bindings

[Chris Whiten]

I have an application that is using NetMQ for pushing some data out to consumer processes.  On some miraculous occasions, this application crashes.  It is set up as a Windows service that automatically restarts after these crashes.  On service start, we bind with the following code:

        private static void InitializePublisher(string _queueAddress)
        {
            ctx = NetMQContext.Create();
            netmqPublisher = ctx.CreatePushSocket();
            netmqPublisher.Bind(_queueAddress);
            netmqPublisher.Options.SendHighWatermark = 50;
            netmqPublisher.Options.ReceivevHighWatermark = 50;
        }

When the service stops through expected behaviour, we unbind and dispose the connection correctly.  However, we have seen a few occasions where the service crashes and on restart it is unable to bind to that same address, since the crashed service is not letting go of its binding.  Here is the exception we are seeing: 

 

NetMQ.NetMQException ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at NetMQ.zmq.TcpListener.SetAddress(String addr)
   --- End of inner exception stack trace ---
   at NetMQ.zmq.TcpListener.SetAddress(String addr)
   at NetMQ.zmq.SocketBase.Bind(String addr)
   at NetMQ.zmq.ZMQ.Bind(SocketBase s, String addr)
   at DataAccessManager.InitializePublisher(String _queueAddress)

Assuming we have control of the machine and know that the address should be available for use, is there a way to kill any open bindings on a given address so that we can bind to it? 

[Doron Somech]

I think the crashed process might still be running and takes some more time until it will stopped completely.

Try to find the crashed process and kill it or wait for it to stop.

--
You received this message because you are subscribed to the Google Groups "netmq-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netmq-dev+...@googlegroups.com.
To post to this group, send email to netm...@googlegroups.com.
Visit this group at http://groups.google.com/group/netmq-dev.
For more options, visit https://groups.google.com/d/optout.

[Chris Whiten]

I suspected that, but from the Windows Task Manager, that process was nowhere to be found.  It's particularly annoying because I can't reproduce it reliably, but it's happened multiple times now.

[Doron Somech]

Winsock should release it automatically when the application is down, it might take some more time. Although I still think the process is not down yet. 

After the bind failed, do you need to restart the machine or just to start the service again?

[Chris Whiten]

It seems like the machine needs to be restarted afterward.  The workaround right now is to make the port configurable and have the user manually go change the port, but that is far from ideal.  It would be nice to be able to run a command on that SocketException that kills that binding somehow, but I don't know if such a thing is possible.

[Doron Somech]

I have a work around, I'm not a big fan of it but I'm pretty sure it will solve your problem:

On TcpListener, line 187 (before the bind) add the following

m_handle.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true).

If you want to make a pull request I think it worth having a SocketOption to enable/disable it (and to disable by default) because I'm not sure how it can affect others.

[Oren Eini (Ayende Rahien)]

The actual problem is likely that the there is socket reuse time

See TIME_WAIT 

Fuller explanation is here: 

http://hea-www.harvard.edu/~fine/Tech/addrinuse.html

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

[Jay Shelton]

I'm not sure that this is an issue with NetMQ as much as an implementation issue. Dispose should be called on netmqPublisher before exiting the application. In the worst case scenario, this could be addressed by an app level exception handler by adding the following to your constructor:

AppDomain.CurrentDomain.UnhandledException += (sender, eventArgs) =>

{

// Log eventArgs.ExceptionObject as Exception

netmqPublisher.Dispose();

if (ctx != null)

{

ctx.Dispose();

}

ExitCode = -1;

Environment.Exit(ExitCode);

};

[Oren Eini (Ayende Rahien)]

You can never assume that your process will have a way to clean up resources.

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

--