Unhandled Exception (Application Crash) on Network Change

[AKarimi]

Greetings,

Unhandled Excpetion and application crash occurs (only) when I switch network meanwhile application is rurning. This can be switching between wireless networks, wireless to lan or lan to wireless.

This happens both in case of Beacon but also for XPublisher & Subscriber. NetMQPoller is being used in all cases, i.e. inconjunction with XPublisherSocket, SubscriberSocket, and NetMQBeacon. 

Any help on best way to monitor and act on network change and reset proparly to the new network would greatly be appreciated. 

Example of Exception:

The process was terminated due to an unhandled exception. Exception Info: System.Net.Sockets.SocketException 

at System.Net.Sockets.Socket.SendTo(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.EndPoint) 

at NetMQ.NetMQBeacon+Shim.SendUdpFrame(NetMQ.NetMQFrame) 

at NetMQ.NetMQBeacon+Shim.PingElapsed(System.Object, NetMQ.NetMQTimerEventArgs) 

at NetMQ.NetMQTimer.InvokeElapsed(System.Object) 

at NetMQ.NetMQPoller.Run() 

at NetMQ.NetMQBeacon+Shim.Run(NetMQ.Sockets.PairSocket) 

at NetMQ.NetMQActor.RunShim() 

at System.Threading.ThreadHelper.ThreadStart_Context(System.Object) 

at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 

at System.Threading.ThreadHelper.ThreadStart()  

The process was terminated due to an unhandled exception. Exception Info: System.ArgumentOutOfRangeException 

at System.ThrowHelper.ThrowArgumentOutOfRangeException(System.ExceptionArgument, System.ExceptionResource) 

at NetMQ.Core.Patterns.Utils.ArrayExtensions.Swap[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Collections.Generic.List`1<System.__Canon>, Int32, Int32) 

at NetMQ.Core.Patterns.Utils.Distribution.Activated(NetMQ.Core.Pipe) 

at NetMQ.Core.Patterns.XSub.XWriteActivated(NetMQ.Core.Pipe) 

at NetMQ.Core.SocketBase.WriteActivated(NetMQ.Core.Pipe) 

at NetMQ.Core.Pipe.ProcessActivateWrite(Int64) 

at NetMQ.Core.ZObject.ProcessCommand(NetMQ.Core.Command) 

at NetMQ.Core.SocketBase.ProcessCommands(Int32, Boolean) 

at NetMQ.Core.SocketBase.GetSocketOption(NetMQ.Core.ZmqSocketOption) 

at NetMQ.NetMQSelector.Select(Item[], Int32, Int64) at NetMQ.NetMQPoller.Run() 

at System.Threading.ThreadHelper.ThreadStart_Context(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 

at System.Threading.ThreadHelper.ThreadStart() 

BR,

Ali

[Doron Somech]

Can you try latest pre release?

--
You received this message because you are subscribed to the Google Groups "netmq-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netmq-dev+unsubscribe@googlegroups.com.
To post to this group, send email to netm...@googlegroups.com.
Visit this group at https://groups.google.com/group/netmq-dev.
For more options, visit https://groups.google.com/d/optout.

[AKarimi]

It did not work with latest NuGet Pre release neither. Same issue and same error message (Event Viewer) as before.

I haven't had the chance yet to build from master locally and test but I guess that the pre release contains already what is included in the master branch.

The process was terminated due to an unhandled exception. Exception Info: System.Net.Sockets.SocketException 

at System.Net.Sockets.Socket.SendTo(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.EndPoint) 

at NetMQ.NetMQBeacon+Shim.SendUdpFrame(NetMQ.NetMQFrame) 

at NetMQ.NetMQBeacon+Shim.PingElapsed(System.Object, NetMQ.NetMQTimerEventArgs) 

at NetMQ.NetMQTimer.InvokeElapsed(System.Object) at NetMQ.NetMQPoller.Run() 

at NetMQ.NetMQBeacon+Shim.Run(NetMQ.Sockets.PairSocket) 

at NetMQ.NetMQActor.RunShim() at System.Threading.ThreadHelper.ThreadStart_Context(System.Object) 

at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 

at System.Threading.ThreadHelper.ThreadStart() 

To unsubscribe from this group and stop receiving emails from it, send an email to netmq-dev+...@googlegroups.com.

[Doron Somech]

Ok, I think the solution is to catch the exception on the SendUdpFrame and check if the SocketError is network changed or down. If so we can just not rethrow it and on the next iteration it will probably work.

https://github.com/zeromq/netmq/blob/master/src/NetMQ/NetMQBeacon.cs#L243

Send a pull request and I will release a pre-release to nuget.

To unsubscribe from this group and stop receiving emails from it, send an email to netmq-dev+unsubscribe@googlegroups.com.

[AKarimi]

Might be a stupid question but how do you intend to check for network change with SocketErrorCode?  I guess this is what your expecting as part of the pull request

                try

                {

                    m_udpSocket.SendTo(frame.Buffer, 0, frame.MessageSize, SocketFlags.None, m_broadcastAddress);

                }

                catch (SocketException ex)

                {

                    if (ex.SocketErrorCode != SocketError.NetworkDown) throw;

                }

[Doron Somech]

Yes, that the change. From your stacktrace it seems the place throwing the exception. I think network it is network down, lets try and see.

To unsubscribe from this group and stop receiving emails from it, send an email to netmq-dev+unsubscribe@googlegroups.com.

[AKarimi]

Ok. Thanks you the quick responses. I really appreciate it. I will make the change and send a pull request a.s.a.p

BR,

Ali

[AKarimi]

Doron,

Before creating a pull request I changed the code accordingly and tested locally but it does not help. Now we have different exception (seems to be same place though in one of the cases) 

How do you suggest to proceed? Should I do the pull request anyway?

On ClientX

The process was terminated due to an unhandled exception. Exceptino Info: System.ArgumentNullException

at System.Net.Sockets.Socket.SendTo(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.EndPoint)

at NetMQ.NetMQBeacon+Shim.SendUdpFrame(NetMQ.NetMQFrame)

at NetMQ.NetMQBeacon+Shim.OnPipeReady(System.Object, NetMQ.NetMQSocketEventArgs)

at NetMQ.NetMQSocket.InvokeEvents(System.Object, NetMQ.PollEvents)

at NetMQ.NetMQPoller.Run()

at NetMQ.NetMQBeacon+Shim.Run(NetMQ.Sockets.PairSocket)

at NetMQ.NetMQActor.RunShim()

at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)

at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

at System.Threading.ThreadHelper.ThreadStart()

On ClientY

The process was terminated due to an unhandled exception. Exception Info: System.ArgumentOutOfRangeException 

at System.ThrowHelper.ThrowArgumentOutOfRangeException(System.ExceptionArgument, System.ExceptionResource) 

at System.Collections.Generic.List`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].get_Item(Int32) 

at NetMQ.Core.Patterns.Utils.ArrayExtensions.Swap[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Collections.Generic.List`1<System.__Canon>, Int32, Int32) 

at NetMQ.Core.Patterns.Utils.Distribution.Activated(NetMQ.Core.Pipe) at NetMQ.Core.Patterns.XSub.XWriteActivated(NetMQ.Core.Pipe) 

at NetMQ.Core.SocketBase.WriteActivated(NetMQ.Core.Pipe) at NetMQ.Core.Pipe.ProcessActivateWrite(Int64) 

at NetMQ.Core.ZObject.ProcessCommand(NetMQ.Core.Command) at NetMQ.Core.SocketBase.ProcessCommands(Int32, Boolean) 

at NetMQ.Core.SocketBase.GetSocketOption(NetMQ.Core.ZmqSocketOption) 

at NetMQ.NetMQSocket.GetSocketOption(NetMQ.Core.ZmqSocketOption) 

at NetMQ.NetMQSelector.Select(Item[], Int32, Int64) at NetMQ.NetMQPoller.Run() 

at System.Threading.ThreadHelper.ThreadStart_Context(System.Object) 

at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 

at System.Threading.ThreadHelper.ThreadStart()  

[Doron Somech]

I will take a look on Sunday and try to solve it.

To unsubscribe from this group and stop receiving emails from it, send an email to netmq-dev+unsubscribe@googlegroups.com.

[AKarimi]

Hello Doron,

Did you had the chance to look at this issue? Please let me know if/when you have a version to test. I will gladly verify it in our environment.

Thank you in advance.

BR,

Ali

[Doron Somech]

Hi Ali,

Any chance you know the SocketErrorCode in the socket exception thrown in the message?

It will make my life easier, you can find in the catch of the exception you show in your code.

Just print it to console, so I would know what to handle.

To unsubscribe from this group and stop receiving emails from it, send an email to netmq-dev+unsubscribe@googlegroups.com.

[AKarimi]

Hello Doron,

Sorry for late response.

This is the SocketException that I have received at lesat based on test done now after logging the issue.

SocketError:AddressNotAvailable,  Exception: System.Net.Sockets.SocketException (0x80004005): The requested address is not valid in its context

   at System.Net.Sockets.Socket.SendTo(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, EndPoint remoteEP)

   at NetMQ.NetMQBeacon.Shim.SendUdpFrame(NetMQFrame frame)

Right now I have a try/catch surrounding SendUdpFrame(...), for everything, thus the issue is logged and then it proceeds, otherwise it would have crashed the application.

Another problem that occurs and crashes the application is the following (even with the try/catch existing for above code but not strange due to the fact that it seems to be related to other part of NetMQ)

Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.ArgumentOutOfRangeException 

at System.ThrowHelper.ThrowArgumentOutOfRangeException(System.ExceptionArgument, System.ExceptionResource) 

at System.Collections.Generic.List`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].get_Item(Int32) 

at NetMQ.Core.Patterns.Utils.ArrayExtensions.Swap[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Collections.Generic.List`1<System.__Canon>, Int32, Int32) 

at NetMQ.Core.Patterns.Utils.Distribution.Activated(NetMQ.Core.Pipe) 

at NetMQ.Core.Patterns.XSub.XWriteActivated(NetMQ.Core.Pipe) 

at NetMQ.Core.SocketBase.WriteActivated(NetMQ.Core.Pipe) 

at NetMQ.Core.Pipe.ProcessActivateWrite(Int64) 

at NetMQ.Core.ZObject.ProcessCommand(NetMQ.Core.Command) 

at NetMQ.Core.SocketBase.ProcessCommands(Int32, Boolean) 

at NetMQ.Core.SocketBase.GetSocketOption(NetMQ.Core.ZmqSocketOption) 

at NetMQ.NetMQSocket.GetSocketOption(NetMQ.Core.ZmqSocketOption) 

at NetMQ.NetMQSelector.Select(Item[], Int32, Int64) 

at NetMQ.NetMQPoller.Run() at System.Threading.ThreadHelper.ThreadStart_Context(System.Object) 

at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 

at System.Threading.ThreadHelper.ThreadStart()  

Both issues are happning only when changing network and these 2 issues are the the remaining issues stopping us from using NetMQ in production environment for the moment which is sad :(

I really hope that you will be able to shed some light on these issues.

[AKarimi]

A pull request has been created related to this matter.

BR,

Ali