System.Net.SocketException: Too many open files and strange Strange Socket being opened

[Dean Roker]

Hi Everyone,

On my server application I get the following error when my server has been running for a long time.

Unhandled Exception: System.Net.Sockets.SocketException: Too many open files
  at System.Net.Sockets.Socket..ctor (AddressFamily family, SocketType type, ProtocolType proto) [0x00000] in <filename unknown>:0
  at AsyncIO.DotNet.NativeSocket..ctor (AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType) [0x00000] in <filename unknown>:0
  at AsyncIO.AsyncSocket.Create (AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType) [0x00000] in <filename unknown>:0
  at NetMQ.Core.Transports.Tcp.TcpListener.Accept () [0x00000] in <filename unknown>:0
  at NetMQ.Core.Transports.Tcp.TcpListener.InCompleted (SocketError socketError, Int32 bytesTransferred) [0x00000] in <filename unknown>:0
  at NetMQ.Core.IOObject.InCompleted (SocketError socketError, Int32 bytesTransferred) [0x00000] in <filename unknown>:0
  at NetMQ.Core.Utils.Proactor.Loop () [0x00000] in <filename unknown>:0
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0

I have traced it back and it seems for every client that connects and disconnects I get 2 entries in lsof that look like this

mono       4797       droker  819u     sock                0,7       0t0     832873 can't identify protocol
mono       4797       droker  820u     sock                0,7       0t0     832872 can't identify protocol

I have looked through my code and I can see nothing that creates any extra sockets when a client connects.

My network layout is like below

Request Socket -> Router Socket -> (For every client that connects a Dealer socket is created) -> Reply Socket (in the server Thread that is started)

Netstat only ever shows 2 connections active for each client (Which is correct)

Any ideas why I would be getting more and more of these open socket files??

Thanks

Dean

[Dean Roker]

Hi Everyone,

I have managed to reproduce this with just a request and router socket in a loop. Please see below. It crashes in a few seconds.

Im running on linux mint 17 with mono 3.12.1 using NetMQ compiled from master.

If you run the code below and type

lsof | grep "can't"

There are several types of sockets that seem to hang around. The more i connect and disconnect, the more of them appear.

The lines look like this
mono      6866          droker   71u     sock                0,7       0t0   150702 can't identify protocol
Finalizer 6866 6868     droker   13u     sock                0,7       0t0   149875 can't identify protocol
mono      6866 6869     droker   13u     sock                0,7       0t0   149875 can't identify protocol
IO        6866 6874     droker   71u     sock                0,7       0t0   150702 can't identify protocol
Threadpoo 6866 6875     droker   13u     sock                0,7       0t0   149875 can't identify protocol

Any ideas?

Here is the code.

using System;
using NetMQ;
using System.Threading.Tasks;
using System.Threading;


namespace Socket_Test
{
    class MainClass
    {
        public static void Main (string[] args)
        {
            using (var context = NetMQContext.Create ())
            using (var router = context.CreateRouterSocket ()) 
            using (var request = context.CreateRequestSocket ())
            {
                var port = router.BindRandomPort ("tcp://127.0.0.1");
                router.Options.Linger = TimeSpan.Zero;
                for (int i = 0; i < 100000; i++) {
                    request.Options.Linger = TimeSpan.Zero;
                    request.Connect ("tcp://127.0.0.1:" + port);
                    Thread.Sleep (TimeSpan.FromSeconds (2));
                    Console.WriteLine ("Sending message");
                    //request.Send ("1", sendMore: false);
                    request.Disconnect ("tcp://127.0.0.1:" + port);
                    Console.WriteLine("Socket Closed");
                }
            }
        }
            
    }
}

[Doron Somech]

Does this also happen if you dispose (instead of disconnect) the request socket or only if you disconnect?

So I think there is a problem with the Disconnect method...

--
You received this message because you are subscribed to the Google Groups "netmq-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netmq-dev+...@googlegroups.com.
To post to this group, send email to netm...@googlegroups.com.
Visit this group at http://groups.google.com/group/netmq-dev.
For more options, visit https://groups.google.com/d/optout.

[Dean Roker]

Same things happens when you dispose the socket too.

Originally I had the request socket be created and disposed inside the loop and the same problem occurred.

I am having a problem with these strange sockets on the client and server end in my application, I believe the mono sockets are a server end problem (From my real app) and the IO, Finallizer and ThreadPool ones are a client side problem.

Thanks

Dean

[Doron Somech]

I think the too many issue is a TIME_WAIT issue, when you dispose a socket it's actually take a little more time until the port is free because of TCP implementation.

Take a look at the following recent pull request that disable that, however it's only disable that for accepted socket, you might want to do it for connecting sockets as well.

https://github.com/zeromq/netmq/pull/304

Bottom line you have a new feature called, DisableTimeWait, which only affect the accepted sockets, I suggest doing the same to the connecting socket which might solve your problem.

Anyway sockets should be long lived, it is expensive to create a socket and connecting.

[Dean Roker]

Hi Doron,

On my server application (not the test code i posted). I do not see any connections in time_wait. but I do get too many open files if a client repeatedly connects and disconnects,  if i run netstat I only see the 2 connections established, when the client disconnects and reconnects I see 2 new connections, but none in time wait, but I still have the problem with the open files.

So I believe the problem effects both ends, not just the client side. I will try and separate out the client and server end of my test code and run them on different machines and look at the behaviour.

Thanks

Dean

[Dean Roker]

Hi Doron,

I haven't had time yet to split out my code yet, but could this be related to this (not a C# thing its a python thing, but read similar things about java and sounds very similar to my probem)
https://idea.popcount.org/2012-12-09-lsof-cant-identify-protocol/

It says you can have a state as CLOSED on the TCP connection but never actually close the socket in the code?

Could it be something to do with the underlying AsyncIO sockets? Rather than being NetMQ specific?

Thanks

Dean

[Dean Roker]

Hi Everyone,

I have been trying to diagnose this, but I am not clever enough to find out what is going on inside NetMQ. I have cloned the NetMQ and AsyncIO repositories to try and work out where the open files are coming from.

I split the code out so the client and server are 2 different applications. I changed the client to use Dispose instead of disconnect and slowed down the rate at which it creates sockets so It doesn't crash straight away.

If I run the Client for a while the number of open files increases and seem to be fairly evenly split between ones that are labeled mono, ThreadPool and IO. If I close the client, the number of open files for the server stays the same indefinitely until I close the server. As soon as the server is closed the number of open files goes to 0.

I can't believe no one else has seen this problem if they have a long running server and clients that come and go (or even poor network conditions). Or could it be specific to Mono on Linux?

Thanks

Dean

Here is my Server code,
using System;
using NetMQ;
using System.Threading;

namespace Socket_Server

{
    class MainClass
    {
        public static void Main (string[] args)
        {
            using (var context = NetMQContext.Create ())

            using (var router = context.CreateResponseSocket ()) 
            {
                router.Options.Linger = TimeSpan.Zero;
                router.Options.DisableTimeWait = true;
                router.Bind("tcp://127.0.0.1:5556");

                for (int i = 0; i < 100000; i++) {

                    Thread.Sleep (TimeSpan.FromSeconds (1));
                }
            }
        }
    }
}


Any my Client Code

using System;
using NetMQ;
using System.Threading.Tasks;
using System.Threading;


namespace Socket_Test
{
    class MainClass
    {
        public static void Main (string[] args)
        {

            using (var context = NetMQContext.Create ()) {

                for (int i = 0; i < 100000; i++) {

                    using (var request = context.CreateRequestSocket ()) {

                        request.Options.Linger = TimeSpan.Zero;
                        request.Connect ("tcp://127.0.0.1:5556");
                        //Thread.Sleep (TimeSpan.FromSeconds (2));

                        Console.WriteLine ("Sending message");
                        //request.Send ("1", sendMore: false);

                        //request.Disconnect ("tcp://127.0.0.1:5556");
                        Console.WriteLine ("Socket Closed");

                        Thread.Sleep (TimeSpan.FromMilliseconds (500));
                    }
                }
                    
            }        
        }
    }
}

[Dean Roker]

I forgot to mention, that the client side has 18 open sockets if the server isn't running and 27 when it is. If you stop the server it goes back down to 18 open, so the problem is only on the server side.

Thanks

Dean

[Dean Roker]

Any ideas on this? I have looked through the NetMQ source and the source for AsyncIO and I can't find anywhere that the underlying socket that NetMQ and AsyncIO io use has the close or shutdown method is called.

I think this might relate to my issue.

Any help with this would be really useful, this is a bit of a show stopper for me at the moment.

Thanks

Dean

[Dean Roker]

This is still causing me problems, I am thinking I might have to re-write my software with a restful API instead of NetMQ.

I have looked over the source code and I am not clever enough to work out what is going on.

Has no one else had this issue? Is no one else using Mono/Linux?

Thanks

Dean