Too many sockets in select, change FD_SETSIZE value

[Rodislav Moldovan]

Hello, I ran into issue these days, about max number of connected clients via socket (websocket) in my haxe/neko application.

So far I can connect max 63 clients x 1 thread  x  1 server loop

the error I see in console is: 

An exception occured in a neko Thread :

Too many sockets in select

After investigation, I've found this: https://github.com/HaxeFoundation/neko/blob/master/libs/std/socket.c#L472

which is related to this: https://support.microsoft.com/en-us/kb/111855

So my question is: Is there a way to change the value for FD_SETSIZE at runtime or via parameters, or somehow otherway !?

[Nicolas Cannasse]

Le 05/09/2016 à 13:05, Rodislav Moldovan a écrit :
> Hello, I ran into issue these days, about max number of connected
> clients via socket (websocket) in my haxe/neko application.
>
> So far I can connect max 63 clients x 1 thread x 1 server loop
> the error I see in console is:
> |
> An exception occured in a neko Thread :
> Too many sockets in select
> |
>
> After investigation, I've found
> this: https://github.com/HaxeFoundation/neko/blob/master/libs/std/socket.c#L472
> which is related to this: https://support.microsoft.com/en-us/kb/111855
>

> So my question is: Is there a way to change the value for *FD_SETSIZE*

> at runtime or via parameters, or somehow otherway !?

You can use neko.net.Poll to create larger set of sockets.

Best,
Nicolas

[Rodislav Moldovan]

Hello, thanks for help

I'll see what I can do with it, because the documentation for Poll does not exist (http://api.haxe.org/neko/net/Poll.html and the old one is not much more better http://old.haxe.org/api/neko/net/poll)

maybe you know where are some real life examples !

Cheers

[Rodislav Moldovan]

I still wonder how Poll can work if it should use Socket from std, and the limit is in the Socket class itself (if( len > FD_SETSIZE ))

I have doubts that Poll will work :-?

[Nicolas Cannasse]

Le 05/09/2016 à 14:23, Rodislav Moldovan a écrit :
> I still wonder how Poll can work if it should use Socket from std, and
> the limit is in the Socket class itself (if( len > FD_SETSIZE ))
> I have doubts that Poll will work :-?

Yes it's not documented, you can see how it's used in Tora server.
https://github.com/HaxeFoundation/tora

Best,
Nicolas

[Rodislav Moldovan]

I see, it'll get complicated :)) the Tora is "most of the things in one file", but I've got the idea.

Thanks

[Rodislav Moldovan]

after reading the Tora's code, and this particular code comment: https://github.com/HaxeFoundation/neko/blob/master/libs/std/socket.c#L847

I've got the idea, Poll and Socket.select can give you the sockes that are reading or writing out of the socket array you provide. Where Socket.select has a limit of FD_SETSIZE (which by default in windows is 64)

so there are 2 solutions, 

either you use Poll, example:

var clientQueue : Deque<Socket> = new Deque();

private function run()
{
    while(true)
    {
         var sock = serv.accept();
            clientQueue.add(sock);

        }

}

private var socks = []; // global socket list
private function readFromSockets():Void

{

var poll:Poll = new Poll(4096); // reserve 4096 positions for sockets upfront

var changed:Bool = false; // a variable that said - hey, your socket list is changed, you have to update it

var localSocks = []; // a local temp variable to hold values from socks

while (true)

{

while (true)

{

var s:Socket = clientQueue.pop(false);

if (s == null)

{

   break;

}

socks.push(s);

changed = true;

}

if(changed)

{

changed = false;

localSocks = socks.concat([]); // we get a fresh copy, so in case socks will change for some reason in other thread or even in this one, we are on the safe side

poll.prepare(localSocks, []);

}

try poll.events(0.05)

catch (e:Dynamic){ changed = true;} // if there is an error, mostly sure some sockets are dead

var arr = [];

var i:Int = 0;

while (true)

{

var idx = poll.readIndexes[i]; // get indexes of sockets that are reading, each index is the position of the socket in localSocks array

if (idx < 0) // because the poll.readIndexes has a size of 4096, look at the constructor, all other values are -1

{

break;

}

arr.push(localSocks[idx]);

i++;

}

for ( s in arr) 

{

try {

readData(socket);

} catch ( e : Dynamic ) {
s.close();

socks.remove(s);

}

}

}

}

either you use Socket.select and send as parameter arrays no longer than 64 items, example to get read sockets:

Socket.select(socks,null,null,updateTime).read