OSError: [Errno 24] Too many open files after trying 1000 loop.create_datagram_endpoints()

[Chris Casilli]

After trying 1000 loop.create_datagram_endpoints() I'm getting this error:
(Note: It goes up to 249 before crashing.)

Traceback (most recent call last):
  File "BLAgent.py", line 19, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/tulip/base_events.py", line 152, in run_until_complete
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/tulip/futures.py", line 240, in result
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/tulip/tasks.py", line 130, in _step
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/tulip/base_events.py", line 397, in create_datagram_endpoint
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/tulip/base_events.py", line 380, in create_datagram_endpoint
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socket.py", line 94, in __init__
OSError: [Errno 24] Too many open files

I understand this might be an OS limit (After all I'm using OS X on a MacBook Pro and think there is a default 256 max file open limit), but after some checking:

Max Files Open:

$sysctl -a | grep files
kern.maxfiles = 12288
kern.maxfilesperproc = 10240
kern.maxfiles: 12288
kern.maxfilesperproc: 10240
kern.num_files: 1795

and further:

Max Sockets Open:

$ sysctl -a | grep somax
kern.ipc.somaxconn: 2048


So, I should theoretically be able to have ~12,000 file and ~2000 sockets open.
Is this just the limit of this small box or is there something else limiting this? (socket.py limit?)
Could anyone check this on a real server box?

Server Code is the example: udp_echo.py
Client code below:
#--------------------------------------------------------------------------------

#!/usr/bin/env python
"""Base Level Agent"""
import argparse
import sys
import tulip
import RPS
import functools

loop = tulip.get_event_loop()

for i in range(0,1000,1):
   
    message = "Message " + str(i)

    t = tulip.async(loop.create_datagram_endpoint(
        functools.partial(RPS.RPS,message,i,i+1), 
        remote_addr = ('127.0.0.1',9999)))

    loop.run_until_complete(t)

loop.run_forever()
#--------------------------------------------------------------------------------
 
#!/usr/bin/env python
import tulip

class RPS:
   
    def __init__(self, message, id, poll_time = 5):
       
       self.message = message
       self.id = id
       self.poll_time = poll_time
   
    def connection_made(self, transport):
        self.transport = transport
        # inital call setup
        loop = tulip.get_event_loop()
        loop.call_soon(self.repeat_method)
       
    def repeat_method(self):
        print('polling')
        print('sending "{}"'.format(self.message))
        self.transport.sendto(self.message.encode())
        print('waiting to receive')
        # repeat again after xx seconds
        loop = tulip.get_event_loop()
        loop.call_later(self.poll_time,self.repeat_method)

    def datagram_received(self, data, addr):
        print("I'm %d" %self.id)
        print('received "{}"'.format(data.decode()))
        #self.transport.close()

    def connection_refused(self, exc):
        print('Connection refused:', exc)

    def connection_lost(self, exc):
        print('closing transport', exc)
        loop = tulip.get_event_loop()
        loop.stop()

[Guido van Rossum]

I think the shell sets the 256 limit. Try ulimit -n.

--
--Guido van Rossum (python.org/~guido)

[Chris Casilli]

Yes, That did the trick! Ran it with 5000 loop.create_datagram_endpoints() no problems at all.

[Antoine Pitrou]

On Tue, 27 Aug 2013 16:02:48 -0700 (PDT)
Chris Casilli

<christoph...@gmail.com> wrote:
> Yes, That did the trick! Ran it with 5000 loop.create_datagram_endpoints()
> no problems at all.

Note that you don't *have* to create many UDP endpoints. One per
protocol is normally enough (datagram_received tells you where the
data comes from).

Regards

Antoine.

[Chris Casilli]

Hi, I'm not sure if I understand. Here is what I'm trying to do.
I have to communicate with several different remote addresses each one has a different IPAddress  (same port number though).
Think of communicating with remote sensors about 1000 of them.

So this script is kind of like a reverse server instead of receiving a lot of connections, I have to communicate with a lot of different hosts.  Normally a client would connect to a single host
and send lots of messages to that single host.

I request data from each and collect it in a center location. I've got it working now with the many UDP endpoint solution, but I'm always interested in a better way.

Is there a better way to achieve this?
How would I modify the remote_addr on the protocol?

Thanks again for all you help.
Chris

[Antoine Pitrou]

Le Wed, 28 Aug 2013 07:56:41 -0700 (PDT),
Chris Casilli
<christoph...@gmail.com> a écrit :

>
> So this script is kind of like a reverse server instead of receiving
> a lot of connections, I have to communicate with a lot of different
> hosts. Normally a client would connect to a single host
> and send lots of messages to that single host.

That's not how UDP works. UDP is an unconnected protocol. There is no
"client" or "server", every endpoint has the same role and can talk to
every other endpoint (*).

This is why sendto() takes a remote_addr parameter: an UDP socket can
send data to any remote endpoint. Similarly, datagram_received() tells
you from which remote endpoint you received the datagram, because
anyone can send data to a listening UDP socket (try it!).

(*) module the listening address, of course: if you listen on
"127.0.0.1" rather than "0.0.0.0", you'll only be able to talk to
other local endpoints :)

Regards

Antoine.