Non blocking sockets with select.poll() ?
Maxim Veksler
I'm trying to write a non blocking socket port listener based on
poll() because select is limited to 1024 fd.
Here is the code, it never gets to "I did not block" until I do a
telnet connection to port 10000.
"""
#!/usr/bin/env python
import socket
import select
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setblocking(0)
s.bind(('0.0.0.0', 10000))
s.listen(5)
__poll = select.poll()
__poll.register(s)
__poll.poll()
print "I did not block"
"""
--
Cheers,
Maxim Veksler
"Free as in Freedom" - Do u GNU ?
Jean-Paul Calderone
>Hi,
>
>I'm trying to write a non blocking socket port listener based on
>poll() because select is limited to 1024 fd.
>
>Here is the code, it never gets to "I did not block" until I do a
>telnet connection to port 10000.
>
What were you expecting?
Jean-Paul
Maxim Veksler
I'll try to explain.
I'm doing a little experiment: Capturing the whole tcp 1-65535 range
of the machine, allowing me to connect to the my service on the
machine on every port. I know that it's probably the most dumb thing
to do with TCP/IP communication please don't forget it's an
experiment.
My first attempt was made with select.select please see here
http://article.gmane.org/gmane.comp.python.general/516155/ and the
next attempt for slice-by 1024 which also didn't work, here:
http://article.gmane.org/gmane.comp.python.general/517036/
Now, after I realized I won't be able to implement this using
select.select I turned to select.poll(). My problem now it that I'm
able to register more then 1024 socket fd but now I can't replicate
the behaviour select gave me. When I used select for <1024 sockets, I
was able to telnet to each of the 1024 ports and select.select would
return the proper object to do accept() on it, shortly speaking: no
exception was thrown, you can see the working code in the second link
above. The situation I have now with the code attached below is that
if I try querying one of the registered ports I get the following :
TERM1:
"""
./listener_sockets_range_poll.py
.
.
Asking 10182
Asking 10183
Asking 10184
Asking 10185
Asking 10186
Found 10186
Traceback (most recent call last):
File "./listener_sockets_range_poll.py", line 35, in <module>
conn, addr = nb_active_socket.accept()
File "/usr/lib/python2.5/socket.py", line 167, in accept
sock, addr = self._sock.accept()
socket.error: (11, 'Resource temporarily unavailable')
"""
TERM2:
"""
telnet localhost 10100
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
"""
and the code running is this:
"""
#!/usr/bin/env python
import socket
import select
class PollingSocket(socket.socket):
__poll = select.poll()
def __init__(self, port_number):
self.tcp_port_number = port_number
socket.socket.__init__(self, socket.AF_INET, socket.SOCK_STREAM)
self.setblocking(0)
self.bind(('0.0.0.0', self.tcp_port_number))
self.listen(5)
self.__poll.register(self)
def poll(self, timeout = 0):
return self.__poll.poll(timeout)
def debugPollingSocket(port_num):
print "BIND TO PORT: ", port_num
return PollingSocket(port_num)
all_sockets = map(debugPollingSocket, xrange(10000, 19169))
print "We have this in stock:"
for nb_active_socket in all_sockets:
print nb_active_socket.tcp_port_number
while 1:
for nb_active_socket in all_sockets:
print "Asking", nb_active_socket.tcp_port_number
if nb_active_socket.poll(1):
print "Found", nb_active_socket.tcp_port_number
conn, addr = nb_active_socket.accept()
while 1:
data = conn.recv(1024)
if not data: break
conn.send(data)
conn.close()
"""
> Jean-Paul
Thank you,
Maxim.
Maxim Veksler
> On 5/4/07, Jean-Paul Calderone <exa...@divmod.com> wrote:
> > On Fri, 4 May 2007 13:04:41 +0300, Maxim Veksler <hq4...@gmail.com> wrote:
> > >Hi,
> > >
> > >I'm trying to write a non blocking socket port listener based on
> > >poll() because select is limited to 1024 fd.
> > >
> > >Here is the code, it never gets to "I did not block" until I do a
> > >telnet connection to port 10000.
> > >
> >
> > What were you expecting?
> >
>
> I'll try to explain.
> I'm doing a little experiment: Capturing the whole tcp 1-65535 range
> of the machine, allowing me to connect to the my service on the
> machine on every port. I know that it's probably the most dumb thing
> to do with TCP/IP communication please don't forget it's an
> experiment.
[snip]
I think I got it working now :)
"""
#!/usr/bin/env python
import socket
import select
class PollingSocket(socket.socket):
def __init__(self, port_number):
self.__poll = select.poll()
self.tcp_port_number = port_number
socket.socket.__init__(self, socket.AF_INET, socket.SOCK_STREAM)
self.setblocking(0)
self.bind(('0.0.0.0', self.tcp_port_number))
self.listen(5)
self.__poll.register(self)
def poll(self, timeout = 0):
return self.__poll.poll(timeout)
def debugPollingSocket(port_num):
print "BIND TO PORT: ", port_num
return PollingSocket(port_num)
all_sockets = map(debugPollingSocket, xrange(10000, 19169))
print "We have this in stock:"
for nb_active_socket in all_sockets:
print nb_active_socket.tcp_port_number
while 1:
for nb_active_socket in all_sockets:
print "Asking", nb_active_socket.tcp_port_number
if nb_active_socket.poll(0):
print "Found", nb_active_socket.tcp_port_number
conn, addr = nb_active_socket.accept()
while 1:
data = conn.recv(1024)
if not data: break
conn.send(data)
conn.close()
"""
--
Jean-Paul Calderone
This will only handle one connection at a time, of course. The polling
it does is also somewhat inefficient. Perhaps that's fine for your use
case. If not, though, I'd suggest this version (untested):
from twisted.internet import pollreactor
pollreactor.install()
from twisted.internet import reactor
from twisted.protocols.wire import Echo
from twisted.internet.protocol import ServerFactory
f = ServerFactory()
f.protocol = Echo
for i in range(10000, 19169):
reactor.listenTCP(i, f)
reactor.run()
This will handle traffic from an arbitrary number of clients at the same
time and do so more efficiently than the loop in your version. You can
also try epollreactor instead of pollreactor, if the version of Linux you
are using supports epoll, for even better performance.
Jean-Paul
Maxim Veksler
Actually, I'm here to learn. I could have used any number of different
approaches to accomplish this; starting from
http://docs.python.org/lib/module-asyncore.html to twisted to
http://oss.coresecurity.com/projects/pcapy.html.
I would appreciate it if you could elaborate on why my loop is
inefficient, I will try to improve it then (and post back).
Besides, this whole story started from me writing a "quick
totalitarian" security testing framework. Once I'm done with the
networking part I will start working on the part that kill's all
current processes listening on TCP/IP of the machine. Obviously thats
not meant for production boxes... The simple idea is having the poller
on one side of the firewall connection and the "monster" on the other
side replying, a kind of primitive and plain firewall testing utility.
> from twisted.internet import pollreactor
> pollreactor.install()
>
> from twisted.internet import reactor
> from twisted.protocols.wire import Echo
> from twisted.internet.protocol import ServerFactory
>
> f = ServerFactory()
> f.protocol = Echo
> for i in range(10000, 19169):
> reactor.listenTCP(i, f)
> reactor.run()
>
> This will handle traffic from an arbitrary number of clients at the same
> time and do so more efficiently than the loop in your version. You can
> also try epollreactor instead of pollreactor, if the version of Linux you
> are using supports epoll, for even better performance.
>
Thanks!
> Jean-Paul
Jean-Paul Calderone
>On 5/4/07, Jean-Paul Calderone <exa...@divmod.com> wrote:
>> >"""
>> >#!/usr/bin/env python
>> >import socket
>> >import select
>> >
>> >class PollingSocket(socket.socket):
>> >
Usually there's no particularly good reason to subclass socket. In this
case you did it to add a poll method, it seems. However...
>> >
>> > def __init__(self, port_number):
>> > self.__poll = select.poll()
>> > self.tcp_port_number = port_number
>> >
>> > socket.socket.__init__(self, socket.AF_INET, socket.SOCK_STREAM)
>> > self.setblocking(0)
>> > self.bind(('0.0.0.0', self.tcp_port_number))
>> > self.listen(5)
>> > self.__poll.register(self)
Creating one poll object per socket isn't necessary. You only need one
overall.
>> >
>> > def poll(self, timeout = 0):
>> > return self.__poll.poll(timeout)
Polling with a zero timeout is just busy looping. You want to poll with a
large (or infinite) timeout, and on more than one socket at a time.
>> >
>> >def debugPollingSocket(port_num):
>> > print "BIND TO PORT: ", port_num
>> > return PollingSocket(port_num)
>> >
>> >all_sockets = map(debugPollingSocket, xrange(10000, 19169))
>> >
>> >print "We have this in stock:"
>> >for nb_active_socket in all_sockets:
>> > print nb_active_socket.tcp_port_number
>> >
>> >while 1:
>> > for nb_active_socket in all_sockets:
>> > print "Asking", nb_active_socket.tcp_port_number
>> > if nb_active_socket.poll(0):
>> > print "Found", nb_active_socket.tcp_port_number
>> > conn, addr = nb_active_socket.accept()
>> > while 1:
>> > data = conn.recv(1024)
>> > if not data: break
>> > conn.send(data)
>> > conn.close()
>> >"""
>> >
Instead of looping over all the sockets repeatedly and asking each one if
it has any data, you want to create one poll object, register every socket
with it, and ask it which sockets have data. You also want to put the
accepted sockets into non-blocking mode and also poll them, reading from
whichever ones have data when they have data.
Even if you don't want to use Twisted, it might be beneficial to read how
it is implemented to learn how to do this.
Jean-Paul