For example: try creating a local client/server (running on the same
computer) where the server sends the client a fixed amount of data.
Using method A, recv(8192) and sendall( ) with 8192 bytes worth of
data. Do this 100 times. Using method B, recv(1) and sendall( ) with 1
byte worth of data. Do this 819200 times.
If you time both methods, method A has much greater throughput than
method B.
Server:
import socket
import random
import string
import time
HOST = 'localhost'
PORT = 50023
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
string_1 = 'A'
string_8192 = ''.join([random.choice(string.letters + string.digits)
for i in range(8192)])
conn.sendall('Start')
start = time.clock()
total_data = 0
for i in range(0,100):
conn.sendall(string_8192)
total_data += len(string_8192)
print 'Send Speed (Long String): ' + str( total_data / (time.clock() -
start) / 1024 / 1024 ) + ' MB/sec\n\n'
start = time.clock()
total_data = 0
for i in range(0,819200):
conn.sendall(string_1)
total_data += len(string_1)
print 'Send Speed (Short String): ' + str( total_data / (time.clock()
- start) / 1024 / 1024 ) + ' MB/sec'
conn.close()
Client:
import socket
import time
HOST = 'localhost' # The remote host
PORT = 50023 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, 50026))
s.connect((HOST, PORT))
data = s.recv(5)
print 'From Server: ' + data
start = time.clock()
total_data = 0
while total_data < 819200:
data = s.recv(8192)
total_data += len(data)
print 'Receive Speed (Long String): ' + str( total_data /
(time.clock() - start) / 1024 / 1024 ) + ' MB/sec\n\n'
start = time.clock()
total_data = 0
while total_data < 819200:
data = s.recv(1)
total_data += len(data)
print 'Receive Speed (Short String): ' + str( total_data /
(time.clock() - start) / 1024 / 1024 ) + ' MB/sec'
s.close()
Why is it faster to drink a liter of water a cupful at a time than to
drink it out of an eyedropper?
- Brian
Well, lets say you have a situation where you're going to be
alternating between sending large and small chunks of data. Is the
solution to create a NetworkBuffer class and only call send when the
buffer is full, always recv(8192)?
>> > For example: try creating a local client/server (running on the same
>> > computer) where the server sends the client a fixed amount of data.
>> > Using method A, recv(8192) and sendall( ) with 8192 bytes
>> > worth of data. Do this 100 times. Using method B, recv(1) and
>> > sendall( ) with 1 byte worth of data. Do this 819200 times.
>>
>> > If you time both methods, method A has much greater
>> > throughput than method B.
>>
>> Why is it faster to drink a liter of water a cupful at a time than to
>> drink it out of an eyedropper?
> Well, lets say you have a situation where you're going to be
> alternating between sending large and small chunks of data. Is the
> solution to create a NetworkBuffer class and only call send when the
> buffer is full, always recv(8192)?
If you need to send large and small chumks of data, the
solution is to send large and small chunks of data.
--
Grant
Are two 'sends' guaranteed to arrive as at least two 'receives'?
Send-3: xxx
Send-3: yyy
Receive-6: xxxyyy
No. Nor are they guaranteed to arrive as at least most two.
> Send-3: xxx
> Send-3: yyy
> Receive-6: xxxyyy
Can happen, though I think the problem with Dennis's code is the
other way. The recv in
leng = ntoh(socket.recv(2))
might return one byte of data, not two. The latter recv is similar.
--
--Bryan
Buffering can often improve performance, but with the above we'd
to too quick to prematurely jump to an as yet unwarranted conclusion.
You measured the one-large and many-small cases, but not the case
you actually have. You might be able to take various measurements
and generally characterize speed as a function of the number of
calls and the size of the send. I wouldn't be surprised if the
result is well approximated by a time per call plus a time per
byte.
In optimization, guessing is bad. Where you cannot reason with
mathematical rigor, measure. Where you can derive a purely
analytic result, taking the occasional measurements still isn't a
bad idea (but don't tell my algorithms students).
--
--Bryan
> Well, lets say you have a situation where you're going to be
> alternating between sending large and small chunks of data. Is the
> solution to create a NetworkBuffer class and only call send when the
> buffer is full, always recv(8192)?
No need to reinvent the wheel. socket objects already have a makefile
method returning a file-like object, which behaves like a buffered socket.
--
Gabriel Genellina
Newbie question: Can you write to the 'file-like object' a pickle,
and receive it intact-- as one string with nothing else?
I want to know because I want to send two pickles.
That wheel is far from round, and needs some reinvention. Python's
file-like objects do not play nice with lower level calls, which
would be tolerable if they supported some well-defiend high-level
asynchronous I/O, but they do not.
> Newbie question: Can you write to the 'file-like object' a pickle,
> and receive it intact-- as one string with nothing else?
Yes, but there's a world of gotcha's. Sockets do not recognize
record boundaries, and Python's 'pickle' has holes one's enemies
could drive a truck through. Still, you can pickle, write, read,
un-pickle, and get back your data intact.
> I want to know because I want to send two pickles.
"Two pickles" sounds like a tasty snack, but also suggests you may
be playing hopscotch in a minefield. This is a helpful group. Give
us more to go on, and you are likely to receive thousands of
dollars worth of consulting for free.
--
--Bryan
It depends on the situation. How generally applicable is this:
fun ListenerGeneric( port, factory, arrivedfun ):
which calls 'factory' on socketA.accept (and loops again), then
arrivedfun( stringA ) on message complete detection. ?. It should
start itself in a separate thread.
Or is this any better:
for x in connections():
startnewthreadwith x:
for y in messages( x ):
arrivedfun( y )
This looks like a SocketServer + ThreadingMixIn + a RequestHandler (your
factory).
But as B. Olson already pointed, pickles are unsafe. Worse, it's not that
someone could send a specially crafted pickle that could execute some
arbitrary code: you're blindy executing whatever you receive!
xmlrpc may be a good alternative in some cases.
--
Gabriel Genellina
I see. Just transmit a binary and execute that. Make sure to have
handles to your program publicly accessible too. I execute this.