Unix vs inet sockets - why the huge difference?
Skip Montanaro
I was interested in comparing the relative performance of unix and inet
domain sockets. I knew unix sockets would be more efficient when they could
be used, but not by how much, so I wrote a simple server and client to check
things out. (This was how I discovered the problem with the unix domain
socket names. Now that's solved, I'mm looking at the performance issue
again.)
The server simply creates a pair of sockets (one of each kind), then enters
an infinite loop blocking on a select call. When it fields a request, it
reads the input (just 100 characters), replicates it to 1000 characters and
repeats.
The client creates a random 100-character string, sends it to the server,
then reads the results. The socket was shutdown after each send/receive
cycle. This cycle is repeated n times for the inet domain socket, then n
times for the unix domain socket with n varying between 50 and 5000.
The results were quite surprising. I knew the unix domain socket would be
faster, however I was not ready for the fact that the inet domain socket's
performance didn't scale linearly with the number of connections.
The results are shown below. Can anyone explain the non-linear performance
of the inet domain socket?
inet domain socket
n time (seconds) sec/conn
--------------------------------
50 0.180 0.004
100 0.382 0.004
200 0.849 0.004
500 2.712 0.005
1000 7.012 0.007
2000 64.042 0.032
5000 86.653 0.017
unix domain socket
n time (seconds) sec/conn
--------------------------------
50 0.035 0.0007
100 0.070 0.0007
200 0.141 0.0007
500 0.352 0.0007
1000 0.774 0.0008
2000 1.512 0.0008
5000 3.603 0.0007
The Python code for the server and client are available at
http://www.automatrix.com/~skip/python/tsttcpsrvr.py
http://www.automatrix.com/~skip/python/tsttcpclnt.py
should anybody wish to run the test.
Needless to say, I'm using unix domain sockets when I can now...
Skip Montanaro | Today's Acronym: YAUWWWI (yow'-ee) Yet Another Useless
sk...@calendar.com | World-Wide Web Idea
(518)372-5583 | --- Check out Musi-Cal: http://concerts.calendar.com/
Andrew Kuchling
I grabbed Skip's benchmark, and ran it on Linux and IRIX after changing
the output formatting slightly.
Skip's results: (On an unknown OS)
inet 50 0.180 0.004 unix 50 0.035 0.0007
inet 100 0.382 0.004 unix 100 0.070 0.0007
inet 200 0.849 0.004 unix 200 0.141 0.0007
inet 500 2.712 0.005 unix 500 0.352 0.0007
inet 1000 7.012 0.007 unix 1000 0.774 0.0008
inet 2000 64.042 0.032 unix 2000 1.512 0.0008
inet 5000 86.653 0.017 unix 5000 3.603 0.0007
On IRIX:
inet ( 50) 0.23059 0.004612 unix ( 50) 0.05448 0.001090
inet ( 100) 0.46142 0.004614 unix ( 100) 0.11758 0.001176
inet ( 200) 1.01412 0.005071 unix ( 200) 0.22006 0.001100
inet ( 500) 2.95366 0.005907 unix ( 500) 0.56178 0.001124
inet (1000) 8.87049 0.008870 unix (1000) 1.16471 0.001165
inet (2000) 29.10101 0.014551 unix (2000) 2.44060 0.001220
inet (5000) 89.60221 0.017920 unix (5000) 6.01880 0.001204
^^ total ^^ time per connection
time
On Linux:
inet ( 50) 0.29362 0.005872 unix ( 50) 0.06630 0.001326
inet ( 100) 0.62723 0.006272 unix ( 100) 0.12870 0.001287
inet ( 200) 1.45454 0.007273 unix ( 200) 0.26432 0.001322
inet ( 500) 3.57699 0.007154 unix ( 500) 0.76198 0.001524
inet (1000) 7.18793 0.007188 unix (1000) 1.29766 0.001298
inet (2000) 13.17963 0.006590 unix (2000) 2.58206 0.001291
inet (5000) 34.14564 0.006829 unix (5000) 6.63037 0.001326
IRIX also shows the non-linearity, but Linux doesn't seem to;
repeated runs on Linux never showed a clear increase from 50 to 5000
connections. If Skip's OS and IRIX are both derived from the BSD
TCP/IP implementation, this may point to a special case that BSD's
code doesn't handle well: opening and closing many
connections so quickly.
This is an interesting result; should we recode the test in C,
to check if the non-linearity is actually in Python somewhere? I
wouldn't expect it to be, but you never know...
Andrew Kuchling
a...@magnet.com
Andrew Kuchling
In a previous article in comp.lang.python, I wrote:
>connections. If Skip's OS and IRIX are both derived from the BSD
>TCP/IP implementation, this may point to a special case that BSD's
>code doesn't handle well: opening and closing many
>connections so quickly.
Someone corrected me, pointing out that IRIX is SYSV-derived.
So replace "BSD" with "SYSV" in the above paragraph.
Andrew Kuchling
a...@magnet.com
Skip Montanaro
Andrew Kuchling writes:
Skip's results: (On an unknown OS)
If Skip's OS and IRIX are both derived from the BSD
TCP/IP implementation, this may point to a special case that BSD's
code doesn't handle well: opening and closing many
connections so quickly.
Sorry, I forgot to mention my environment is BSDi's BSD/OS 2.0.
Aaron Watters
Skip noticed that inet sockets have nonlinear performance
if you open/close a lot of them very fast:
Andrew Kuchling wrote:
>
> I grabbed Skip's benchmark, and ran it on Linux and IRIX after changing
> the output formatting slightly.
>
> IRIX also shows the non-linearity, but Linux doesn't seem to;
> repeated runs on Linux never showed a clear increase from 50 to 5000
> connections. If Skip's OS and IRIX are both derived from the BSD
> TCP/IP implementation, this may point to a special case that BSD's
> code doesn't handle well: opening and closing many
> connections so quickly.
I asked a TCP expert and he said that what is happening is that
after a socket connection is closed it still requires kernel
resources for about 4 minutes due to some horrible technical
issues having to do with the possible late arrival of packets.
Consequently, if you open/close too many sockets too quickly
the kernel resources get hosed up at some threshhold.
This may indicate that either linux is doing something really
clever or it doesn't exactly conform to the TCP spec. Let's
hope it's the former.
-- Aaron Watters.
===
I'm gonna hafta move out of my nice comfy office in Holmdel
(which is Lucent) and the rumour is that the new place will
have an ``open architecture.'' Now normally I prefer ``open''
to ``closed'' architectures...