UPD sending perfomance in Gbit Ethernet
JTL
I have written a small testing application that sends as much UDP traffic to
the network as possible. The sending uses Winsock sendto function.
While testing, I've seen very peculiar results. It seems that when using UDP
payload of size 1024 bytes, I've achieved transfer rates of 500+
Mbits/second, but when using payload of size 1025 bytes, the transfer rate
drops dramatically, in some cases to 40 Mbits/second.
This seems to have nothing to do with network MTU. However, network
interface card settings do affect the performance. But, I've tested this
using several NICs, and they all behave in the same way.
Is there some simple explanation to this?
Greetings,
JTL
Michael K. O'Neill
news:6656B1C5-E5D6-4EC6...@microsoft.com...
I don't know the answer, but I have seen your symptoms described elsewhere
in the ng's, for example
http://groups.google.com/groups?q=udp+1024+1025+(%221024+bytes%22+OR+%221025+bytes%22)+sendto+bytes&start=0&hl=en&filter=0
Mike
JTL
Thanks, I found similar articles also. From what I've read and experienced,
I have concluded that the problem is specific to Windows and that it only
appears with UDP - and, it has been around for a while (your link showed
posts that reported the problem in NT SP 3).
I now conclude that the problem is not solely in my own code and that in the
future I should use at most 1024 bytes payload in high-performance UDP
traffic in Windows.
I still think that this is not the optimal situation. Is there any way to
get any further information or explanation about this problem?
Greetings,
Juha
Arkady Frenkel
of 3rd party ( ethereal e.g. ) what happen inside TCP stack in that case
Arkady
"JTL" <J...@discussions.microsoft.com> wrote in message
news:FF7ED4BD-850A-4657...@microsoft.com...
JTL
I have used network sniffer while testing. Could you be more specific about
your question?
I have noticed the following things:
1) When increasing packet size from 1024 to 1025, nothing special happens in
the network, but throughput decreases significantly. That is, the packets are
not fragmented - only their size is bigger.
2) When I increase the packet size from 1025 to up to 10 000 bytes, i see
linear increase in throughput. However, after the MTU the network starts
fragmenting the packets. The peculiar thing is, that fragmenting does not
decrease the throughput - using big, fragmented packets is faster than to use
packets from sizes 1025 to maximum allowed by MTU.
However, I have one thing I want to test as soon as I get equipment for the
test. I'll post my findings after that.
Thank you for your input,
Juha
m
blocking / non-blocking, overlapped / non-overlapped socket? are you
measuring the throughput on the sending or receiving side? how do you
receive the data? what size buffers do you pass when receiving?
FYI: when testing the throughput of an IOCP based server app, I did not see
any performance degradation with datagrams over 1024 bytes (actually my best
results were at about 1300 bytes per datagram)
"JTL" <J...@discussions.microsoft.com> wrote in message
news:D6145BA0-3883-444C...@microsoft.com...
Arkady Frenkel
Arkady
"JTL" <J...@discussions.microsoft.com> wrote in message
news:D6145BA0-3883-444C...@microsoft.com...
JTL
"m" wrote:
> How are you testing this? a single thread in a tight loop?
Yes
> are you using a blocking / non-blocking, overlapped / non-overlapped socket?
Blocking, non-overlapped.
> are you measuring the throughput on the sending or receiving side?
Both, but the issue is the amount of data on the wire i.e. the throughput of
the sending side.
> how do you receive the data? what size buffers do you pass when receiving?
Another tight loop, buffer size is about 10 kilobytes (the same than the
maximum packet size). However, with UDP, it should not matter (to the sender)
that the receiver cannot receive all the data.
> FYI: when testing the throughput of an IOCP based server app, I did not see
> any performance degradation with datagrams over 1024 bytes (actually my best
> results were at about 1300 bytes per datagram)
This is an interesting fact. I have not used IOCP, so this could be an
option. Could you give an estimate of the throughput (performance) you have
achieved?
But, I'd still like to point out Michael K. O'Neill's first reply - this
issue has been detected by others a long time ago.
Thanks for the comments,
Juha
JTL
> I mean time parameters just when you change from 1024 to 1025.
> Arkady
Hi,
This is just the thing I'm about to test. Hopefully I manage to do the test
this week. I'll post results after that.
Greetings,
Juha
Arkady Frenkel
Arkady
"JTL" <J...@discussions.microsoft.com> wrote in message
news:2953D834-3E4F-434B...@microsoft.com...
m
"JTL" <J...@discussions.microsoft.com> wrote in message
news:46972F7A-8706-4348...@microsoft.com...
> Hello,
>
> "m" wrote:
>
>> How are you testing this? a single thread in a tight loop?
>
> Yes
>
>> are you using a blocking / non-blocking, overlapped / non-overlapped
>> socket?
>
> Blocking, non-overlapped.
okay
>> are you measuring the throughput on the sending or receiving side?
>
> Both, but the issue is the amount of data on the wire i.e. the throughput
> of
> the sending side.
FYI: the throughput of the sending side is not equal to the data on the
wire. Packets may be droped before they even leave the sening PC.
>> how do you receive the data? what size buffers do you pass when
>> receiving?
>
> Another tight loop, buffer size is about 10 kilobytes (the same than the
> maximum packet size). However, with UDP, it should not matter (to the
> sender)
> that the receiver cannot receive all the data.
Correct, it dosen't matter to the sender; but if you measure the throughput
at the reciever, then this may matter.
>> FYI: when testing the throughput of an IOCP based server app, I did not
>> see
>> any performance degradation with datagrams over 1024 bytes (actually my
>> best
>> results were at about 1300 bytes per datagram)
>
> This is an interesting fact. I have not used IOCP, so this could be an
> option. Could you give an estimate of the throughput (performance) you
> have
> achieved?
My results were bounded by network hardware (at the time 100 Mb/s full
duplex ethernet). To a single host, the max was a net throughput of about
70% wire speed. To 8 hosts, beter than 90% wire speed. To 16 or more
hosts, basiclly full wire speed (aggragate as mesured at the recipient). In
these tests, the optimal packet size was about 1300 bytes + headers
JTL
> "JTL" <J...@discussions.microsoft.com> wrote:
> > This is an interesting fact. I have not used IOCP, so this could be an
> > option. Could you give an estimate of the throughput (performance) you
> > have
> > achieved?
>
> My results were bounded by network hardware (at the time 100 Mb/s full
> duplex ethernet). To a single host, the max was a net throughput of about
> 70% wire speed. To 8 hosts, beter than 90% wire speed. To 16 or more
> hosts, basiclly full wire speed (aggragate as mesured at the recipient). In
> these tests, the optimal packet size was about 1300 bytes + headers
Hello,
I think this thing is occurring only on gigabit interfaces. I have achieved
wire speeds of > 90%, when I use exactly the same test program on 100 Mbps
interface. The weirdest thing is, that actually on some gigabit interfaces
the throughput is _smaller_ than on 100 megabit interfaces.
See the answer to A. Frenkel for more info.
Greetings,
Juha
JTL
> I mean time parameters just when you change from 1024 to 1025.
> Arkady
Hello,
Now I have some strange results.
I chose a 3COM PCI gigabit adapter (3c2000). It has a setting for "maximum
interrupt rate", which is by default 5000. Furthermore, I added a counter to
the sending side so that I now if the sender drops packets (mainly due to m's
suggestions).
When I use packets of size 1024, I receive them at about 250 Mbits / second.
The number might be a bit small, because the testing equipment was not
state-of-the-art. The packets were received at roughly the same intervals.
When I use packets of size 1025, I receive data at about 40 Mbits / second
(!), and the packets are received almost exactly at intervals of 0,2 ms. That
is, at exactly the rate set from the 3c2000 driver (5000 per second).
When I use packets of size 2800, I receive data at about 80 Mbits / second,
and the UDP packets are fragmented. The first fragment arrives at time zero,
and the second fragment right after that. The first fragment of the next
packet arrives 0,2 ms after the first fragment of the first packet. So, the
wire is idle for about 0,18 ms between the packets.
When I set the interrupts / sec to a amount of 10000, the throughput of the
1024 byte packets is roughly the same. The throughput of 1025 byte packets
roughly doubles, but the packets no longer arrive at equally clear intervals
than with 5000 interrupts / sec.
None of these tests showed any dropped packets, i.e. the receiver received
all the packets that were sent from the sender (this I deduce from my sending
program's packet counter that is embedded in the data packets).
It seems to me that 1024 bytes is some magical number inside the UDP stack
or UDP implementation. What is most peculiar is the case where IP packets are
fragmented.
Any ideas?
Greetings,
Juha
m
the problem is more lily with the NIC or driver than with the stack.
To test this, run a test using the loopback adapter - depending on your PC,
the throughput should be very high, but if there is somthing fundamentally
wrong with the UDP stack (or the test app), then it should show up here.
BTW: The app I talked about was written and tested when 100 Mb/s was fast
(NT4 on PII 266) - they are now running on gagabit networks, but i have
never retested the performance.
"JTL" <J...@discussions.microsoft.com> wrote in message
news:0E6C0319-576A-4612...@microsoft.com...
Arkady Frenkel
> "Arkady Frenkel" wrote:
>
>> I mean time parameters just when you change from 1024 to 1025.
>> Arkady
>
> Hello,
>
> Now I have some strange results.
>
> I chose a 3COM PCI gigabit adapter (3c2000). It has a setting for "maximum
> interrupt rate", which is by default 5000. Furthermore, I added a counter
> to
> the sending side so that I now if the sender drops packets (mainly due to
> m's
> suggestions).
>
> When I use packets of size 1024, I receive them at about 250 Mbits /
> second.
> The number might be a bit small, because the testing equipment was not
> state-of-the-art. The packets were received at roughly the same intervals.
>
how you receive
6 times more throughput or intervals are 0.2/6 ?
TIA
Arkady
JTL
> Hi!
> "JTL" <J...@discussions.microsoft.com> wrote in message
> > When I use packets of size 1024, I receive them at about 250 Mbits /
> > second.
> > The number might be a bit small, because the testing equipment was not
> > state-of-the-art. The packets were received at roughly the same intervals.
> >
> If in both cases interrupt rate the same 40 Mbps seems correct value , so
> how you receive
> 6 times more throughput or intervals are 0.2/6 ?
> TIA
> Arkady
>
> > When I use packets of size 1025, I receive data at about 40 Mbits / second
> > (!), and the packets are received almost exactly at intervals of 0,2 ms.
> > That
> > is, at exactly the rate set from the 3c2000 driver (5000 per second).
> >
Hello,
I guess that the correct way to say it would be that I receive packets at
even intervals. That is, with 5000 interrupts per second:
Size 1024 packets arrive:
1st packet time 0.000000
2nd packet time 0.000033
3rd packet time 0.000069
4th packet time 0.000114
etc, about 30 000+ packets per second
Size 1025 packets arrive:
1st packet time 0.000000
2nd packet time 0.000199
3rd packet time 0.000399
4th packet time 0.000599
etc, practically exactly 5000 packets per second
Size 2800 packets arrive (fragment 1/fragment 2):
1st packet time 0.000000/0.000018
2nd packet time 0.000199/0.000217
3rd packet time 0.000399/0.000418
4th packet time 0.000599/0.000619
etc, practically exactly 10 000 fragments / 5000 UDP packets per second
I hope this clarified my previus description a bit. With 10 000 interrupts
per second, the 1025 and 2800 bytes packets are not arriving at equally
"beautiful" intervals, but still, I'm getting approximately 10 000 packets or
20 000 fragments per second. With size 1024 packets, there is no significant
difference between 5000 and 10 000 interrupts per second.
Greetings and thanks a lot for your input,
Juha
JTL
> If the throughput changes with network card settings (interupt rate), then
> the problem is more lily with the NIC or driver than with the stack.
Hi,
Yes, that would seem likely, but why on earth I see exactly the same
behaviour with each and every NIC I test this with. Currently I have tested
at least the following NICs:
- 3COM 3c2000 adapter
- Intel integrated motherboard NIC
- Intel workstation NIC
- Nvidia NForce integrated motherboard NIC
The throughputs change and the effect of the 1024->1025 transition change,
but it is clearly visible with each NIC. If this is a driver problem, then
I'd like to know if there exists a driver without this "feature".
Still, I'd like to point out the previous discussions about this subject.
All the authors have seen exactly the same behaviour, and I guess they did
not have equal hardware (at least they had different OS).
> To test this, run a test using the loopback adapter - depending on your PC,
> the throughput should be very high, but if there is somthing fundamentally
> wrong with the UDP stack (or the test app), then it should show up here.
OK, another weird discovery:
With uniprocessor system, I see no (significant) effect in the 1024->1025
transition. The troughput decreases very slightly, but the processor
utilization is 100% so I cannot say anything reliable.
With (real) dual processor system, I see this behaviour: with loopback
interface the throughput in 1024->1025 transition drops about 50%.
This was a very fast test, so these results are not 100% reliable.
> BTW: The app I talked about was written and tested when 100 Mb/s was fast
> (NT4 on PII 266) - they are now running on gagabit networks, but i have
> never retested the performance.
I guess that the 'real' problem between GigE and 100 Mbps is that the PCI
bus cannot transfer the data coming from the network. This was the thing I
was about to test, when I discovered the problem under discussion.
Greetings,
Juha
Pavel A.
in microsoft.public.development.device.drivers ?
Regards,
--PA
"JTL" <J...@discussions.microsoft.com> wrote in message news:2680A58B-C13B-4322...@microsoft.com...