[lwip-users] lwIP performance with FreeRTOS
Chris Ponder
Hi,
I am evaluating RTOS’s and IP stacks and have come across a strange result. I am using the ST32F207 G-EVAL board and ST have a ready-made package for the STM32F207 at http://www.st.com/internet/com/SOFTWARE_RESOURCES/SW_COMPONENT/FIRMWARE/stm32f2x7_eth_lwip.zip. This uses lwIP 1.3.2 and FreeRTOS 6.1.0.
I am evaluating single direction UDP packet send performance and so modified the standalone UDP echo client to send packets of just over 300 bytes repeatedly as fast as it could and clocked nearly 93Mbps average over 10 seconds using WireShark on Windows 7 to capture the packets.
I then switched to the FreeRTOS version of the echo client which contains TCP and UDP tasks. I disabled the TCP task and the pretty flashing LED task and replaced the echo code with the same code I used in the standalone version but using netconn and netbuf. The performance plummeted to barely over 36Mbps, nearly one third of the standalone performance.
The IAR compiler is set for high optimisation and speed. Are there any other performance metrics I should be tweaking such as #defines etc. The UDP task is at tskIDLE_PRIORITY + 3 and appears to be the only task that is now running, aside from lwIP itself.
Can anyone help?
Many thanks
Chris
********************************************************
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. If you are not the addressee, any disclosure, reproduction,
copying, distribution, or other dissemination or use of this communication is
strictly prohibited. If you have received this transmission in
error please notify the sender immediately and then delete this e-mail.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted lost, destroyed, arrive late or
incomplete, or contain viruses.
The sender therefore does not accept liability for any errors or omissions
in the contents of this message which arise as a result of e-mail
transmission. If verification is required please request a hard copy
version.
********************************************************
Chris Ponder
I have now tried with the latest FreeRTOS 7.1.0 and lwIP 1.4.0. This produced some improvement from 36 Mbps to 37.1 Mbps but this is still not great. Any help would be much appreciated.
Many thanks
Chris
Simon Goldschmidt
I then switched to the FreeRTOS version of the echo client which contains TCP and UDP tasks. I disabled the TCP task and the pretty flashing LED task and replaced the echo code with the same code I used in the standalone version but using netconn and netbuf. The performance plummeted to barely over 36Mbps, nearly one third of the standalone performance.
Chris Ponder
Thanks for the info, that is really helpful. I only have one thread talking to the IP stack so I have been considering using lwIP in that thread and that may speed things up.
I have tried adding TCPIP_CORE_LOCKING to the project pre-processor defines but it has made no difference L, I guess it is all the task switching, it just seems such a performance hit.
Thanks
Chris
********************************************************
Simon Goldschmidt
I have tried adding TCPIP_CORE_LOCKING to the project pre-processor defines but it has made no difference L, I guess it is all the task switching, it just seems such a performance hit.
Chris Ponder
Ahh, that helps J Yes the correct macro is LWIP_TCPIP_CORE_LOCKING and increased the speed to 45.3 Mbps, getting better J
Thanks
Chris
From: Simon Goldschmidt
Sent: 24 February 2012 13:40
To: Mailing list for lwIP users
Subject: Re: [lwip-users] lwIP performance with FreeRTOS
Chris Ponder wrote:
I have tried adding TCPIP_CORE_LOCKING to the project pre-processor defines but it has made no difference L, I guess it is all the task switching, it just seems such a performance hit.
I'm not really sure about the spelling of that macro, you might want to look it up in opt.h to see if you did the right thing.
Simon
gold...@gmx.de
Ahh, that helps J
Hehe, sorry for the confusion. I'm didn't have access to the lwIP sources when writing that.
Yes the correct macro is LWIP_TCPIP_CORE_LOCKING and increased the speed to 45.3 Mbps, getting better J
Simon
Chris Ponder
Lol, that is no problem, it is good to have the help J
Yes that is still a long way from what I had hoped for. I had thought that the drop from standalone+raw to RTOS+netconn would only be 2 or 3 Mbps at most, half seems a little drastic.
Thanks again
Chris
From: gold...@gmx.de
Sent: 24 February 2012 14:35
To: Mailing list for lwIP users
Subject: Re: [lwip-users] lwIP performance with FreeRTOS
Chris Ponder wrote:
********************************************************
FreeRTOS Info
> Lol, that is no problem, it is good to have the help J
>
>
>
> Yes that is still a long way from what I had hoped for. I had thought
> that the drop from standalone+raw to RTOS+netconn would only be 2 or 3
> Mbps at most, half seems a little drastic.
How is the network interface signalling that packets have arrived/been
transmitted?
Normally the MAC interrupt will use a semaphore to signal to a task that
new data has arrived and needs processing. The semaphore will unblock
the task that was waiting for the data, and the task will be prioritised
such that it runs immediately (just as if all the processing was done in
the interrupt itself).
Regards,
Richard.
+ http://www.FreeRTOS.org
Designed for Microcontrollers.
More than 7000 downloads per month.
>
>
>
> Thanks again
>
> Chris
>
>
>
> *From:*gold...@gmx.de
> *Sent:* 24 February 2012 14:35
> *To:* Mailing list for lwIP users
> *Subject:* Re: [lwip-users] lwIP performance with FreeRTOS
> _______________________________________________
> lwip-users mailing list
> lwip-...@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/lwip-users
_______________________________________________
lwip-users mailing list
lwip-...@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users
Chris Ponder
while (1)
{
// Create a netbuf to use and allocate ciPayloadSize of buffer space
buf = netbuf_new();
netbuf_alloc(buf, ciPayloadSize); // ciPayloadSize = 272
// Send some data if the buffer created ok
if (buf!= NULL)
{
netconn_connect(conn, &DestIPAddr, UDP_SERVER_PORT);
netconn_send(conn, buf);
netbuf_delete(buf);
}
}
As you can see I am not even putting any real data in the packet, it is just the empty buffer that is being sent.
The thread containing this code is started with:
sys_thread_new("udpecho_thread", udpecho_thread, NULL, DEFAULT_THREAD_STACKSIZE, UDPECHO_THREAD_PRIO );
and
UDPECHO_THREAD_PRIO = tskIDLE_PRIORITY + 3
Cheers
Chris
Chris Ponder
I'm sorry if I am asking stupid questions but this is my first time with both FreeRTOS and lwIP and there is so much to learn.
Cheers
Chris
Simon Goldschmidt
> Should I be looking more at lwIP (adding LWIP_TCPIP_CORE_LOCKING made quite an improvement) or is this a FreeRTOS thing?
That's a good question, I'm afraid I can't answer it. I haven't used the netconn API when performance was needed, and I haven't ever benchmarked it.
Is the rest of your application the same? In other words, are you using the same pbuf/netbuf type (allocated and creating the data into it vs. sending by reference)? Are there many thread switches from/to your application thread? Keep in mind that response packets are still passed into the tcpip_thread, so there may still be many (potentially slow) thread changes if you are using a ping-pong protocol where every packet sent results in a packet received (no matter how big that's, of course).
As to FreeRTOS, I can't comment on that, I haven't really used it, yet.
Simon
Chris Ponder
Our application is really one of throughput, taking data in and spitting it out as UDP packets on the Ethernet as fast as we can. Unfortunately we do have other tasks that need to run while this is happening. Packets coming the other way number less than 100 for the 20,000 to 30,000 typical up-stream packet count.
Having said that, I am expecting performance degradation from the down-stream packets and other tasks. Hence my desire to ensure we have acceptable performance before introducing those elements. I have been wondering if there is a way of using the pbuf interface and encapsulating the whole lwIP plus send code in a single thread, but so far this has proved an elusive target.
Chris
-----Original Message-----
From: Simon Goldschmidt
Sent: 27 February 2012 18:34
To: Mailing list for lwIP users
Subject: Re: [lwip-users] lwIP performance with FreeRTOS
Chris Ponder wrote:
Simon
********************************************************
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. If you are not the addressee, any disclosure, reproduction,
copying, distribution, or other dissemination or use of this communication is
strictly prohibited. If you have received this transmission in
error please notify the sender immediately and then delete this e-mail.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted lost, destroyed, arrive late or
incomplete, or contain viruses.
The sender therefore does not accept liability for any errors or omissions
in the contents of this message which arise as a result of e-mail
transmission. If verification is required please request a hard copy
version.
********************************************************
Simon Goldschmidt
> I have been wondering if
> there is a way of using the pbuf interface and encapsulating the whole lwIP
> plus send code in a single thread, but so far this has proved an elusive
> target.
You can let the code that creates pbufs (and sends them) run in tcpip_thread by calling tcpip_callback(), e.g. when your interrupt (or whatever it is) detects that data is to be sent, just call tcpip_callback() with a function pointer and the function pass as parameter runs in tcpip_thread.
When doing this from an ISR, you only have one single task switch from the ISR to tcpip_thread. Of course RX packets are not processed while doing this, but from what you wrote, that should be OK.
Oh, and while looking at the lwIP sources, I saw you'd better call netconn_sendto() instead of netconn_connect() and netconn_send(), as it is implemented with only 1 call into the core instead of 2.
Another thing to speed up might be cumulating TX-finished interrupts (if you use them) and generally checking your netif driver is as fast as it can be (are you using DMA?).
Simon
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
Simon Goldschmidt
> Oh, and while looking at the lwIP sources, I saw you'd better call
> netconn_sendto() instead of netconn_connect() and netconn_send(), as it is
> implemented with only 1 call into the core instead of 2.
Plus I jstu noticed that netconn_connect() doesn't use the core-locking feature but uses the traditional message-passing (don't ask me why), so that's no even sped up by enabling core-locking.
Simon Goldschmidt
> "Simon Goldschmidt" <gold...@gmx.de> wrote:
> > Oh, and while looking at the lwIP sources, I saw you'd better call
> > netconn_sendto() instead of netconn_connect() and netconn_send(), as it
> is
> > implemented with only 1 call into the core instead of 2.
>
> Plus I jstu noticed that netconn_connect() doesn't use the core-locking
> feature but uses the traditional message-passing (don't ask me why), so
> that's no even sped up by enabling core-locking.
Chris, I've pushed a fix for that to git master which you can try (it should also speed up sending as I directly call the send function now instead of calling via a function pointer). That is, I don't know if you can switch to git master or are already using it?
Chris Ponder
With sendto, empty packet sending rose to 81.99Mbps which is much better. It occurred to me that we actually only have one destination and once set it rarely changes so I switched back to netconn_send/connect but moved the connect outside the main send loop, performance was 82.13 Mbps, re-instating the packet copy used in the standalone benchmark dropped this to 75.3Mbps.
I am going to try the tcpip_callback() next.
Any chance you could send those files to me directly, my company's web policy is stopping me downloading the tar.gz file from the git repo and TortoiseGIT, so I can't actually get them :( I assume that they are ok to use with 1.4.0?
Cheers
Chris
********************************************************
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. If you are not the addressee, any disclosure, reproduction,
copying, distribution, or other dissemination or use of this communication is
strictly prohibited. If you have received this transmission in
error please notify the sender immediately and then delete this e-mail.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted lost, destroyed, arrive late or
incomplete, or contain viruses.
The sender therefore does not accept liability for any errors or omissions
in the contents of this message which arise as a result of e-mail
transmission. If verification is required please request a hard copy
version.
********************************************************
Mason
Chris,
Could you please make you e-mail client thread correctly?
--
Regards.
Chris Ponder
What is it that you want the email client to do that it is not doing, or what is it doing that it shouldn't?
Chris
Chris Ponder
Tritech International Limited
Morecambe Road, Ulverston Cumbria LA12 0BH
++ 44 (0)1229 586672 (tel)
++ 44 (0)1229 586696 (fax)
DDI:++ 44 (0)(0)1229 484261
Tritech International Limited is registered in Scotland.
Registered number: 85501. Registered office: Tritech International Ltd, Peregrine Road, Westhill, Aberdeen.
This message (and any associated files) is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tritech International Limited. Tritech International Limited is a Halma Plc company.
please consider the environment before printing this e-mail
-----Original Message-----
From: Mason
Sent: 28 February 2012 12:18
To: Mailing list for lwIP users
Subject: Re: [lwip-users] lwIP performance with FreeRTOS
Chris Ponder wrote: [snip]
Chris,
--
Regards.
********************************************************
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. If you are not the addressee, any disclosure, reproduction,
copying, distribution, or other dissemination or use of this communication is
strictly prohibited. If you have received this transmission in
error please notify the sender immediately and then delete this e-mail.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted lost, destroyed, arrive late or
incomplete, or contain viruses.
The sender therefore does not accept liability for any errors or omissions
in the contents of this message which arise as a result of e-mail
transmission. If verification is required please request a hard copy
version.
********************************************************
Simon Goldschmidt
> Any chance you could send those files to me directly, my company's web
> policy is stopping me downloading the tar.gz file from the git repo and
> TortoiseGIT, so I can't actually get them :( I assume that they are ok to use
> with 1.4.0?
Sending the files is not a problem, but they aren't compatible to 1.4.0 :-(
I'll backport that fix to 1.4.1, so you can update to 1.4.1 RC1 (which will be downloadable from our download section) when it is released, but that might take a week or two.
Anyway, I don't expect performance to rise too much with these changes: after moved to not calling netconn_connect() too often, the only additional speedup would be calling do_send() directly instead of by pointer. You can try to change that yourself in your sources by looking at the diff in git web access:
http://git.savannah.gnu.org/cgit/lwip.git/commit/?id=4fca628d36c44f927e9326e43e2408f4d886f258
Simon
--
NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
Mason
> What is it that you want the email client to do that it is not doing,
> or what is it doing that it shouldn't?
Your latest message (this one I am replying to) threads correctly,
as it provides a valid "References:" header field.
Your previous messages, on the other hand, did not provide any
such header field, and were thus displayed as new threads.
Chris Ponder
-----Original Message-----
From: Mason
Sent: 28 February 2012 12:54
To: Mailing list for lwIP users
Subject: Re: [lwip-users] lwIP performance with FreeRTOS
Chris Ponder wrote:
--
Regards.
********************************************************
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. If you are not the addressee, any disclosure, reproduction,
copying, distribution, or other dissemination or use of this communication is
strictly prohibited. If you have received this transmission in
error please notify the sender immediately and then delete this e-mail.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted lost, destroyed, arrive late or
incomplete, or contain viruses.
The sender therefore does not accept liability for any errors or omissions
in the contents of this message which arise as a result of e-mail
transmission. If verification is required please request a hard copy
version.
********************************************************
Mason
> So is this one correct or not?
Nope, no References header.
It seems Microsoft could not care less about standards...
(I wish people didn't use their tools. Sigh.)
Chris Ponder
Chris
Chris Ponder wrote:
Nope, no References header.
--
Regards.
********************************************************
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. If you are not the addressee, any disclosure, reproduction,
copying, distribution, or other dissemination or use of this communication is
strictly prohibited. If you have received this transmission in
error please notify the sender immediately and then delete this e-mail.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted lost, destroyed, arrive late or
incomplete, or contain viruses.
The sender therefore does not accept liability for any errors or omissions
in the contents of this message which arise as a result of e-mail
transmission. If verification is required please request a hard copy
version.
********************************************************
Chris Ponder
Using tcpip_callback only gave 65.4 Mbps :(
...
/* Infinite loop */
while (1)
{
// Store then increment the packet count
*puiPacketNumber = uiPacketCount++;
szPacket[4] = (unsigned char)NULL;
tcpip_callback(udpSendData, szPacket);
}
...
void udpSendData(void *arg)
{
char *szPacket = (char*)arg;
struct pbuf *p;
/* allocate pbuf from pool*/
p = pbuf_alloc(PBUF_TRANSPORT, ciPayloadSize, PBUF_POOL);
if (p != NULL)
{
/* copy data to pbuf */
pbuf_take(p, (char*)szPacket, ciPayloadSize);
/* send udp data */
udp_send(upcb, p);
/* free pbuf */
pbuf_free(p);
}
}
Cheers
Chris
Chris Ponder
Tritech International Limited
Morecambe Road, Ulverston Cumbria LA12 0BH
++ 44 (0)1229 586672 (tel)
++ 44 (0)1229 586696 (fax)
DDI:++ 44 (0)(0)1229 484261
Tritech International Limited is registered in Scotland.
Registered number: 85501. Registered office: Tritech International Ltd, Peregrine Road, Westhill, Aberdeen.
This message (and any associated files) is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tritech International Limited. Tritech International Limited is a Halma Plc company.
please consider the environment before printing this e-mail
-----Original Message-----
From: Simon Goldschmidt
Sent: 28 February 2012 10:24
To: Mailing list for lwIP users
Subject: Re: [lwip-users] lwIP performance with FreeRTOS
Chris Ponder wrote:
> I have been wondering if
> there is a way of using the pbuf interface and encapsulating the whole
> lwIP plus send code in a single thread, but so far this has proved an
> elusive target.
You can let the code that creates pbufs (and sends them) run in tcpip_thread by calling tcpip_callback(), e.g. when your interrupt (or whatever it is) detects that data is to be sent, just call tcpip_callback() with a function pointer and the function pass as parameter runs in tcpip_thread.
When doing this from an ISR, you only have one single task switch from the ISR to tcpip_thread. Of course RX packets are not processed while doing this, but from what you wrote, that should be OK.
Oh, and while looking at the lwIP sources, I saw you'd better call netconn_sendto() instead of netconn_connect() and netconn_send(), as it is implemented with only 1 call into the core instead of 2.
Another thing to speed up might be cumulating TX-finished interrupts (if you use them) and generally checking your netif driver is as fast as it can be (are you using DMA?).
Simon
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
_______________________________________________
Simon Goldschmidt
> Using tcpip_callback only gave 65.4 Mbps :(
> ...
> /* Infinite loop */
I'm not sure using an infinite loop is the best example: don't you call tcpip_callback too often? Also, you don't check the return value...
I'd rather let your ISR check if a call to 'udpSendData()' is already queued and if so, only set a flag for udpSendData() to loop (maybe with a maximum loop count to prevent RX from stopping. On the other hand, using that technique you risk using 100% CPU power for sending...
> while (1)
> {
> // Store then increment the packet count
> *puiPacketNumber = uiPacketCount++;
> szPacket[4] = (unsigned char)NULL;
>
> tcpip_callback(udpSendData, szPacket);
> }
> ...
>
> void udpSendData(void *arg)
> {
> char *szPacket = (char*)arg;
> struct pbuf *p;
>
> /* allocate pbuf from pool*/
> p = pbuf_alloc(PBUF_TRANSPORT, ciPayloadSize, PBUF_POOL);
Pay attentention here to the fact that you now a) use PBUF_POOL instead of PBUF_RAM so you might end with a pbuf-chain instead of a single pbuf and...
>
> if (p != NULL)
> {
> /* copy data to pbuf */
> pbuf_take(p, (char*)szPacket, ciPayloadSize);
b) pbuf_take() calls memcpy (maybe multiple times) which might slow down the whole thing.