[lwip-users] LWIP configuration to maximize TCP throughput given RAM constraints
Bruce Sutherland
We are using lwip on an embedded target (LPC2368 microcontroller).
We have successfully used lwip to act as a serial device server. That is to
say, we accept data via a TCP connection on an Ethernet link, and transmit
the data out on an RS-232 serial port.
Now, our goal is to maximize throughput over the link. My impression is that
maximum TCP throughput is limited by lwip configuration options. These
options are mostly constrained by the available memory. This means that
maximum throughput is some function of memory size, capped by any other
bottlenecks in a system, something like this:
http://er6australia.com/images/tcp_thruput_vs_memory.png
My question is: given a certain amount of free RAM, how should lwip options
be configured for best throughput?
Say for example, we have 10 kb of RAM available for lwip to use. How should
we set options such as MEM_SIZE, MEMP_SANITY_CHECK, MEMP_OVERFLOW_CHECK,
TCP_WND, TCP_QUEUE_OOSEQ, TCP_MSS and so on? I know this is quite a broad
question. Links to discussions of this subject would be welcomed.
Regards,
Bruce.
_______________________________________________
lwip-users mailing list
lwip-...@nongnu.org
http://lists.nongnu.org/mailman/listinfo/lwip-users
Piero 74
Bruce Sutherland
Thanks for sharing your file. This is a good starting point for discussion.
A couple of important differences between our system and yours:
1) We do not run an operating system, and define NO_SYS = 1. Hence we do not
use the mailboxes like you do.
2) I think you may be using a slightly different version of LWIP to us. We
are using STABLE-1.3.0.
3) We use the raw API, not the sockets API.
Now, I looked at all your options and how they differ from defaults. Changes
you made which seem most likely to affect TCP performance are as follows:
TCP_MSS increased from 128 to 1460
TCP_SND_BUF correspondingly increased to 1460 * 2
TCP_SND_QUEUELEN increased from 8 to 16
TCP_LISTEN_BACKLOG enabled
TCPIP_THREAD_STACKSIZE set from 0 to 200
TCP_MBOX options set
I will try these in our code and see how performance is affected.
Regards,
Bruce
________________________________
Kieran Mansley
> Now, I looked at all your options and how they differ from defaults. Changes
> you made which seem most likely to affect TCP performance are as follows:
>
> TCP_MSS increased from 128 to 1460
> TCP_SND_BUF correspondingly increased to 1460 * 2
Those are the important ones, together with TCP_WND if you're receiving
data. Making SND_BUF and TCP_WND large enough that you're able to get
enough data onto the network in each round trip time so you (or the
other end) are not waiting for ACKs is the biggest hurdle.
Kieran
Piero 74
2) I think you may be using a slightly different version of LWIP to us. We
are using STABLE-1.3.0.
i use the same version
3) We use the raw API, not the sockets API.
yes, i use socket
bye
piero
Bruce Sutherland
I've done some experimentation, but I am a bit stuck. My configuration
appears below.
I would like to change PBUF_POOL_BUFSIZE from the default of TCP_MSS + 40 +
14, to Piero's value of 128, then increase PBUF_POOL_SIZE as appropriate.
However, when I do this, I find that incoming TCP packets are being
truncated to 74 bytes of data (128 - (40 + 14)).
In my stream receive callback function:
err_t StreamRecvCallback(void *arg, struct tcp_pcb *tpcb, struct pbuf* p,
err_t err);
I always receive only a single buffer in variable p. The p->next field is
always null, although it should point to the next portion of the data. Do I
have an issue in my configuration, or is it likely in my code? Configuration
pasted below.
Thank you,
Bruce.
/* Align memory on 4 byte boundery (32-bit) */
#define MEM_ALIGNMENT 4
/* No operating system present */
#define NO_SYS 1
#define LWIP_SOCKET 0
#define LWIP_NETCONN 0
#define LWIP_CALLBACK_API 1
/* LWIP's self-managed heap size, used for malloc defined in mem.c */
#define MEM_SIZE 2048
// used if doing tcp_write with TCP_WRITE_FLAG_COPY flag
/* LWIP's pool memory, used for structures defined in memp.c */
//#define MEM_USE_POOLS 1
#define MEMP_SANITY_CHECK 1
#define MEMP_OVERFLOW_CHECK 2
#define LWIP_STATS 1
#define IP_STATS 1
#define LWIP_ARP 1
//#define LWIP_UDP 1
#define LWIP_UDP 0
/* Changes to reduce memory usage */
#define MEMP_NUM_PBUF 10 // default 16
//#define MEMP_NUM_UDP_PCB 2 // default 4
#define MEMP_NUM_TCP_PCB 3 // default 6
#define MEMP_NUM_TCP_PCB_LISTEN 4 // default 8
//#define PBUF_POOL_BUFSIZE LWIP_MEM_ALIGN_SIZE(128) //
default TCP_MSS + 40 + 14
#define IP_FRAG 0
/* Changes to improve TCP performance */
//#define TCP_MSS 1460 // default 128
#define TCP_MSS 768 // default 128
#define TCP_SND_BUF (TCP_MSS<<1) // default 256
Mike Kleshov
> 14, to Piero's value of 128, then increase PBUF_POOL_SIZE as appropriate.
In my application I chose to go with a small PBUF_POOL_BUFSIZE and
increased PBUF_POOL_SIZE too. In theory, this should decrease memory
use when you have many small incoming packets. When you have large
incoming packets, extra processing power will be required for chained
pbufs, and memory use will increase due to the overhead of pbuf
headers.
> However, when I do this, I find that incoming TCP packets are being
> truncated to 74 bytes of data (128 - (40 + 14)).
They are not truncated. The packets contain Ethernet headers, IP
headers, TCP headers. So there will be less data in the first pbuf of
a packet.
> In my stream receive callback function:
>
> err_t StreamRecvCallback(void *arg, struct tcp_pcb *tpcb, struct pbuf* p,
> err_t err);
>
> I always receive only a single buffer in variable p. The p->next field is
> always null, although it should point to the next portion of the data. Do I
> have an issue in my configuration, or is it likely in my code? Configuration
> pasted below.
You should look at your Ethernet driver. The pbufs are filled there.
Apparently, your driver expects that pbufs from the pbuf pool are
large enough to hold a complete packet. With smaller pbufs, the driver
should chain them when storing incoming packets.
Bruce Sutherland
I found the problem. I was correctly splitting up the incoming data into a
chain of multiple pbufs were necessary in the Ethernet driver. However, as I
was writing data in my receive callback, which just echoes the incoming
data, I was deallocating the pbufs before I had finished writing them all.
Bruce.
> -----Original Message-----
> From:
> lwip-users-bounces+bruce.sutherland=rfinnovations.com.au@nongn
> u.org
> [mailto:lwip-users-bounces+bruce.sutherland=rfinnovations.com.
> a...@nongnu.org] On Behalf Of Mike Kleshov
> Sent: Tuesday, 21 October 2008 1:41 PM
> To: Mailing list for lwIP users
> Subject: Re: [lwip-users] LWIP configuration to maximize
> TCPthroughputgivenRAM constraints
>