[lwip-users] tcp_write() errors on snd_queuelen
Tim Lambrix
I am trying to make sure my device can handle the throughput requirements I have. I am finding that when I start to send more data out, I begin dropping packets on the tcp_write function. I tried playing with many of the settings in the lwipopts.h file and can’t seem to find something that will work continuously.
I have a Cortex-M3 device running as a server and I only allow 1 client to connect. With a light load, everything performs well. However, when I start sending data packets of about 50 bytes every 10 ms, packets start dropping out and the data is lost. In most configurations I have tried, it gets lost due to the snd_queuelen getting too large. I can’t figure out why it is getting so big. I also notice that if I send data for a brief period at a higher rate, that value climbs up and then when I stop sending, it doesn’t always return back down to 0. This makes the next blast of data spin the value up to and over the limit causing data lost. Should it return to 0?
How do I prevent this error from occurring and are there other settings in the options file that I should change to obtain the performance without lost packets? I included some of defines below. I don’t mind if several of the 50 byte packets are combined into 1 TCP packet and I have about 30K of RAM to dedicate to the lwip for good performance. I read the Tuning TCP wiki page but can’t find the right combination of values. Based on this information, can someone offer suggestions or point me to a better resource for my application?
//*****************************************************************************
// ---------- Memory options ----------
#define MEM_ALIGNMENT 4 // default is 1
#define MEM_SIZE (22 * 1024) // default is 1600, was 16K
//*****************************************************************************
// ---------- Internal Memory Pool Sizes ----------
#define MEMP_NUM_PBUF 24 // Default 16, was 16
#define MEMP_NUM_TCP_PCB 1 // Default 5, was 12
//*****************************************************************************
// ---------- TCP options ----------
LWIP_TCP 1
#define TCP_WND 4096 // default is 2048
#define TCP_MSS 1024 // default is 128
#define TCP_SND_BUF (16 * TCP_MSS// default is 256, was 6 *
//#define TCP_SND_QUEUELEN (4 * (TCP_SND_BUF/TCP_MSS))
Kieran Mansley
> I have a Cortex-M3 device running as a server and I only allow 1
> client to connect. With a light load, everything performs well.
> However, when I start sending data packets of about 50 bytes every 10
> ms, packets start dropping out and the data is lost. In most
> configurations I have tried, it gets lost due to the snd_queuelen
> getting too large. I can't figure out why it is getting so big. I
> also notice that if I send data for a brief period at a higher rate,
> that value climbs up and then when I stop sending, it doesn't always
> return back down to 0. This makes the next blast of data spin the
> value up to and over the limit causing data lost. Should it return to
> 0?
Packets are added to the send queue whenever the application calls send.
Packets are removed from the send queue whenever the stack is able to
send them; there are lots of factors that limit this for TCP, e.g.
congestion window, available receive window Nagle's algorithm, etc. I
wouldn't expect the send queue to immediately drop back to zero, but it
should after all the sent data have been acknowledged.
Packets are freed and available for reuse once they are acknowledged by
the other end. You therefore need enough packet buffers to cover the
rate of sends you want to make times the round trip time of the network
(assuming the other end acknowledges fairly promptly, and there isn't
much loss).
Note that I don't really count this as packets getting dropped. You
should get an error from your call to tcp_write() that tells you that it
can't enqueue the data. It's up to your application to deal with it
then (e.g. by blocking if it wants to preserve the data, or dropping it
if it doesn't). Fundamentally an application can generate packets
faster than the network can send them, so something has to give. The
send queue will provide a small amount of smoothing but it's not a
panacea.
> How do I prevent this error from occurring and are there other
> settings in the options file that I should change to obtain the
> performance without lost packets?
Which version of lwIP are you using? I would suggest updating to the
current 1.4.0 release candidate as there was a significant re-write of
the tcp_write/tcp_enqueue path recently and if there was a bug there's a
good chance it has been fixed. There may also be improvements in the
combining of writes together to make a single packet.
There are two limits on the send queue: the number of bytes and the
number of packets. If you're hitting the packet limit (most likely with
your small sends) you could try increasing that.
Kieran
_______________________________________________
lwip-users mailing list
lwip-...@nongnu.org
http://lists.nongnu.org/mailman/listinfo/lwip-users
Tim Lambrix
I do not see the send queue ever return to zero even when the load is removed. For example, I see it start at 0 and I blast a little data where it may climb to 50 and then I stop the volume of data. Usually it returns to 0 within a couple seconds. I blast a little more data and then it climbs again to 40 and I stop again. This time, the value doesn't drop below 6 even after 10 minutes. I blast a little more data and it climbs to 100 and stop again. This time the value won't drop below 63. It never returns to 0. If this doesn't return, does that number represent data not received by the destination? It is hard for me to verify right now as the tcp_write fail doesn't send data.
So it seems the lwip believes there is data someplace and so I am trying to discover the underlying problem. I could just set the value to 0 but I don't know what that would break or why lwip thinks it can't go back down. I was wondering if it was due to a problem with my configuration file with the buffers?
Since my typical data packet is pretty small, about 50 bytes, I figured it would be more efficient to combine my packets into a larger TCP packet. I realize this impacts the latency of my data. What is the best way to configure the TCP window for small and large data packets? Is the window the problem with the send queue?
I will figure out what to do with my data when the tcp_write() fails.
Kieran Mansley
>
> I do not see the send queue ever return to zero even when the load is
> removed. For example, I see it start at 0 and I blast a little data
> where it may climb to 50 and then I stop the volume of data. Usually
> it returns to 0 within a couple seconds. I blast a little more data
> and then it climbs again to 40 and I stop again. This time, the value
> doesn't drop below 6 even after 10 minutes. I blast a little more
> data and it climbs to 100 and stop again. This time the value won't
> drop below 63. It never returns to 0. If this doesn't return, does
> that number represent data not received by the destination? It is
> hard for me to verify right now as the tcp_write fail doesn't send
> data.
>
> So it seems the lwip believes there is data someplace and so I am
> trying to discover the underlying problem. I could just set the value
> to 0 but I don't know what that would break or why lwip thinks it
> can't go back down. I was wondering if it was due to a problem with
> my configuration file with the buffers?
I wouldn't suggest manually setting the value to zero - it will likely
confuse lwIP greatly.
Perhaps a good way to see what is going on would be to make sure that
your calls to tcp_write() have identifiable (human readable) data, e.g.
first write is all zeros, second write all ones, and so on. Then get a
wireshark packet capture to coincide with your tests as described above.
This should show us what is going on the wire, and by deduction what is
queued but not sent. The packet capture may also reveal why lwIP can't
send any more.
Tim Lambrix
I did change the data I am sending to more human readable data and found that all the data is going out in WireShark until the tcp_write returns the error.
I spent a lot of time digging into this and finally found the problem. I am hoping you can help me determine if it is something I am doing wrong or Texas Instruments or lwip.
First, below is a portion of the attached log file from lwip running with the TCP_QLEN_DEBUG enabled.
... [line 3642]
tcp_enqueue: 37 (after enqueued)
tcp_enqueue: queuelen: 37
tcp_enqueue: 38 (after enqueued)
tcp_receive: queuelen 38 ... 23 (after freeing unacked)
tcp_receive: queuelen 23 ... 0 (after freeing unacked)
tcp_enqueue: queuelen: 0
tcp_enqueue: 1 (after enqueued)
tcp_enqueue: queuelen: 1
...
... [line 3764]
tcp_enqueue: 59 (after enqueued)
tcp_enqueue: queuelen: 59
tcp_receive: queuelen 59 ... 35 (after freeing unacked)
tcp_receive: queuelen 35 ... 11 (after freeing unacked)
tcp_enqueue: 60 (after enqueued)
tcp_enqueue: queuelen: 60
tcp_enqueue: 61 (after enqueued)
tcp_enqueue: queuelen: 61
tcp_enqueue: 62 (after enqueued)
tcp_enqueue: queuelen: 62
...
... [line 3853]
tcp_enqueue: queuelen: 102
tcp_enqueue: 103 (after enqueued)
pcb->nrtx > 12
tcp_enqueue: queuelen: 103
tcp_enqueue: 104 (after enqueued)
tcp_enqueue: queuelen: 104
tcp_enqueue: 105 (after enqueued)
tcp_enqueue: queuelen: 105
tcp_enqueue: 106 (after enqueued)
tcp_enqueue: queuelen: 106
tcp_enqueue: 107 (after enqueued)
tcp_receive: queuelen 107 ... 95 (after freeing unacked)
tcp_receive: queuelen 95 ... 71 (after freeing unacked)
tcp_receive: queuelen 71 ... 48 (after freeing unacked)
tcp_receive: valid queue length
tcp_enqueue: queuelen: 48
tcp_enqueue: pbufs on queue => at least one queue non-empty
tcp_enqueue: 49 (after enqueued)
tcp_receive: queuelen 49 ... 48 (after freeing unacked)
tcp_receive: valid queue length
Normally, we see the tcp_receive prefix take out queues and usually to 0 (though not always - line 3241 in file). However, at line 3765, the tcp receive interrupt went off during a tcp_write (tcp_enqueue). We found that in the tcp_write, the queue length is read near the beginning of the function into a local variable and then stored back into the global variable toward the end of the function (see below). From lwip version 1.3.2 in the tcp_out.c file:
Line 195 queuelen = pcb->snd_queuelen;
Line 411 pcb->snd_queuelen = queuelen;
As you can see from line 3765 of the log file, the tcp_receive removed queues from the buffer after the tcp_enqueue had read the value to process. It then sets the queue length at the end to the internally modified local value.
So when we get to line 3866, the queue length is incorrect and the value will not ever get back to a zero. If this situation happens enough over time, eventually it will reach the TCP_SND_QUEUELEN limit and not function any longer. So it explains why I don't see missed packets on WireShark as it is an lwip variable that is getting set wrong. I assume this would affect the number of pbufs in use after this point.
It would seem to me that the tcp_enqueue function should only add to the global value the number of packets that it uses and not resave the entire value over the global to something that could now be old. Do you agree? Is there something else in the lwipopts.h file that I may not have configured correctly that is suppose to prevent this? I am using a TI Cortex-M3 Stellaris port of the lwip code. If you think it is a porting issue, I can try to explain this TI?
Thanks,
Tim
-----Original Message-----
From: lwip-users-bounces+timl=fleetwood...@nongnu.org [mailto:lwip-users-bounces+timl=fleetwood...@nongnu.org] On Behalf Of Kieran Mansley
Sent: Monday, March 14, 2011 8:33 AM
To: Mailing list for lwIP users
Simon Goldschmidt
Andrea Merello <andrea....@gmail.com> wrote:
> What are in general the rules about concurrency in lwip for the Enet
> ISR, the lwip timers etharp_tmr() and tcp_tmr(), and the user APIs ?
lwIP's concurrency rules are very simple: parallel execution of most parts of the stack is *not* allowed. This includes almost everything except the mem(p)_*(), pbuf_*() and sys_*() functions. The netconn- and socket API functions may also be used from multiple threads, however, these have other limitations in that their objects may not be shared among multiple threads.
Simon
--
GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit
gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
Simon Goldschmidt
Tim Lambrix <Ti...@fleetwoodgroup.com> wrote:
> However, at line 3765, the tcp
> receive interrupt went off during a tcp_write (tcp_enqueue).
That statement makes me nervous as lwIP's concurrency model does not allow the core stack to be called from different threads, which includes almost everything except the mem(p)_*(), pbuf_*() and sys_*() functions (apart from the netconn- and socket API functions).
If your driver (ISR- or receive-thread) directly calls into the stack, that's a very good explanation of the error you see (and I think I remember Kieran had also already suspected something like this).
> I am using a TI Cortex-M3
> Stellaris port of the lwip code. If you think it is a porting issue, I can try
> to explain this TI?
Is this an old version of the port? I think I remember them having a problem of that sort somewhere in an older port, but I'm not sure...
Tim Lambrix
I am using the latest TI library code that they have. I asked if they were working on a port of version 1.4.0 but they said no. Maybe they are waiting for it to be released first.
I do have a couple questions relating to the lwIP processing. First, tcp_write is almost always going to be called from the application code outside of the interrupts. So when a Ethernet hardware receive ISR goes off, how is it normally prevented from interrupting the tcp_write function? Or, should the interrupt service routine be allowed to go off but know that a tcp_write function call is in progress?
What should I be asking TI at this point to solve this problem?
Thanks,
Tim
-----Original Message-----
From: lwip-users-bounces+timl=fleetwood...@nongnu.org [mailto:lwip-users-bounces+timl=fleetwood...@nongnu.org] On Behalf Of Simon Goldschmidt
Sent: Wednesday, March 16, 2011 9:43 AM
To: Mailing list for lwIP users
Subject: RE: [lwip-users] tcp_write() errors on snd_queuelen
Simon Goldschmidt
Tim Lambrix <Ti...@fleetwoodgroup.com> wrote:
> I am using the latest TI library code that they have. I asked if they
> were working on a port of version 1.4.0 but they said no. Maybe they are
> waiting for it to be released first.
That's got nothing to do with 1.4.0, lwIP threading hasn't been changed since a long time, and the bug I think they had was calling ARP in the wrong thread, I think (which was fixed in our example driver in 1.3.0, I think).
> I do have a couple questions relating to the lwIP processing. First,
> tcp_write is almost always going to be called from the application code outside
> of the interrupts.
That's OK.
> So when a Ethernet hardware receive ISR goes off, how
> is it normally prevented from interrupting the tcp_write function? Or,
> should the interrupt service routine be allowed to go off but know that a
> tcp_write function call is in progress?
The lwIP way is to put the packet on a list in the ISR and process that list in the main application ("main loop") somewhere, the same thing goes for timers: the timer functions should be called from the main loop when a time-check says that it's time to call them.
Overall, the strict rule which has to be followed is that it's not allowed to call the core code of lwIP from more than one context (i.e. thread or ISR) at once.
> What should I be asking TI at this point to solve this problem?
I can't answer you that without knowing their code (or the code you are using). Unfortunately, I'm a little short of time for lwIP support at the moment. I'll see if I find the time to look into their lwIP download package to see if there's something wrong there.
Simon
--
NEU: FreePhone - kostenlos mobil telefonieren und surfen!
Jetzt informieren: http://www.gmx.net/de/go/freephone
Simon Goldschmidt
> I am using a TI Cortex-M3
> Stellaris port of the lwip code.
Can you post the link to the download of the TI port? That would help me speed up the task of having a look at their code...
Simon
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
Andrew Foster
Sorry for jumping in on this late in the thread. I've used the TI
Port(1.3.2) a couple of times on different projects. Typically, I've
used it in conjunction with FreeRTOS. One thing to be mindful of is
making sure you have SYS_ARCH_PROTECT defined correctly.
stellarisif_output and stellarisif_transmit must be ran in critical
sections with both the PBUF structure and the ETH TX FIFO protected.
Otherwise, the stellarisif_interrupt can come in and modify the PBUF chain.
Again, I apologize if I'm stating the obvious here or things you've
already ruled out. I wasn't able to find the original post.
Andrew
On 03/16/2011 10:34 AM, Simon Goldschmidt wrote:
> Tim Lambrix<Ti...@fleetwoodgroup.com> wrote:
>> I am using a TI Cortex-M3
>> Stellaris port of the lwip code.
> Can you post the link to the download of the TI port? That would help me speed up the task of having a look at their code...
>
> Simon
_______________________________________________
Tim Lambrix
Thanks for your help too. I will take all I can get right now. It looks like SYS_ARCH_PROTECT is defined as follows and it does hit this line of code (only from the interrupt itself however):
sys_prot_t
sys_arch_protect(void)
{
return((sys_prot_t)MAP_IntMasterDisable());
}
I think this is correct? However, the stellarisif_transmit function you mention is called in three places and only one of them has the SYS_ARCH_PROTECT called before it. I don't see the calls in the interrupt itself stellarisif_interrupt.
Maybe I am missing something here but the problem I seem to have is the Ethernet interrupt goes off while I am in the tcp_write->tcp_enqueue function and changes the values of pcb->snd_queuelen. I have modified the local variable in that function queuelen to not use the value read at line 195 from:
queuelen = pcb->snd_queuelen; to
queuelen = 0;
and line 411 from:
pcb->snd_queuelen = queuelen; to
pcb->snd_queuelen += queuelen;
This fixes the issue but I welcome your insight on why it should not happen in the first place. I don't see any calls close to SYS_ARCH_PROTECT in the tcp_write function to prevent the interrupt from going off. I also tried disabling the Ethernet ISR for both ETH_INT_RX and ETH_INT_TX (the only two enabled) before calling tcp_write and enabling both after the call. That also works to prevent the corruption of pcb-> snd_queuelen.
From what I have traced in code, the protection prevents the interrupt from going off again while in the interrupt. I have been trying to find where lwIP gets processed outside of the interrupts if the interrupt is not suppose to mess with the pbufs. It looks to me like they can.
My original post of findings is here:
http://lists.nongnu.org/archive/html/lwip-users/2011-03/msg00069.html
Tim
-----Original Message-----
From: lwip-users-bounces+timl=fleetwood...@nongnu.org [mailto:lwip-users-bounces+timl=fleetwood...@nongnu.org] On Behalf Of Andrew Foster
Sent: Wednesday, March 16, 2011 11:17 AM
To: Mailing list for lwIP users
Subject: Re: [lwip-users] tcp_write() errors on snd_queuelen
Tim,
Sorry for jumping in on this late in the thread. I've used the TI
Port(1.3.2) a couple of times on different projects. Typically, I've
used it in conjunction with FreeRTOS. One thing to be mindful of is
making sure you have SYS_ARCH_PROTECT defined correctly.
stellarisif_output and stellarisif_transmit must be ran in critical
sections with both the PBUF structure and the ETH TX FIFO protected.
Otherwise, the stellarisif_interrupt can come in and modify the PBUF chain.
Again, I apologize if I'm stating the obvious here or things you've
already ruled out. I wasn't able to find the original post.
Andrew
Mullanix, Todd
Could I get a little back-ground. I know you are using version 1.3.2 from TI. Is this bundled in the StellarisWare package?
If so, which version of StellarisWare?
Which chip are you using?
What OS are you using?
I'm currently working on porting lwIP to TI's SYS/BIOS for the LM3S9B92. I'm basically using the same Ethernet code as StellarisWare (revision 6852). I might be seeing the same type of issue that you are. Based on what you've seen, I'll focus my debugging in that area.
Thanks,
Todd
P.S. I did find one issue that you might want to change in the src/include/lwip/tcp.h file. As noted in the file, there is a compiler bug that is impacting the TCP_SEQ_XXX macros. There is a work-around in this file. Unfortunately, the work-around is not correct for some versions of the compiler. Here is the one I'm using in tcp.h. While heavier, it is correct on different versions of the compiler.
/* Modified by TI to work around a compiler bug */
#if 0
#define TCP_SEQ_LT(a,b) ((s32_t)((a)-(b)) < 0)
#define TCP_SEQ_LEQ(a,b) ((s32_t)((a)-(b)) <= 0)
#define TCP_SEQ_GT(a,b) ((s32_t)((a)-(b)) > 0)
#define TCP_SEQ_GEQ(a,b) ((s32_t)((a)-(b)) >= 0)
#else
/* TI replacement definitions... */
static inline int TCP_SEQ_LT(u32_t a, u32_t b)
{
volatile s32_t zero = 0;
return ((s32_t)(a - b) < zero);
}
static inline int TCP_SEQ_LEQ(u32_t a, u32_t b)
{
volatile s32_t zero = 0;
return ((s32_t)(a - b) <= zero);
}
static inline int TCP_SEQ_GT(u32_t a, u32_t b)
{
volatile s32_t zero = 0;
return ((s32_t)(a - b) > zero);
}
static inline int TCP_SEQ_GEQ(u32_t a, u32_t b)
{
volatile s32_t zero = 0;
return ((s32_t)(a - b) >= zero);
}
#endif
Simon Goldschmidt
As I already said before, the lwIP way is to prevent the driver calling into lwIP from interrupt context.
Tim Lambrix <Ti...@fleetwoodgroup.com> wrote:
> Andrew,
> Thanks for your help too. I will take all I can get right now. It looks
> like SYS_ARCH_PROTECT is defined as follows and it does hit this line of
> code (only from the interrupt itself however):
> sys_prot_t
> sys_arch_protect(void)
> {
> return((sys_prot_t)MAP_IntMasterDisable());
> }
> I think this is correct?
It seems like a correct implementation of sys_arch_protect, yes.
> However, the stellarisif_transmit function you
> mention is called in three places and only one of them has the
> SYS_ARCH_PROTECT called before it. I don't see the calls in the interrupt itself
> stellarisif_interrupt.
That's because you don't need to disable interrupts in an ISR, I guess (unless there can be a nested interrupt with a higher priority from which you need to protect yourself).
> Maybe I am missing something here but the problem I seem to have is the
> Ethernet interrupt goes off while I am in the tcp_write->tcp_enqueue function
> and changes the values of pcb->snd_queuelen.
Not really. The problem is only that the Ethernet interrupt seems to call into lwIP where it shouldn't.
> I have modified the local
> variable in that function queuelen to not use the value read at line 195 from:
>
> queuelen = pcb->snd_queuelen; to
> queuelen = 0;
>
> and line 411 from:
>
> pcb->snd_queuelen = queuelen; to
> pcb->snd_queuelen += queuelen;
>
> This fixes the issue but I welcome your insight on why it should not
> happen in the first place.
That's only one of many things which won't work: lwIP is simply not designed to work in that scenario!
> I don't see any calls close to SYS_ARCH_PROTECT in
> the tcp_write function to prevent the interrupt from going off.
Again, lwIP's core is not designed for multithreading, that's why there are not SYS_ARCH_PROTECT calls. It would be pretty unperformant to disable interrupts for the whole call to tcp_write!
I really could help you more if I had insight on the code you are using.
Simon
--
GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit
gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
_______________________________________________
Andrew Foster
For the sake of our discussion I'm going to assume
MAP_IntMasterDisable() does what it as advertised to do.
Moving forward, how does your project use lwIP?
More specifically:
Are you running an RTOS? If so which?
How does your application call tcp_write? I.e. from a thread or other
context?
Can you include your lwipopts.h in this post?
Andrew
Simon Goldschmidt
I think there's a bug in the stellarisif.c, function stellarisif_interrupt(). The following code seems not correct:
>>>>> snip >>>>>
#if NO_SYS
if(ethernet_input(p, netif)!=ERR_OK) {
#else
if(tcpip_input(p, netif)!=ERR_OK) {
#endif
<<<<< snap <<<<<
a) In the NO_SYS==1 case, the driver *must not* call into the lwIP code from interrupt context (unless you can make sure that lwIP is not used from any other ISR context or from the main loop). This will most certainly result in threading problems as seen by Tim. Instead, the packet must be put on a queue and fed into lwIP from the main context where lwIP is used.
b) In both cases, netif->input should be used. (Although I admit that's rather a cosmetic issue than a bug.)
Simon
--
NEU: FreePhone - kostenlos mobil telefonieren und surfen!
Jetzt informieren: http://www.gmx.net/de/go/freephone
_______________________________________________
Tim Lambrix
> Moving forward, how does your project use lwIP?
I am in simplest form, taking in SPI data and sending it out Ethernet. I can get an SPI packet of 50 bytes approximately every 3ms though I am throughput testing at 8ms currently.
> More specifically:
> Are you running an RTOS? If so which?
No RTOS.
> How does your application call tcp_write? I.e. from a thread or other
> context?
It is not called from an interrupt. I check while looping through main if any packets have been added to a buffer and then send out that packet using tcp_write().
> Can you include your lwipopts.h in this post?
Should be attached. I could use some help verifying I have it configured correctly for my application with respect to the memory requirements at some point too.
Tim Lambrix
It appears that the rest of the lwIP processing is all done in the Ethernet interrupt. The interrupt function looks like the following:
lwIPEthernetIntHandler(void)
{
// Read and Clear the interrupt.
ulStatus = EthernetIntStatus(ETH_BASE, false);
EthernetIntClear(ETH_BASE, ulStatus);
// The handling of the interrupt is different based on the use of a RTOS.
// No RTOS is being used. If a transmit/receive interrupt was active,
// run the low-level interrupt handler.
if(ulStatus)
{
stellarisif_interrupt(&g_sNetIF);
}
// Service the lwIP timers.
lwIPServiceTimers();
}
It looks like all the work of the lwip processing, transmitting, and receiving occurs in the function stellarisif_interrupt() which I included below:
stellarisif_interrupt(struct netif *netif)
{
struct stellarisif *stellarisif;
struct pbuf *p;
/* setup pointer to the if state data */
stellarisif = netif->state;
/**
* Process the transmit and receive queues as long as there is receive
* data available
*
*/
p = stellarisif_receive(netif);
while(p != NULL) {
/* process the packet */
if(ethernet_input(p, netif)!=ERR_OK) {
/* drop the packet */
LWIP_DEBUGF(NETIF_DEBUG, ("stellarisif_input: input error\n"));
pbuf_free(p);
/* Adjust the link statistics */
LINK_STATS_INC(link.memerr);
LINK_STATS_INC(link.drop);
}
/* Check if TX fifo is empty and packet available */
if((HWREG(ETH_BASE + MAC_O_TR) & MAC_TR_NEWTX) == 0) {
p = dequeue_packet(&stellarisif->txq);
if(p != NULL) {
stellarisif_transmit(netif, p);
}
}
/* Read another packet from the RX fifo */
p = stellarisif_receive(netif);
}
/* One more check of the transmit queue/fifo */
if((HWREG(ETH_BASE + MAC_O_TR) & MAC_TR_NEWTX) == 0) {
p = dequeue_packet(&stellarisif->txq);
if(p != NULL) {
stellarisif_transmit(netif, p);
}
}
-----Original Message-----
From: lwip-users-bounces+timl=fleetwood...@nongnu.org [mailto:lwip-users-bounces+timl=fleetwood...@nongnu.org] On Behalf Of Simon Goldschmidt
Sent: Wednesday, March 16, 2011 12:16 PM
To: Mailing list for lwIP users
If you try to protect tcp_write() against the ETH interrupt by disabling interrupts, that would mean you would have to *always* disable the ETH interrupt while calling into lwIP. That's pretty unperformant, I think.
As I already said before, the lwIP way is to prevent the driver calling into lwIP from interrupt context.
Simon Goldschmidt
Tim Lambrix <Ti...@fleetwoodgroup.com> wrote:
> It looks like all the work of the lwip processing, transmitting, and
> receiving occurs in the function stellarisif_interrupt() which I included below:
Except for your call to tcp_write(). You need to sort this out with TI: either
a) you must call tcp_write() from the same interrupt context or
b) the ethernet driver interrupt must not call into lwIP or
c) you need to disable the ethernet interrupt while calling any lwIP function from your main application.
Simon
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
_______________________________________________
Simon Goldschmidt
Tim Lambrix <Ti...@fleetwoodgroup.com> wrote:
> > Can you include your lwipopts.h in this post?
>
> Should be attached. I could use some help verifying I have it configured
> correctly for my application with respect to the memory requirements at
> some point too.
I can comment on generic lwIP options (without knowing your platform in detail):
>>>>> snip >>>>>
#define TCP_WND 2048 // default is 2048
#define TCP_MSS 1050 // default is 128
#define TCP_SND_BUF (8 * TCP_MSS) // default is 256, was 6 *
#define TCP_SND_QUEUELEN 128//(4 * (TCP_SND_BUF/TCP_MSS))
<<<<< snap <<<<<
In general, you might run into receive problems with the TCP window being so small compared to the MSS (TCP_WND is only used as receive window, the transmit window is taken from the remote host's SYN packet). It should be at least 2*MSS but is encouraged to be at least 4*MSS if your memory constraints allow you to. This is to make sure the TCP sliding window algorithm can work correctly.
TCP_SND_BUF and TCP_SND_QUEUELEN seem OK. Those settings allow you to make 64 or 128 calls to tcp_write() (one pbuf per call when copying data, 2 per call when not copying) and to enqueue 8400 bytes of data in these pbufs.
The other memory settings seem OK, too. However, that depends on the number of parallel connections.
Simon
--
GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit
gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
_______________________________________________
Tim Lambrix
> Could I get a little back-ground. I know you are using version 1.3.2 from TI. > Is this bundled in the StellarisWare package?
I started my project from the lwIP_enet example project in the LM3S9B90. It is a simple web server which I ripped out and got a little help from a TI FAE to get the raw tcp functioning.
> If so, which version of StellarisWare?
I am using the latest version too, 6852.
> Which chip are you using?
LM3S9790
> What OS are you using?
No OS.
Tim Lambrix
> I can comment on generic lwIP options (without knowing your platform in detail):
> compared to the MSS (TCP_WND is only used as receive window, the transmit window is
> taken from the remote host's SYN packet). It should be at least 2*MSS but is
> encouraged to be at least 4*MSS if your memory constraints allow you to. This is to
> make sure the TCP sliding window algorithm can work correctly.
> TCP_SND_BUF and TCP_SND_QUEUELEN seem OK. Those settings allow you to make 64 or 128
> calls to tcp_write() (one pbuf per call when copying data, 2 per call when not
> copying) and to enqueue 8400 bytes of data in these pbufs.
> The other memory settings seem OK, too. However, that depends on the number of
> parallel connections.
Lastly, how big is a pbuf and is there a relationship between the MEMP_NUM_TCP_SEG and a pbuf? If there is a way to set a pbuf, should it be set to a typical call into tcp_write for optimum memory usage?
Mullanix, Todd
My understanding of the NO_SYS=1 case was the same as Tim's. So as long as nothing is done in main() or another interrupt, it should be fine since there are no threads.
Todd
Kieran Mansley
> What is actually getting put into that TCP window memory? Is that the
> ack for each tcp packet? Is that memory coming out of the MEM_SIZE
> define?
It doesn't refer to real memory as such; the receive window is how much
space the receiver is advertising to the sender. The sender is allowed
to send this much data before it has to wait for an acknowledgement that
provides a further tranche of receive window. It refers to memory
indirectly in that the receiver has to commit to receiving the data it
has advertised space for, so it needs to have sufficient buffers
available. (Strictly speaking it is allowed to drop received packets
even if they are in the advertised receive window, but that should be
avoided)
>
> > TCP_SND_BUF and TCP_SND_QUEUELEN seem OK. Those settings allow you
> to make 64 or 128
> > calls to tcp_write() (one pbuf per call when copying data, 2 per
> call when not
> > copying) and to enqueue 8400 bytes of data in these pbufs.
>
> I am copying the data into the tcp_write. So I can then hold 128 * my
> 50 byte packets in the TCP_SND_BUF of ~8K? However, when I have
> viewed the buffer size debug output, I see pcb->snd_queuelen get up to
> 100 or more but pcb->snd_buf only has a couple hundred bytes removed.
> Where is the rest of the queues (pbufs, I assume) stored in memory?
There are two lengths for the send queue: the byte length and the
segment length. It is considered full when either limit is reached.
Metadata structures like pbufs can be stored in a variety of places
depending on how you've configured lwIP. It's possible to take them
from a fixed pool for example, or it can just use malloc.
> > The other memory settings seem OK, too. However, that depends on the
> number of
> > parallel connections.
>
> I only support 1 connection at a time. If I support 2, does it double
> the memory requirements and does it happen automatically or is the
> TCP_SND_QUEUELEN shared between the connections?
TCP_SND_QUEUELEN is the length of each socket's send queue, not shared
between connections, so the memory requirement is proportional to the
number of sockets.
> Lastly, how big is a pbuf and is there a relationship between the
> MEMP_NUM_TCP_SEG and a pbuf?
I can't remember exactly what its size is - take a look at the
definition. It's deliberately pretty small though, I think about 32
bytes.
There is a many-to-one mapping between pbufs and segments (i.e. each
segment can be split across multiple pbufs by chaining them together).
> If there is a way to set a pbuf, should it be set to a typical call
> into tcp_write for optimum memory usage?
I don't understand that bit.
Kieran
Simon Goldschmidt
> There are two lengths for the send queue: the byte length and the
> segment length.
Actually, I think it's the number of pbufs queued, not the number of segments. A segment can consist of multiple pbufs depending on how tcp_write() is called to enqueue the data to send.
Simon
--
GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit
gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
_______________________________________________
Kieran Mansley
>
> Kieran Mansley <kie...@recoil.org> wrote:
> > There are two lengths for the send queue: the byte length and the
> > segment length.
>
> Actually, I think it's the number of pbufs queued, not the number of
> segments. A segment can consist of multiple pbufs depending on how
> tcp_write() is called to enqueue the data to send.
I think you're right: I sat and tried to remember which it was when
writing that, but must have settled on the wrong one. I should have
checked the source.
Thanks for the correction,
Kieran
Tim Lambrix
A few more questions: What are the performance tradeoffs of a larger TCP_MSS versus a smaller one when calling tcp_write with a typical 40 byte write? Obviously, I want to keep the memory requirements as small as possible but be able to handle the load in (practically) any network.
I see the default for MEM_SIZE is only 1600. I have mine set to 22K but is this really necessary? How should one go about picking reasonable values for memory and TCP requirements that are sufficient? Is there a formula or calculation for any of these based on the frequency that tcp_write is called?
While running yesterday on with the TCP_QLEN_DEBUG enabled, I saw two times that over a second went by that no tcp_recieve was called and therefore the snd_queuelen value reached the TCP_SND_QUEUELEN limit I have set in the options file. Is this typical and what approach would you recommend such that this doesn't happen - keep growing the TCP_SND_QUEUELEN or is there something else not configured correctly (perhaps the TCP_WND not being at least 2* or 4* the TCP_MSS)?
Kieran Mansley
> A few more questions: What are the performance tradeoffs of a larger
> TCP_MSS versus a smaller one when calling tcp_write with a typical 40
> byte write? Obviously, I want to keep the memory requirements as
> small as possible but be able to handle the load in (practically) any
> network.
A large MSS allows you to send and receive larger segments. Typically
there is a per-byte overhead and a per-packet overhead, so
sending/receiving fewer segments will mean less overhead in total.
However if all the packets you send and receive are less than the MSS
anyway then you won't see any difference. Note that the size you call
tcp_write() with isn't the size of the packets - they may be batched
together by the stack to form larger packets (up to the MSS).
On the other hand a larger MSS can mean that you need to commit more
memory, and that potentially some of that will be wasted. E.g. if you
have configured a large MSS but only receive small packets and each
packet goes into an MSS-sized buffer then most of the buffer will be
unused.
To counter that lwIP allows (if your driver also supports it) the
splitting of packets across buffers. This means you can have small
buffers, and append together as many as you need to hold the packet.
Therefore there isn't the waste in unused memory, but this in turn has
the downside that there is now some extra overhead in dealing with all
these chained buffers (it is more complex than just a single buffer per
packet).
In summary, there is no right way; you need to understand the trade-offs
and choose what is best for your application and network. I personally
would start with a standard sized MSS and have lots of small pbufs
chained together.
> I see the default for MEM_SIZE is only 1600. I have mine set to 22K
> but is this really necessary? How should one go about picking
> reasonable values for memory and TCP requirements that are sufficient?
> Is there a formula or calculation for any of these based on the
> frequency that tcp_write is called?
The default is very conservative. Probably too small. But lwIP is
supposed to work in such small amount of memory. It won't get very good
performance though. With a more sensible amount of memory (such as you
have allocated) it will start to be able to stream data more
efficiently. The best approach for getting the allocation of all the
different pools and structures is unfortunately quite iterative: choose
a starting point, run with your expected workload, use LWIP_STATS to see
what it is running out of (or not using much of), adjust appropriately
and repeat.
> While running yesterday on with the TCP_QLEN_DEBUG enabled, I saw two
> times that over a second went by that no tcp_recieve was called and
> therefore the snd_queuelen value reached the TCP_SND_QUEUELEN limit I
> have set in the options file. Is this typical
It depends on your workload. If you didn't receive anything from the
network for a second then I wouldn't expect the stack to pass anything
to the application for a second. But I'm guessing that you do have
packets delivered from the network more often than that. In that case
there are few reasons why you might see gaps: (i) loss requires
retransmissions (and associated round-trip-times to detect and fix);
(ii) lack of free buffers may mean the stack isn't able to handle
received packets, so it will drop them, and there will be loss - see
(i); (iii) some problem with your port could mean that lwIP just didn't
get called in that time, and so couldn't do any work. I would guess (i)
or (ii) - a packet capture would show the retransmissions, and
LWIP_STATS will highlight if you're running out of buffers (possible if
they're all on the send queue).
> and what approach would you recommend such that this doesn't happen -
> keep growing the TCP_SND_QUEUELEN or is there something else not
> configured correctly (perhaps the TCP_WND not being at least 2* or 4*
> the TCP_MSS)?
I think growing the send queue length is unlikely to be helpful - it
will just increase the amount of memory in use. What you need to do is
work out why the stack is unable to currently send data and solve that.
I.e. something is limiting your outgoing bandwidth (or you have a bug
that is resulting in an ever-increasing-send-queue). If we can increase
your outgoing bandwidth then the send queue won't grow. If we can't
increase your outgoing bandwidth then increasing the send queue length
will just delay the time it takes to overflow. The send queue is only
useful for smoothing bursts in application writes and network sends;
once it is big enough to do that make it larger will be harmful. Also,
and more fundamentally, if you don't want to drop data your application
needs to cope with the send queue becoming full either by blocking or by
having its own buffering scheme.
Kieran
gold...@gmx.de
> For my port to SYS/BIOS, I have NO_SYS=0 and I'm making sure the netif->input is being called from a thread and not in the interrupt. So I think I'm adhering to the rules of the game in my case. Note: I'll make the change to use netif->input.
automatically by tcpip_thread and tcpip_input() is used for input packets.
> My understanding of the NO_SYS=1 case was the same as Tim's. So as long as nothing is done in main() or another interrupt, it should be fine since there are no threads.
calls tcp_write from main(). I know this is a point in lwIP which is not
as easy to understand and at least Tim (as a user of Stellarisware) got
it wrong resulting in some days of bug-hunting :-(
Also, having had a look at the code, I'm not sure at which point it
would be safe to call tcp_write with NO_SYS=1: there seems to be no
obvious way to call user code from the ethernet interrupt handler (which
is obviously the "lwip execution context" in your port for NO_SYS=1).
Note that this is only a problem for asynchronous writes. Server-writes
(like in the httpd) are either triggered by received packets or by lwIP
timers, so their execution context is always correct.
Simon
Tim Lambrix
I am trying to learn how the tcp_poll timer function works. I have a function and set:
tcp_poll(pcb, TCP_Poll, 1);
I found that increasing this value above a 1 can either A) never get called at a half second interval and B) from wireshark, only sends data out at whatever the value is. For example, I put a 4 in there and WireShark displays a 2 second gap between send and receives. However, with a 1 in there it sends and receives data at a 200ms interval and if needed, due to more than one TCP outgoing packet being full (2*TCP_MSS), it will send all packets at that time. I don't understand this timer value behavior as the TCP_Poll function I have written, doesn't really do much (residual from StellarisWare).
if ((hs == NULL) && (pcb->state == ESTABLISHED))
{
tcp_abort(pcb);
return ERR_ABRT;
}
Is there an explanation of how the poll interval timer works this way? I found that the TCP_SND_QUEUELEN limit is reached very quickly if the value is NOT a 1 which should map to 500 ms based on the slow TCP timer. This makes some sense as I can send up to sixty tcp_writes in 200 ms alone. In two seconds, this is over 200. In other words, my code only functions without crashing (with the disable Ethernet interrupt fix around tcp_write of course) with the poll interval set to a 1.
Sorry for the piecemeal email. Also, I don't want to be a bother so if I am asking too many questions, just let me know. If you know of another source I should tap into (in the US?), I would be interested.
Andrea Merello
Just to make sure I've got it; this means also that user APIs like
udp_sendto must be called with ints disabled (or priority rised) when
used in the "main loop" to avoid races with eth ISR and lwip timer
ISR?
Thanks
Andrea
Kieran Mansley
> Just to make sure I've got it; this means also that user APIs like
> udp_sendto must be called with ints disabled (or priority rised) when
> used in the "main loop" to avoid races with eth ISR and lwip timer
> ISR?
That's one way, although personally I would instead make sure that
interrupts weren't calling into the stack directly, and were instead
queueing work (received packets, timers firing, etc) for the main loop
to process. Your main loop is then free to do what it likes and leave
interrupts enabled.
Kieran