[lwip-users] low_level_output question

[John Kennedy]

Hi,
I'm porting lwip (using an RTOS and the socket API) and I have a question regarding the low_level_output function.
If I can't send the pbuf immediately (netif busy), to avoid copying the pbuf, I'd like to do the following:
* Increment the reference count to prevent lwip from re-using or freeing the pbuf.
* Put the pbuf on my output queue and try to send it later.
* Return ERR_OK.
Later when the netif is not busy...
* Remove the pbuf from my output queue.
* Send the pbuf.
* Free the pbuf.

Will this scheme work with lwip?

Thanks,
John Kennedy

________________________________________
John Kennedy

Idaho Technology Inc.
390 Wakara Way
Salt Lake City, UT 84108, USA

USA: 1-800-735-6544
Bus:+1 (801)736-6354 x448
Fax:+1 (801)588-0507

http://www.idahotech.com/

________________________________________
CONFIDENTIALITY NOTICE: This E-mail and any attachments are confidential information of the sender and are for the exclusive use of the intended recipient. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of this E-mail or any attachment is prohibited. If you have received this E-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

_______________________________________________
lwip-users mailing list
lwip-...@nongnu.org
http://lists.nongnu.org/mailman/listinfo/lwip-users

[Bill Auerbach]

Yes, it will work this way. In fact, the Stellaris lwIP Ethernet driver is done just like this.

Bill

[Fabian Koch]

Hello everyone,

> Hi,
> I'm porting lwip (using an RTOS and the socket API) and I have a
> question regarding the low_level_output function.
> If I can't send the pbuf immediately (netif busy), to avoid copying
> the pbuf, I'd like to do the following:
> *       Increment the reference count to prevent lwip from re-using
> or freeing the pbuf.
> *       Put the pbuf on my output queue and try to send it later.
> *       Return ERR_OK.
> Later when the netif is not busy...
> *       Remove the pbuf from my output queue.
> *       Send the pbuf.
> *       Free the pbuf.
>
> Will this scheme work with lwip?

This is a very valid discussion point for the Stack, IMHO.
It is not really explicitly documented that the Stack will free the pbufs for a packet after it handed it over to the driver layer.
If the Driver is sending asynchronously and the data is not memcpy'd by the driver, the datablock might be allocated by someone else and overwritten with other data. This might be an explaination for mysteriously lost or changed packets.
Also given the fact that LwIP is a zero-copy stack, people might not expect to memcpy stuff on the driver layer?

I propose we do either of the following:

1) In the ethernetif-skeleton explicitly document the behavior and tell the developer to memcpy the data if their transmission is async.
2) See 1), but tell the developer to increment the reference count and pbuf_free() themselves (probably faster than memcpy)
3) Remove the freeing of the pbuf after handing it over to the driver layer and document that and leave it to the driver developer to pbuf_free() after the xmit is really done

comments?

regards,
Fabian

[Simon Goldschmidt]

There's a task on savannah on this (#7896 ): https://savannah.nongnu.org/task/?7896

Basically, the scheme will work on _POOL and _RAM pbufs, but not on _ROM/_REF pbufs, which is why usage of these types would have to be adjusted (including a copy-on-delayed-usage flag or something like that).

However, as sad as it is, this won't be included in 1.3.1 as we first have to agree on one solution.

Simon

--
Neu: GMX FreeDSL Komplettanschluss mit DSL 6.000 Flatrate + Telefonanschluss für nur 17,95 Euro/mtl.!* http://dslspecial.gmx.de/freedsl-surfflat/?ac=OM.AD.PD003K11308T4569a

[Kieran Mansley]

On Tue, 2009-04-07 at 09:39 +0200, Fabian Koch wrote:
> I propose we do either of the following:
>
> 1) In the ethernetif-skeleton explicitly document the behavior and
> tell the developer to memcpy the data if their transmission is async.
> 2) See 1), but tell the developer to increment the reference count and
> pbuf_free() themselves (probably faster than memcpy)

Those are fine: more documentation is always good.

> 3) Remove the freeing of the pbuf after handing it over to the driver
> layer and document that and leave it to the driver developer to
> pbuf_free() after the xmit is really done

I don't like that one. It's essentially changing the driver/stack
interface, and there may be reasons why the stack wants to keep the pbuf
around without it being freed.

Kieran

[Bill Auerbach]

Incrementing the reference count is the way to move the freeing of the pbuf
from the stack to the Ethernet driver. It's been done by several
implementers successfully. A comment in low_level_output to increment the
ref count if the packet isn't being physically sent before the function
returns and to free it at a later time when it has is sufficient. What
hasn't been clear to me is the safety of handling pbufs in interrupt service
routines. Can a TX dma complete ISR call pbuf_free?

Bill

>-----Original Message-----
>From: lwip-users-bounces+bauerbach=arrayon...@nongnu.org
>[mailto:lwip-users-bounces+bauerbach=arrayon...@nongnu.org] On

[John Kennedy]

Since pbuf_free decrements the pbuf reference count and only frees the pbuf when the reference count goes to zero I assumed that by incrementing the reference count in the Ethernet driver it would prevent the stack from freeing or reusing the pbuf, and that the Ethernet driver should then call pbuf_free when done with the pbuf. My original question was to verify this assumption.

Also pbuf_free guards decrementing the pbuf reference count by surrounding it with SYS_ARCH_PROTECT and SYS_ARCH_UNPROTECT. If the pbuf is type PBUF_POOL, PBUF_ROM, or PBUF_REF then memp_free gets called, otherwise mem_free gets called. The function memp_free is also guarded with SYS_ARCH_PROTECT and SYS_ARCH_UNPROTECT. In my implementation these macros simply disable and re-enable interrupts so it's not a good idea to call pbuf_free from an ISR. In general I think these macros are intended to be invoked outside an interrupt context, but I suppose this depends on the port. In my implementation for pbuf type PBUF_RAM, pbufs are ultimately allocated off the heap with malloc and freed with free, so for my implementation pbufs of type PBUF_RAM shouldn't be allocated or freed from an ISR either.

John

________________________________________
John Kennedy

Idaho Technology Inc.
390 Wakara Way
Salt Lake City, UT 84108, USA

USA: 1-800-735-6544
Bus:+1 (801)736-6354 x448
Fax:+1 (801)588-0507

http://www.idahotech.com/

Bill

________________________________________

CONFIDENTIALITY NOTICE: This E-mail and any attachments are confidential information of the sender and are for the exclusive use of the intended recipient. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of this E-mail or any attachment is prohibited. If you have received this E-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

[gold...@gmx.de]

Bill Auerbach wrote:
> Incrementing the reference count is the way to move the freeing of the pbuf
> from the stack to the Ethernet driver. It's been done by several
> implementers successfully.

I can tell you that won't work if
- sending data over UDP using the sockets API and
- having a fast device and
- modifying the buffer passed to lwip_send(to) after it returns

The reason is that (like I said in my previous post), incrementing the
ref-count will do for _RAM and _POOL, but not for _REF and _ROM pbufs;
udp sockets use _REF. Here, incrementing the ref-count prevents the pbuf
from being freed, but it does not prevent the actual from getting
modified before sending. The only two solutions are a) not returning
from lwip_send(to) until the data is sent by the hardware and b) copying
the data before sending (using PBUF_RAM, not PBUF_REF). Both solutions
have their problems, thus task #7896 to discuss it.

> A comment in low_level_output to increment the
> ref count if the packet isn't being physically sent before the function
> returns and to free it at a later time when it has is sufficient.

I'm afraid it's not. See above.

> What hasn't been clear to me is the safety of handling pbufs in interrupt service
> routines. Can a TX dma complete ISR call pbuf_free?
>

Yes, it can, as long as LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT is set to
1. Note this has been fixed after 1.3.0, so unless you are using a CVS
version from after that, you will have to wait for 1.3.1 to use it.

Simon

[gold...@gmx.de]

John Kennedy wrote:
> Since pbuf_free decrements the pbuf reference count and only frees the pbuf when the reference count goes to zero I assumed that by incrementing the reference count in the Ethernet driver it would prevent the stack from freeing or reusing the pbuf, and that the Ethernet driver should then call pbuf_free when done with the pbuf. My original question was to verify this assumption.
>

Your assumtption is indeed correct, but it doesn't account for
application buffers used with PBUF_REF/_ROM.

Simon

[John Kennedy]

Ah, that's unfortunate since using the reference count as I described is so intuitive and eliminates the need to copy the pbuf in the driver (if it can't be sent immediately). It makes one wonder why UDP sockets use the reference count differently, seems like a mistake?

John

________________________________________
John Kennedy

Idaho Technology Inc.
390 Wakara Way
Salt Lake City, UT 84108, USA

USA: 1-800-735-6544
Bus:+1 (801)736-6354 x448
Fax:+1 (801)588-0507

http://www.idahotech.com/
-----Original Message-----
From: gold...@gmx.de [mailto:gold...@gmx.de]
Sent: Tuesday, April 07, 2009 9:26 AM
To: Mailing list for lwIP users
Subject: Re: [lwip-users] low_level_output question

Simon

________________________________________

CONFIDENTIALITY NOTICE: This E-mail and any attachments are confidential information of the sender and are for the exclusive use of the intended recipient. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of this E-mail or any attachment is prohibited. If you have received this E-mail in error, please notify us immediately by returning it to the sender and delete this copy from your system. Thank you for your cooperation.

[Jan Wester]

Hello
If you in SYS_ARCH_PROTECT save the interrupt flags before you disable the interrupt and in SYS_ARCH_UNPROTECT restore the interrupt flags,
You can use the functions in ISR

Med vänliga hälsningar/Best Regards
Jan Wester

WHI Konsult AB
Scheelegatan 11, SE-112 28 Stockholm
www.whi.se
in...@whi.se
+46 8 449 05 30

-----Ursprungligt meddelande-----
Från: lwip-users-bounces+j.wester=whi...@nongnu.org [mailto:lwip-users-bounces+j.wester=whi...@nongnu.org] För John Kennedy
Skickat: den 7 april 2009 17:00
Till: 'Mailing list for lwIP users'
Ämne: RE: [lwip-users] low_level_output question

[Simon Goldschmidt]

Same old story...

> Hello
> If you in SYS_ARCH_PROTECT save the interrupt flags before you disable the
> interrupt and in SYS_ARCH_UNPROTECT restore the interrupt flags,
> You can use the functions in ISR

I guess you mean 'pbuf_free' by 'the functions'? If so, I have to dissapoint you: if you use the lwIP heap (mem_malloc) for PBUF_RAM pbufs (which is the standard), you can only use pbuf_free from ISR context if you use at least a CVS version post-1.3.0 and manually set LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT to 1 in your lwipopts.h.

Otherwise, the heap will be protected by a *semaphore*, not by SYS_ARCH_PROTECT, which will prevent you from using it from ISR!

Simon
--
Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger01