[lwip-users] restarting a connection with the raw api

[Richie Bonventre]

I was wondering if anyone knew of any examples using the tcp raw api
that involved closing and restarting the connection (specifically as
the client using tcp_connect), as well as properly handling a
tcp_connect() timeout so that it keeps trying until it successfully
gets a connection. When in that process do I have to call tcp_new(),
tcp_close(), when can I safely call tcp_connect() again?
Thanks,
RB

_______________________________________________
lwip-users mailing list
lwip-...@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

[Kieran Mansley]

On Thu, 2011-09-01 at 16:09 -0400, Richie Bonventre wrote:
> I was wondering if anyone knew of any examples using the tcp raw api
> that involved closing and restarting the connection (specifically as
> the client using tcp_connect), as well as properly handling a
> tcp_connect() timeout so that it keeps trying until it successfully
> gets a connection. When in that process do I have to call tcp_new(),
> tcp_close(), when can I safely call tcp_connect() again?

1) First call tcp_new()

2) Then set up the callbacks needed (e.g. by calling tcp_err() to
specific the error callback, and so on)

3) Then call tcp_connect()
- this will either succeed and call the connected callback passed to
tcp_connect, or fail and call the error callback specified by tcp_err()
- if the error callback is called because of a timeout I think you
could immediately call tcp_connect() again, but I haven't checked this.

4) When you've finished with a connection, call tcp_close(). After this
returns success you can't touch the pcb struct again, and must call
tcp_new() to get another (i.e. back to step 1).

Hope that helps. This is all taken from lwip/doc/rawapi.txt, and there
is more detail there.

Kieran

[gold...@gmx.de]

Kieran Mansley wrote:
> 3) Then call tcp_connect()
> - this will either succeed and call the connected callback passed to
> tcp_connect, or fail and call the error callback specified by tcp_err()
> - if the error callback is called because of a timeout I think you
> could immediately call tcp_connect() again, but I haven't checked this.

And if the default connect timeout is too long for you, you can
implement a timeout yourself by registering a poll function using
tcp_poll(). This function will get called periodically and if after the
x'th call, neither the connect nor the error callback has been called,
you can abort the connect request by calling tcp_close().

> Hope that helps. This is all taken from lwip/doc/rawapi.txt, and there
> is more detail there.

Hehe...

Simon

[Richie Bonventre]

Hey thanks for helping out.
It doesn't seem quite so straightforward as that. For example suppose
you connect and then unplug the cable. If you try and tcp_close() and
the other computer is no longer connected, the connection doesnt
actually end because the close request is never acked and everything
will get messed up if you just try and tcp_connect again. I've had
problems where if you call tcp_new() or tcp_connect() at the wrong
time, it will overwrite the current connection and repush it on to
tcp_active_pcbs which causes pcb->next to point to itself. This causes
lots of the lwip timers and functions to loop forever and freeze the
code. Other times, even if the other computer is still connected and
the closing is completely acked and everything, when you try and
tcp_connect again on the same connection, it no longer seems to get
any response from the other computer. It seems like even though the
connection makes it all the way to the CLOSED state, the other
computer isn't completely aware that the connection is done and wont
accept another request from the same connection.

I'm basically trying to account for 4 situations:
1 - A connection is established from lwip to some code on another
machine, and then I ctrl-C the code listening on the other machine. I
want lwip to recognize this and try to reconnect, to be ready whenever
I restart the listener on B.
2 - I attempt to connect but the cable is not plugged in between the machines.
3 - I attempt to connect and the cable is plugged in, but nothing is
listening to the socket yet to accept the connection on the other
machine.
4 - I have connected successfully, and I then unplug the cable. My
code that uses lwip periodically sends a ping packet, and if it
doesn't get a response in time, it should close the connection and try
to reconnect.

So for which of these do I have to call tcp_close(), which can I
recall tcp_new() or should I reuse the old tcp_pcb, and what do I need
to monitor before I call either of these or call tcp_connect() again?
Do I need to be checking the state of the pcb or something else?
thanks,
RB

[Kieran Mansley]

On Sun, 2011-09-04 at 00:08 -0400, Richie Bonventre wrote:
> Hey thanks for helping out.
> It doesn't seem quite so straightforward as that. For example suppose
> you connect and then unplug the cable. If you try and tcp_close() and
> the other computer is no longer connected, the connection doesnt
> actually end because the close request is never acked

It will (eventually) time out.

> and everything
> will get messed up if you just try and tcp_connect again.

It shouldn't get messed up. If it does, that's a bug. Can you give
more details?

> I've had
> problems where if you call tcp_new() or tcp_connect() at the wrong
> time, it will overwrite the current connection and repush it on to
> tcp_active_pcbs which causes pcb->next to point to itself. This causes
> lots of the lwip timers and functions to loop forever and freeze the
> code.

Can you describe your threading model? It sounds like you might have
two threads racing here and modifying state in the core of lwIP, which
isn't supported and would cause exactly the sort of problem you
describe.

> Other times, even if the other computer is still connected and
> the closing is completely acked and everything, when you try and
> tcp_connect again on the same connection, it no longer seems to get
> any response from the other computer. It seems like even though the
> connection makes it all the way to the CLOSED state, the other
> computer isn't completely aware that the connection is done and wont
> accept another request from the same connection.

If you can get a packet capture of this I'll take a look. by
"tcp_connect again on the same connection" are you trying to use the
same pcb after calling tcp_close()? This isn't allowed: once
tcp_close() has returned success, you must not touch the pcb structure
as it could have been freed.

> I'm basically trying to account for 4 situations:
> 1 - A connection is established from lwip to some code on another
> machine, and then I ctrl-C the code listening on the other machine. I
> want lwip to recognize this and try to reconnect, to be ready whenever
> I restart the listener on B.

This rather depends on what happens with you ctrl-C the code on the
other machine:
1) if it closes the connection (with a FIN) then lwIP will notify the
application by calling the recv callback with zero length.
2) if it aborts the connection (with a RST) then lwIP will notify the
application by either the recv callback or the err callback (I can't
remember which off the top of my head).
3) if it doesn't send us anything, or the packet it sends it lost, then
lwIP will only notice next time it tries to send to the other side,
which will then likely cause it to send us a RST. This could take some
time. There is a feature in TCP called keep-alive which probes the
connection at regular intervals to see if it is still up to help deal
with the case where an idle connection is no longer being handled by the
other end.

> 2 - I attempt to connect but the cable is not plugged in between the machines.

This should time out, but it may take some time. TCP is very tolerant
and keeps re-trying for a while.

> 3 - I attempt to connect and the cable is plugged in, but nothing is
> listening to the socket yet to accept the connection on the other
> machine.

This should fail quickly, as the other end will reply to our connection
attempt. Your application can then wait a while and attempt to connect
again. I would avoid putting this in a tight loop though, as it would
be rather antisocial to flood the network with repeated connection
attempts.

> 4 - I have connected successfully, and I then unplug the cable. My
> code that uses lwip periodically sends a ping packet, and if it
> doesn't get a response in time, it should close the connection and try
> to reconnect.

TCP is very tolerant of broken links and will keep trying to send your
pings for a long time before it gives up and closes the connection.
There is nothing to stop you doing as you describe and having a much
more strict time-out in your application if it doesn't receive a reply.
Just call tcp_close() and try and open a new one (although as before, be
careful you don't flood the network with connection attempts).

> So for which of these do I have to call tcp_close().

Call tcp_close() if you want to close a connection. It is probably
safest and simplest to do this in all cases you describe, then create a
new PCB, then call tcp_connect() again.

> which can I
> recall tcp_new() or should I reuse the old tcp_pcb

No, after you've called tcp_close(), don't touch the PCB again.

> and what do I need
> to monitor before I call either of these or call tcp_connect() again?

See above.

> Do I need to be checking the state of the pcb or something else?

You should get all the notifications you need via the RAW api callbacks.
It shouldn't be necessary to look into the PCB internals

Kieran

[vincent cui]

Dear :
I am porting the ftpd.c to my system, and find the following code is used to create a new connection for send / recv data.
But after tcp_connection return correctly, the ftpd_dataconnected callback is not executed.
Why ?

fsm->datapcb = tcp_new();
if (fsm->datapcb == NULL)
{
printf ("create data pcb fail \r\n");
}
if ( ERR_USE == tcp_bind(fsm->datapcb, &pcb->local_ip, 20))
{
printf(" port 20 is in used \r\n");
}

tcp_arg(fsm->datapcb, fsm->datafs);
if ( ERR_OK != tcp_connect(fsm->datapcb, &fsm->dataip, fsm->dataport, ftpd_dataconnected))
{
return 1;

[Kieran Mansley]

On Tue, 2011-09-06 at 09:21 +0000, vincent cui wrote:
> But after tcp_connection return correctly, the ftpd_dataconnected
> callback is not executed.

The callback is only called when the connection is fully established and
acknowledged by the server. If you not getting the callback in that
situation, then something is wrong.

Kieran

[vincent cui]

Thanks .

I check the ftp server in windows 7, it is not installed. Thank you ... the default install package seems not include ftp server

锘?Vincent Cui
Sr.Firmware Engineer
Mobile: +8613482482211
Tel: +86 21 34612525x6104
Fax: +86 21 34619770
E-Mail: vince...@enlogic.com
Shanghai EnLogic Electric Technology Co., Ltd.
Address: 1104-1106, Building A, No.391, Guiping Road, Xuhui District, Shanghai, 200233
http://www.enlogic.com

[vincent cui]

Sorry, Kieran

My windows7 system install the ftp server. Because I can get file from other ftp server ...

It is really odd.

锘?Vincent Cui
Sr.Firmware Engineer
Mobile: +8613482482211
Tel: +86 21 34612525x6104
Fax: +86 21 34619770
E-Mail: vince...@enlogic.com
Shanghai EnLogic Electric Technology Co., Ltd.
Address: 1104-1106, Building A, No.391, Guiping Road, Xuhui District, Shanghai, 200233
http://www.enlogic.com

-----Original Message-----
From: lwip-users-bounces+vincent.cui=enlog...@nongnu.org [mailto:lwip-users-bounces+vincent.cui=enlog...@nongnu.org] On Behalf Of Kieran Mansley
Sent: 2011年9月6日 17:27
To: Mailing list for lwIP users
Subject: Re: [lwip-users] tcp_connection cann't start the callback function after connection successfully

[vincent cui]

Keiran:

I trace the code, and found that the pcb->state is CLOSED. Why? It is created by tcp _new(),

锘?Vincent Cui
Sr.Firmware Engineer
Mobile: +8613482482211
Tel: +86 21 34612525x6104
Fax: +86 21 34619770
E-Mail: vince...@enlogic.com
Shanghai EnLogic Electric Technology Co., Ltd.
Address: 1104-1106, Building A, No.391, Guiping Road, Xuhui District, Shanghai, 200233
http://www.enlogic.com

-----Original Message-----
From: lwip-users-bounces+vincent.cui=enlog...@nongnu.org [mailto:lwip-users-bounces+vincent.cui=enlog...@nongnu.org] On Behalf Of Kieran Mansley
Sent: 2011年9月6日 17:27
To: Mailing list for lwIP users
Subject: Re: [lwip-users] tcp_connection cann't start the callback function after connection successfully

[Kieran Mansley]

On Tue, 2011-09-06 at 09:55 +0000, vincent cui wrote:
> I trace the code, and found that the pcb->state is CLOSED. Why?

Because connections start out as closed until you connect them. Can you
get a packet capture?

[vincent cui]

Keiran:

I use netstat to dump the currect tcp status. The attached show that "TCP 192.168.1.104:60530 192.168.1.48:ftp ESTABLISHED"

But the TCP core still enter the callback function.
The ftp at windows7 side message is followed:
------------------------------------------------------------------------------------
ftp> open 192.168.1.48
Connected to 192.168.1.48.
220 lwIP FTP Server ready.
331 User name okay, need password.
230 User logged in, proceed.
Anonymous login succeeded for vi...@vicui-LT.enlogic.com
ftp> get ff.dd
200 Command okay.
150 Opening BINARY mode data connection for ff.dd (8 bytes).

Halt here
----------------------------------------------------------------------------------------

The capture package show that my ftp server doesn't send file data to client...instead, the client pc send a ACK to my server.
Please check attached file

BTW, 192.168.1.48 is my server ip, 192.168.1.104 is my client ip

锘?Vincent Cui
Sr.Firmware Engineer
Mobile: +8613482482211
Tel: +86 21 34612525x6104
Fax: +86 21 34619770
E-Mail: vince...@enlogic.com
Shanghai EnLogic Electric Technology Co., Ltd.
Address: 1104-1106, Building A, No.391, Guiping Road, Xuhui District, Shanghai, 200233
http://www.enlogic.com

-----Original Message-----
From: lwip-users-bounces+vincent.cui=enlog...@nongnu.org [mailto:lwip-users-bounces+vincent.cui=enlog...@nongnu.org] On Behalf Of Kieran Mansley
Sent: 2011年9月6日 17:27
To: Mailing list for lwIP users
Subject: Re: [lwip-users] tcp_connection cann't start the callback function after connection successfully

[Kieran Mansley]

On Tue, 2011-09-06 at 10:17 +0000, vincent cui wrote:
> Please check attached file

A screenshot of the packet capture? If you have the packet capture,
please send that instead of a screenshot, as it has so much more useful
information.

Let me summarise what I think the situation is, please correct me if I'm
wrong:

- You have an FTP server running on your lwIP device.
- You use an FTP client on windows to try to connect to it
- It connects successfully and you can log in.
- You've called tcp_connect() on your lwIP device to open a data
connection.
- We see the SYN for this new connection in the packet capture, but
Windows seems to ignore it.
- You don't see the connected callback invoked (which makes sense, as it
is not connected)

I would guess that this is nothing to do with lwIP at all. It's
probably the common problem of Active FTP and firewalls, where you
windows PC is not allowing the data connection to be opened. Please
check that.

[vincent cui]

Keiran:

I will capture it again tomorrow... it is not related with windows 7 firewall... because it works with other FTP server running on CENTOS in another PC..
It is confusing too ... but I guess the root cause is the stack size of the ftp task created by FreeRTOS...

The stack is limit so that FreeRTOS has no stack space to store the pointer of call back function.

It is only my guess , I will try it tomorrow

锘?Vincent Cui
Sr.Firmware Engineer
Mobile: +8613482482211
Tel: +86 21 34612525x6104
Fax: +86 21 34619770
E-Mail: vince...@enlogic.com
Shanghai EnLogic Electric Technology Co., Ltd.
Address: 1104-1106, Building A, No.391, Guiping Road, Xuhui District, Shanghai, 200233
http://www.enlogic.com
-----Original Message-----
From: lwip-users-bounces+vincent.cui=enlog...@nongnu.org [mailto:lwip-users-bounces+vincent.cui=enlog...@nongnu.org] On Behalf Of Kieran Mansley
Sent: 2011年9月6日 18:36
To: Mailing list for lwIP users
Subject: Re: [lwip-users] tcp_connection cann't start the callback function after connection successfully

[vincent cui]

Keiran:

Centftp file is captured to windows7 -> centos
Myftp file is captured to windows -> my ftp.

Please help check

[vincent cui]

Keiran:

ftp works now ... the data port from PORT command is wrong....

锘?Vincent Cui
Sr.Firmware Engineer
Mobile: +8613482482211
Tel: +86 21 34612525x6104
Fax: +86 21 34619770
E-Mail: vince...@enlogic.com
Shanghai EnLogic Electric Technology Co., Ltd.
Address: 1104-1106, Building A, No.391, Guiping Road, Xuhui District, Shanghai, 200233
http://www.enlogic.com

-----Original Message-----
From: lwip-users-bounces+vincent.cui=enlog...@nongnu.org [mailto:lwip-users-bounces+vincent.cui=enlog...@nongnu.org] On Behalf Of Kieran Mansley
Sent: 2011年9月6日 18:36
To: Mailing list for lwIP users
Subject: Re: [lwip-users] tcp_connection cann't start the callback function after connection successfully

[Richie Bonventre]

So just an update on what I've managed to do to get things working - 
I think I was confused about a few lwip things and distracted by xilinx overhead other times.

The first thing that confused me was that when you successfully tcp_close() a connection, the tcp_pcb pointer that you made will still be a valid pointer to a tcp_pcb struct - the memory is not actually cleared. In addition, the state the struct returns to is identical to the state of a brand new tcp_pcb, "CLOSED". This caused me to believe that I could tcp_connect() again on the same tcp_pcb. As you've said, this is not actually ok. After tcp_close returns, lwip no longer knows about your tcp_pcb struct, and although the memory isn't cleared, lwip has designated that memory space as free so the next time you create a connection with tcp_new() it may place it there and overwrite your old tcp_pcb.

So my four situations:
1) I ctrl-C the listener on the other machine.
This will send some sort of packet to your lwip machine telling you the connection is closing. Lwip will move tcp_pcb into the CLOSE_WAIT state, and will call the recv_callback with a NULL argument. In the recv_callback you should then check that this packet is coming from your currently opened connection. If it isn't, it must be from an old connection that you already attempted to close, so you can ignore it. If it is, then you must call tcp_close() yourself. This then completes some more acking with the other machine and eventually places the tcp_pcb into the CLOSED state, and then calls the err_callback with ERR_ABRT. At this point, the tcp_pcb is "freed" and should not be touched again. A new tcp_pcb should be tcp_new()'d and then you can tcp_connect() again.

2) I try to connect from lwip but no cable is plugged in.

Lwip will try and send a SYN packet and wait for an ack. Until it gets the ack, a counter is being incremented until it times out. When it times out, it calls the err_callback with ERR_ABRT. It is not clear to me if the tcp_pcb is CLOSED at this point, but it looks like the memory is freed anyway. Should I have to call tcp_close() here? Once the memory is freed in lwip you can again tcp_new() and tcp_connect().

3) I try to connect from lwip and the cable is plugged in, but nothing is listening.

The other machine should immediately send Lwip back a rst packet. Lwip will then call the err_callback with ERR_RST. I noted though that at this point when the error callback is called, the old tcp_pcb is not yet closed nor freed. I was worried here that creating a new connection before the old one was closed would mess something up since the old connection is still in lwips active pcbs lists. Currently I just wait until the state of the old connection is closed, knowing that lwip will immediately close and free it after calling the callback. I can then tcp_new() and tcp_connect().

4) I connect fine, but then unplug the cable. My manually sent ping never gets a response.

If you call tcp_close() but the other machine is not connected to respond, the close request wont get acked and the tcp_pcb will sit in FIN_WAIT_1 forever. I guess it is ok to leave this connection hanging and start a new one. You wont get any callbacks so you'd have to call tcp_new() immediately after calling tcp_close(). When you eventually reconnect, the old connection will close and call the recv_callback with a NULL argument, but as described above you will know it's not from the current connection so you will ignore it anway. On the other hand, if you call tcp_abort(), lwip immediately gets rid of it and sets the connection to CLOSED, and then err_callback is called with ERR_ABRT. At this point you can tcp_new() and tcp_connect() again.

The other weird detail for me was that every time I closed the connection I had to re call the xilinx function platform_enable_interrupts() for the lwip timers to work. For some reason xilinx automatically turns off the interrupts whenever the connection was closed.

Hopefully this is somewhat accurate. At the least my code is stable for now.
Thanks,
Richie

[gold...@gmx.de]

Richie Bonventre wrote:

[..] In addition, the state the struct returns to is identical to the state of a brand new tcp_pcb, "CLOSED". This caused me to believe that I could tcp_connect() again on the same tcp_pcb. As you've said, this is not actually ok. After tcp_close returns, lwip no longer knows about your tcp_pcb struct, and although the memory isn't cleared, lwip has designated that memory space as free so the next time you create a connection with tcp_new() it may place it there and overwrite your old tcp_pcb.

Clearing the memory would not help: it might already be re-allocated and thus have a valid state. Plus "CLOSED" is 0, I think...

Anyway, it is documented that a pcb may not be referenced after tcp_close() returned ERR_OK (even if the pcb might not be deallocated at that point, it will eventually be).

So my four situations:
1) I ctrl-C the listener on the other machine.
This will send some sort of packet to your lwip machine telling you the connection is closing. Lwip will move tcp_pcb into the CLOSE_WAIT state, and will call the recv_callback with a NULL argument. In the recv_callback you should then check that this packet is coming from your currently opened connection. If it isn't, it must be from an old connection that you already attempted to close, so you can ignore it.

If you do get a NULL-recv-callback for an older connection, it means you haven't closed it correctly (remember to set the recv callback to NULL before calling tcp_close - that way you won't get informed when the remote FIN arrives, if you don't need that). However, it doesn't hurt to always call tcp_close() when you get a NULL-recv-callback.

If it is, then you must call tcp_close() yourself. This then completes some more acking with the other machine and eventually places the tcp_pcb into the CLOSED state, and then calls the err_callback with ERR_ABRT.

No, this is wrong: the err-callback is only called in an error condition (e.g. a timeout, RST received or something like that). It is *not* called in a normal graceful-close-scenario.

At this point, the tcp_pcb is "freed" and should not be touched again.

Again, do *not* reference the pcb after tcp_close() has returned ERR_OK, please read the documentation (doc/rawapi.txt)!

A new tcp_pcb should be tcp_new()'d and then you can tcp_connect() again.

2) I try to connect from lwip but no cable is plugged in.

Lwip will try and send a SYN packet and wait for an ack. Until it gets the ack, a counter is being incremented until it times out. When it times out, it calls the err_callback with ERR_ABRT. It is not clear to me if the tcp_pcb is CLOSED at this point, but it looks like the memory is freed anyway.

From an application point of view, a pcb is always freed *before* the err-callback is called.

Should I have to call tcp_close() here? Once the memory is freed in lwip you can again tcp_new() and tcp_connect().

3) I try to connect from lwip and the cable is plugged in, but nothing is listening.

The other machine should immediately send Lwip back a rst packet. Lwip will then call the err_callback with ERR_RST. I noted though that at this point when the error callback is called, the old tcp_pcb is not yet closed nor freed.

How did you notice that? The pcb is freed after the err-callback is called. There's no need to set the state to CLOSED before freeing the memory. Please don't try to assume anything from reading the pcb contents here.

Simon

[Richie Bonventre]

Ok, thanks for the clarifications!
In case you guys are interested, the project that I am working on is part of the readout electronics for the SNO+ experiment (http://snoplus.phy.queensu.ca/), an underground neutrino detector. We are using LwIP to replace piles and piles of slow, noisy ribbon cables and increase the experiment's readout speed. LwIP has been a super helpful tool for us so thanks for all the hard work!
Richie

[Kieran Mansley]

On Thu, 2011-09-22 at 17:57 -0400, Richie Bonventre wrote:
> LwIP has been a super helpful tool
> for us so thanks for all the hard work!

It's always good to hear how lwIP is being used. If you'd like to add a
brief description to the wiki that would be great:

http://lwip.wikia.com/wiki/Projects_that_use_lwIP

Kieran