Returning from ACE_SOCK_Stream::recv() after closing socket

[elhoushi]

Dear All,
I am having an application which deals with TCP sockets using ACE.

I have one thread waiting to a receive any packet using blocking
function :
recv (void *buf, size_t n, const ACE_Time_Value *timeout = 0) const;
using timeout = 0 so that it will wait for ever.

In another thread, I close that same socket using close() function.

The problem is that in Linux, the recv function keeps blocking even so
the socket has been closed.
Although in Windows, with that same code, when the socket in one
thread closes, the recv function in the other thread returns -1.

How can I solve this problem?

Thanks in advance.

[Douglas C. Schmidt]

Hi Mostafa,

>I am having an application which deals with TCP sockets using ACE.

To ensure that we have proper version/platform/compiler information,
please make sure you fill out the appropriate problem report form
(PRF), which is in

$ACE_ROOT/PROBLEM-REPORT-FORM
$TAO_ROOT/PROBLEM-REPORT-FORM

or in

$ACE_ROOT/BUG-REPORT-FORM
$TAO_ROOT/BUG-REPORT-FORM

in older versions of ACE+TAO. Make sure to include this information
when asking any questions about ACE+TAO since otherwise we have to
"guess" what version/platform/compiler/options you've using, which is
very error-prone and slows down our responsiveness. If you don't use
the PRF, therefore, it is less likely that someone from the core
ACE+TAO developer team will be able to answer your question.
Naturally, we encourage and appreciate other members of the ACE+TAO
user community who can respond to questions that they have the answers
to.

>I have one thread waiting to a receive any packet using blocking
>function :
>recv (void *buf, size_t n, const ACE_Time_Value *timeout = 0) const;
>using timeout = 0 so that it will wait for ever.
>
>In another thread, I close that same socket using close() function.
>
>The problem is that in Linux, the recv function keeps blocking even so
>the socket has been closed.
>Although in Windows, with that same code, when the socket in one
>thread closes, the recv function in the other thread returns -1.

Right - the behavior of sockets in this case is not standardized, so
different operating systems will behave differently.

>How can I solve this problem?

There are several portable solutions:

. Pass a timeout value to recv() and have it return periodically check
to see if the thread has been shutdown.

. Use an ACE Reactor to wait for data or for a shutdown event to occur
on the ACE Reactor's notification pipe.

Doug
--
Dr. Douglas C. Schmidt Professor and Associate Chair
Electrical Engineering and Computer Science TEL: (615) 343-8197
Vanderbilt University WEB: www.dre.vanderbilt.edu/~schmidt
Nashville, TN 37203 NET: d.sc...@vanderbilt.edu

[elhoushi]

Thank you very much Dr. Doug for your help.
I have not found the PROBLEM-REPORT-FORM.

Concerning the details of my environment, the problem existed on both
SUSE 10 & RedHat:
- Operating System: SUSE Linux Enterprise Server 10 SP1 (x86_64) -
Kernel \r (\l)
- ACE library version: 5.5.7
- g++ compiler version: gcc version 4.1.2 20070115 (prerelease)

- Operating System: Red Hat Enterprise Linux Server release 5.2
(Tikanga)
- ACE library version: 5.5.7
- g++ compiler version: gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)

Concerning using timeout value on recv() I do not want to use it for 2
reasons:
- This will make high CPU utilization since we are looping through a
receive function
- There may be a rare chance that a packet is received while I am not
withing the receive function
Or am I wrong?

Concerning the ACE Reactor, I will try to read what it is and try to
use it.

Thanks

[Douglas C. Schmidt]

Hi Mostafa,

>Thank you very much Dr. Doug for your help.

You are very welcome.

>I have not found the PROBLEM-REPORT-FORM.

For future refrence please see

ACE_ROOT/PROBLEM-REPORT-FORM

or point your browser at

http://www.dre.vanderbilt.edu/~schmidt/DOC_ROOT/ACE/PROBLEM-REPORT-FORM

>Concerning the details of my environment, the problem existed on both
>SUSE 10 & RedHat:
>- Operating System: SUSE Linux Enterprise Server 10 SP1 (x86_64) -
>Kernel \r (\l)
>- ACE library version: 5.5.7
>- g++ compiler version: gcc version 4.1.2 20070115 (prerelease)
>
>- Operating System: Red Hat Enterprise Linux Server release 5.2
>(Tikanga)
>- ACE library version: 5.5.7
>- g++ compiler version: gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)
>
>Concerning using timeout value on recv() I do not want to use it for 2
>reasons:
> - This will make high CPU utilization since we are looping through a
>receive function
>- There may be a rare chance that a packet is received while I am not
>withing the receive function
>Or am I wrong?

Those are both good reasons not to use the timeout approach for recv().

>Concerning the ACE Reactor, I will try to read what it is and try to
>use it.

Ok, sounds good. There's lots of examples of the ACE_Reactor in

ACE_ROOT/tests/

and it's described in Chapters 3 and 4 of the C++NPv2 book <www.cs.wustl.edu/~schmidt/ACE/book2/>

Thanks,

[Douglas C. Schmidt]

Hi Mostafa,

Thanks very much for your email. Please make sure to send all
questions related to TAO or ACE to the ACE mailing list or ACE+TAO
newsgroup, rather than to me directly since I travel frequently and
often don't have ready access to email.

> I have had a look on ACE Reactors...
> Unfortunately, my whole application was based on ACE Threads... restructuring the whole application to event handlers is
> too time consuming and means that I will meet other problems and may find new bugs...
>
> Isn't there another solution?

Sure, to wake up threads blocked on I/O calls you can also use:

. Signals, which not very portable/reliable

. select() and ACE_Pipe (which is what the ACE_Reactor does internally).

Doug

[elhoushi]

On Jul 17, 3:55 pm, "Douglas C. Schmidt" <schm...@dre.vanderbilt.edu>
wrote:

Concerning Signals: I have tried to use it but I am not sure how. I
read that sending a SIGALRM signal to a thread will cause a blocking I/
O operation to return is that true.
I used a kill_thrd() function to do that and the result was that the
whole application closed with a "Alarm Clock" message on the screen.
I tried to handle the SIGALRM method, the application did not close,
but at the same time the original bug did not solve: the recv function
remained blocking even after the socket is closed.

Concerning ACE_Pipe: I think using it means that I will have to
restructure my whole application since my application is based on TCP
sockets and threads sending and receiving to each other.

Concerning select: I am not sure how to use it.

But I found one possible modification to the ACE and I have made it
and I am in the process of re-building ACE.
The possible modification is the following:
File ace/ACE.inl:
Line: 240
The function handle_read_ready() is written as follows:

ACE_INLINE int
ACE::handle_read_ready (ACE_HANDLE handle,
const ACE_Time_Value *timeout)
{
return ACE::handle_ready (handle,
timeout,
1,
0,
0);
}

I though that it should change to:
ACE_INLINE int
ACE::handle_read_ready (ACE_HANDLE handle,
const ACE_Time_Value *timeout)
{
return ACE::handle_ready (handle,
timeout,
1,
0,
1);//change exception from 0 to 1
}

so that the function will return if an exception occurs in the socket.
Isn't closing the socket an exception which should make the function
return?

[Douglas C. Schmidt]

Hi Mostafa,

> Concerning Signals: I have tried to use it but I am not sure how. I read that sending a SIGALRM signal to a thread will

> cause a blocking I/O operation to return is that true.

> I used a kill_thrd() function to do that and the result was that the whole application closed with a "Alarm Clock" message
> on the screen.
> I tried to handle the SIGALRM method, the application did not close, but at the same time the original bug did not solve:
> the recv function remained blocking even after the socket is closed.

Using signals is non-trivial but it should work. I recommend you read a
good book on UNIX systems programming for info on this. Richard Stevens
has written some good books that will provide the info you need.

> Concerning ACE_Pipe: I think using it means that I will have to restructure my whole application since my application is
> based on TCP sockets and threads sending and receiving to each other.

No, that's not what I'm talking about. The trick is never to do a
blocking read()/recv() on an I/O handle, but instead to use select() to
wait on the I/O *and* on a pipe handle. If you need to shutdown a
thread, therefore, all you need to do is to write 1 byte to the
appropriate pipe, which will cause select() to return and the thread can
exit cleanly.

> Concerning select: I am not sure how to use it.

Please see a good book on UNIX network programming, such as the Richard
Stevens books.

> But I found one possible modification to the ACE and I have made it and I am in the process of re-building ACE.
> The possible modification is the following:
> File ace/ACE.inl:
> Line: 240
> The function handle_read_ready() is written as follows:
>
> ACE_INLINE int
> ACE::handle_read_ready (ACE_HANDLE handle,
>                         const ACE_Time_Value *timeout)
> {
>   return ACE::handle_ready (handle,
>                             timeout,
>                             1,
>                             0,
>                             0);
> }
>
> I though that it should change to:
> ACE_INLINE int
> ACE::handle_read_ready (ACE_HANDLE handle,
>                         const ACE_Time_Value *timeout)
> {
>   return ACE::handle_ready (handle,
>                             timeout,
>                             1,
>                             0,
>                             1);//change exception from 0 to 1
> }
>
> so that the function will return if an exception occurs in the socket. Isn't closing the socket an exception which should
> make the function return?

No, I don't think so, but try it out and let us know if it works.

Thanks,

Doug

[D.J. Stachniak]

Robert Love's Linux System Programming is another good book with a chapter
on signals - and some sage advice at the end to avoid them whenever
possible. ;) Proceed with caution...

Hi Mostafa,

Thanks,

Doug
_______________________________________________
ace-users mailing list
ace-...@list.isis.vanderbilt.edu
http://list.isis.vanderbilt.edu/mailman/listinfo/ace-users

[Chris Cleeland]

On Jul 20, 2009, at 7:44 AM, Douglas C. Schmidt wrote:

>> Isn't closing the socket an exception which should
>> make the function return?
>
> No, I don't think so, but try it out and let us know if it works.

With BSD sockets API, closure is not an exceptional condition.
Closure is something normal in the lifecycle of a socket, and triggers
both read- and write-ready. It does not trigger the exception mask.
The only thing I remember that triggers the exception mask is receipt
of OOB data.

---
Chris Cleeland, Principal Software Engineer
http://www.theaceorb.com AND http://www.ociweb.com

[Douglas C. Schmidt]

Hi Chris,

>On Jul 20, 2009, at 7:44 AM, Douglas C. Schmidt wrote:
>
>>> Isn't closing the socket an exception which should
>>> make the function return?
>>
>> No, I don't think so, but try it out and let us know if it works.
>
>With BSD sockets API, closure is not an exceptional condition.
>Closure is something normal in the lifecycle of a socket, and triggers
>both read- and write-ready. It does not trigger the exception mask.
>The only thing I remember that triggers the exception mask is receipt
>of OOB data.

Right, that was my recollection, as well, though I was hoping to be
surprised ;-)

Thanks,

[elhoushi]

Dear all,

It finally worked using the ACE Pipe method!

Here is what I did:

In one thread:
while( m_bthreadRunning )
{
// Wait until a new connection request is received
if( m_oAceSocketAcceptor.accept(m_oAceSocket, 0, &m_oAceTime, 1 )
== -1 )
{
continue;
}

while(m_bthreadRunning)
{
bytesReceived = 0;

// Attempt to read a group of bytes over the ACE connection
#ifndef LINUX
bytesReceived = m_oAceSocket.recv(m_ucUsbBytes, MAX_ACE_BYTES);
#else
ACE_Handle_Set handle_set;
handle_set.set_bit (m_oAceSocket.get_handle());
handle_set.set_bit (m_oAcePipe.read_handle());
int select_width = max(int (m_oAceSocket.get_handle()), int
(m_oAcePipe.write_handle())) + 1;
ACE::select(select_width, &handle_set);
m_oAcePipe.recv(nullBuf,0);
bytesReceived = m_oAceSocket.recv(m_ucBytes, MAX_ACE_BYTES, &
(ACE_Time_Value(0)));
#endif
if( bytesReceived == -1 )
{
m_oAceSocket.close(); // Close the connection and wait for
another request if the connection fails for any reason
m_oAceSocketAcceptor.close();
break;
}

// Handle the bytes received
if(bytesReceived > 0)
{
// DO SOMETHING
}
}
}

In the other thread, when we want to close the socket:
void Stop()
{
m_bThreadRunning = false;

try
{
// close the connection since the thread is terminating
m_oAceSocket.close();
m_oAceSocketAcceptor.close();

#ifdef LINUX
m_oAcePipe.send("A",1);
#endif
}
catch(...){}

[Douglas C. Schmidt]

Hi Mostafa,

>It finally worked using the ACE Pipe method!

Super!!

Perfect - you got it, well done!!!