Custom Socket streambuf and Non-blocking IO

[ron.r...@gmail.com]

I've created my own socket stream library and I'm now adding a few stream manipulators to set a few of the socket options. Things like keepalive/nokeepalive and block/nonblock.

The problem I'm having is with non blocking reading. What I'd like to have is a read return nothing if no input is pending (empty string via >> or read return 0), but without the EOF flag set.

Here's my underflow method within my socket streambuf:

sockets::socketbuf::int_type sockets::socketbuf::underflow() {
if (gptr() >= egptr()) {
// The buffer has been exhausted, read more in from the socket.

int res = read(sockfd, &_ibuf.front(), _ibuf.size());
if (res == 0)
return traits_type::eof();

if (res < 0) {
if (errno == EAGAIN or errno == EWOULDBLOCK) {
// Non blocked IO.
// Reset the input buffer and return ???
setg(&_ibuf.front(), &_ibuf.front(), &_ibuf.front());
return 0; // How is this interrupted by the stream?
} else {
throw sockets::exception("socket read error");
}
}

setg(&_ibuf.front(), &_ibuf.front(), &_ibuf.front() + res);
}

return traits_type::to_int_type(*gptr());
}

// Possible example of it's usage (a bit of a bad example).

sockets::iocstream httpclient("www.some.server", "http");
httpclient << sockets::nonblock;
httpclient << "GET /index.html" << std::endl;

std::string results;
httpclient >> results;
while (result.empty() && !httpclient.eof()) {
// Do other stuff here.
httpclient >> results;
}

This functionality is 100% necessary for this library, but could be useful in other places. I do have an have an has_input_pending() method that uses poll, so if it's not possible then no worries.

[Chris Vine]

You cannot do proper asynchronous (non-blocking) i/o using the standard
streambuffers. basic_streambuf::underflow() returns a signed integer
representing either the character at the start of the "pending
sequence" (the front of the buffered characters available) following the
read operation, or traits_type::eof() (normally -1) if end-of-file or an
error has arisen. As far as I can tell, underflow() returning 0 means
that a character of value '\0' has been received and is at the front of
the buffer, which would conflict with the fact that after underflow()
gptr() does not in fact point to that (non-existent) '\0' character.
The call to uflow() which invokes underflow() (and which does advance
gptr()) will therefore cause the character in fact at the gptr()
position to be lost.

In short, the standard streams (and streambuffers) expect a read
operation to block until the read request has been satisfied unless
either end-of-file has been reached or an error has arisen.

You could fudge it by ensuring that underflow() will never in
fact return with nothing added to the buffer. You could do
this by extracting a single character at a time from the buffer,
looping on basic_streambuf::in_avail() and when the buffer is empty
testing the underlying file descriptor for readiness with poll() or
select() in your asynchronous event loop. One of the problems with
this when using linux (if that is your OS) is that select() and poll()
on linux are not POSIX compliant and can spuriously report a file
descriptor as ready, so this arrangement could still block when you do
not want it to or (if the file descriptor is set non-blocking) cause
EAGAIN or EWOULDBLOCK to put the streambuffer in an error state when
there is in fact no error.

Chris

[Chris Vine]

I should add that you _can_ do it with coroutines in association with
whatever asynchronous event loop you are using. What your underflow()
would do on encountering EAGAIN or EWOULDBLOCK is to yield on the
coroutine instead of returning, and that coroutine is then resumed by
the event loop when select() or poll() indicates that the descriptor has
become ready for reading.

I have written a library in another language (a language which supports
coroutines and explicit continuations), that does exactly that.

I believe coroutines did not make the cut for C++17, although I also
believe that Microsoft's VS does in fact support its own version of
them. So if you are programming with windows you could try writing your
underflow() using them.

Chris

[Ron]

Thanks for the replies. It's pretty much as I first expected. As flexible as the stream API is in c++, it's proving to have some short comings. Right now this library is for Linux systems, but eventually will be ported to Windows via mingw. I don't use Windows enough to justify Visual Studios.

The coroutines suggestion has given me an idea of a possible internal event dispatcher when a socket is ready to read (or write). I think I saw something like this in tcl. This might work to prevent client side lockups. If a server doesn't respond exactly as expected, the blocking IO tends to be a problem.

[Jorgen Grahn]

On Mon, 2017-04-03, Chris Vine wrote:
...

> select() in your asynchronous event loop. One of the problems with
> this when using linux (if that is your OS) is that select() and poll()
> on linux are not POSIX compliant and can spuriously report a file
> descriptor as ready,

IIRC, this was (last time I looked, almost a decade ago) limited to UDP
sockets. But you're still right: they don't feel like promising POSIX
compliance here, so we can't rely on it.

> so this arrangement could still block when you do
> not want it to or (if the file descriptor is set non-blocking) cause
> EAGAIN or EWOULDBLOCK to put the streambuffer in an error state when
> there is in fact no error.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Chris Vine]

On 3 Apr 2017 12:20:43 GMT

Jorgen Grahn <grahn...@snipabacken.se> wrote:
> On Mon, 2017-04-03, Chris Vine wrote:
> ...
> > select() in your asynchronous event loop. One of the problems with
> > this when using linux (if that is your OS) is that select() and
> > poll() on linux are not POSIX compliant and can spuriously report a
> > file descriptor as ready,
>
> IIRC, this was (last time I looked, almost a decade ago) limited to
> UDP sockets. But you're still right: they don't feel like promising
> POSIX compliance here, so we can't rely on it.

The man page for select, which is imported into the man page for poll(),
says:

"Under Linux, select() may report a socket file descriptor as "ready
for reading", while nevertheless a subsequent read blocks. This
could for example happen when data has arrived but upon
examination has wrong checksum and is discarded. There may be other
circumstances in which a file descriptor is spuriously reported as
ready. Thus it may be safer to use O_NONBLOCK on sockets that should
not block."

As you say, if they don't feel like promising POSIX behaviour I will
not rely on POSIX behaviour, and it's not really a hardship: if you
are doing asynchronous i/o you must cater for EAGAIN and EWOULDBLOCK so
it doesn't really matter if very occasionally select() gives you
one more EAGAIN than you were expecting.

[Chris Vine]

There are coroutine libraries out there, for C at least. One which
comes to mind is GNU Pth. But then you risk imposing considerable
restraints on users of your library - I have never used Pth but it
doesn't look, for example, as if Pth has any provision for combining it
with native OS threads. One consequence is that nothing in the program
can block except by specialist functions which suspend such as
pth_read(), or everything will block. So once you put pth_read() in
your std::streambuf::overflow() function, all your i/o ends up having to
be done by the coroutine library.

(My coroutine library does use native threads, up to a point, in the
sense that user code can invoke native worker threads which can block
and there can be as many event loops as wanted each running in their
own native thread with each operating its own intra-thread
asynchronicity - say, one native thread per CPU. Intra-thread
synchronization uses CSP-style "meetings" and synchronization between
native threads uses message passing.)

The most promising synthesis of these kinds of ideas which I have seen
is this one: https://github.com/wingo/fibers/wiki/Manual . But that
isn't going to do you any good C++ wise, as it is written in a
different language and runs on a VM partly designed for the purpose.

[Ron]

I did stumble across a solution for this problem and thought I'd share it. In the stream manipulator, I set the stream exceptions flags: ios->exceptions(std::ostream::badbit). This now lets exceptions thrown in the underflow method to be re-thrown and caught outside of the stream. When the socket read returns EAGAIN or EWOULDBLOCK I throw an ionotready exception that can be caught and sleep/attempts counter/failed actions can be take rather than locking up the program.

Hopefully this can help someone with a similar problem ;)

[woodb...@gmail.com]

On Monday, April 3, 2017 at 8:14:46 AM UTC-5, Chris Vine wrote:
> On 3 Apr 2017 12:20:43 GMT
> Jorgen Grahn <grahn...@snipabacken.se> wrote:
> > On Mon, 2017-04-03, Chris Vine wrote:
> > ...
> > > select() in your asynchronous event loop. One of the problems with
> > > this when using linux (if that is your OS) is that select() and
> > > poll() on linux are not POSIX compliant and can spuriously report a
> > > file descriptor as ready,
> >
> > IIRC, this was (last time I looked, almost a decade ago) limited to
> > UDP sockets. But you're still right: they don't feel like promising
> > POSIX compliance here, so we can't rely on it.
>
> The man page for select, which is imported into the man page for poll(),
> says:
>
> "Under Linux, select() may report a socket file descriptor as "ready
> for reading", while nevertheless a subsequent read blocks. This
> could for example happen when data has arrived but upon
> examination has wrong checksum and is discarded. There may be other
> circumstances in which a file descriptor is spuriously reported as
> ready. Thus it may be safer to use O_NONBLOCK on sockets that should
> not block."
>

I've got Linux 4.14 and my docs for select() have the above text,
but the docs for poll() don't. I'm not sure why this doesn't get
fixed and not sure if this problem is confined to select() or also
affects poll().

> As you say, if they don't feel like promising POSIX behaviour I will
> not rely on POSIX behaviour, and it's not really a hardship: if you
> are doing asynchronous i/o you must cater for EAGAIN and EWOULDBLOCK so
> it doesn't really matter if very occasionally select() gives you
> one more EAGAIN than you were expecting.

In one of my programs I was able to stop using poll() as there's
only one socket involved. I'm still using poll(), though, in
this program:
https://github.com/Ebenezer-group/onwards/blob/master/src/cmw/tiers/cmwA.cc

so would like to know if I should take some defensive
measures related to this. FreeBSD seems to be free of
these problems.


Brian
Ebenezer Enterprises
http://webEbenezer.net

[woodb...@gmail.com]

I was looking at docs in the "3P" section on poll() -- I guess
the 'P' may stand for Posix. When I checked poll() in another
section, there's a note referring you to select() spurious bugs.
So I guess this is still a problem for poll().

[Jorgen Grahn]

Yes, and I think the man page states so clearly. Another hint is that
it's written in Legalese.

> When I checked poll() in another
> section, there's a note referring you to select() spurious bugs.
> So I guess this is still a problem for poll().

Yes, and also for epoll(7). The problem/feature is in the kernel code
that's underlying for all these functions.

[woodb...@gmail.com]

It looks like this may finally be fixed. What I wrote last
year was about Linux 4.14. Now I have 4.19 installed and I
can't find any mention of this problem in the docs for poll,
(p)select or epoll. Am I missing something? Maybe my
perseverance is starting to pay off. Thanks.



Brian
Ebenezer Enterprises - In G-d we trust.
https://github.com/Ebenezer-group/onwards