File descriptor-backed I/O stream?

[Jim Porter]

Sometimes, I find that I need to interact directly with file descriptors
in my code, e.g. when working with POSIX pipes. Since a file descriptor
represents a kind of stream, it would be convenient to be able to wrap
it inside an I/O stream object. (This is what Boost.IOStreams does
currently[1].)

With C++11 having added the thread support library, including a way to
get the native handles of the various threading types (e.g. pthread_t),
it seems reasonable to me that the I/O streams library would do
something similar and add a file descriptor-backed I/O stream type. I
know even Windows supports file descriptors to some degree, but it also
uses HANDLEs as the native type for files.

The standard could simply allow implementations to open file-descriptor
streams (`fdstream`s) using whatever implementation-defined types the
platform supports, so POSIX would use FDs and Windows would use HANDLEs
and FDs. The fdstream could also be made optional, like the
native_handle members of the thread support classes, for platforms that
have no reasonable implementation of an fdstream.

I think this would make writing C++ that interacts with POSIX (or Win32)
code a fair bit easier. If this sounds reasonable to others, I can work
on a proposal for it.

- Jim

[1]
http://www.boost.org/doc/libs/1_55_0/libs/iostreams/doc/classes/file_descriptor.html#file_descriptor

[Ville Voutilainen]

It sounds very reasonable to me, so I'd be very happy if someone would explore
the possibilities of supporting creating fstreams over a native handle.

[Myriachan]

Win32 is a weird one in this case.  "HANDLE"s are a concept from the kernel (*), whereas "int" file descriptors are a wrapper in Microsoft's C standard library - which is optional and/or has many variants.  The C library isn't built into the OS in Windows (**), unlike POSIX.  I feel as though the implementation should choose only one.

For platforms for which there is no lower-level concept, the definition of the function to get the lower-level type could be to return a pointer to itself.  Perhaps also, when streams are a thin wrapper around stdio in an implementation for which there is no lower level than FILE *, such a function could return FILE *.

An alternative would be to have a function to get the next level deeper.  get_next_native_handle(get_next_native_handle(f)) would return a HANDLE in Win32, maybe.


(*) Except that kernel32.dll fakes it for stdin/stdout/stderr by using special HANDLE values that it special-cases.

(**) Windows comes with various C runtimes for use by its own bundled applications, but none of them are to be used directly by applications.  MinGW does, but it shouldn't.


Melissa

[Ville Voutilainen]

On 24 October 2014 11:53, Myriachan <myri...@gmail.com> wrote:
> For platforms for which there is no lower-level concept, the definition of
> the function to get the lower-level type could be to return a pointer to
> itself. Perhaps also, when streams are a thin wrapper around stdio in an
> implementation for which there is no lower level than FILE *, such a
> function could return FILE *.

Wrapping iostreams over FILE* is vastly different from what is aimed for
here - file descriptors, whether ints or HANDLEs, are necessary for
cases where you need to use a platform-specific facility to open a file
and then want to wrap an fstream over it. As far as I know, very few such
platform-specific facilities return FILE*.

[Вадим]

I proposed it here in the most minimal configuration to be practically useful (i.e. just exposing a purely implementation defined native_handle) a year ago
https://groups.google.com/a/isocpp.org/forum/?fromgroups#!searchin/std-proposals/native_handle/std-proposals/oCEErQbI9sM/Oy57TLzCpj4J
but wasn't able to support the proposal with enough arguments at the time. I hope you will be better at pushing it through, because it's a feature that sometimes are sorely needed.

[Myriachan]

Ville, I used FILE * as a theoretical example of an embedded system in which the code stands alone with no operating system. A C runtime library in this sort of environment could use the FILE structure to manage its file I/O if it wanted, rather than have a more-traditional file descriptor model with stdio above it. I just thought that FILE * could resolve the issue of having no reasonable answer to native_handle.

Vadim(*), for the question posed in that thread, I think that the definition ought to be that the handle remains owned by the object from which it came. Only during the lifetime should it be valid. Users would have to duplicate the handle if they need one they own. I suppose that there could be a way to destroy/deactivate the object and return the handle into the caller's possession, like a move constructor and the thread thing.

(*) I hope I read your Cyrillic name correctly...sorry if not.

[Jim Porter]

On 10/24/2014 3:00 PM, Myriachan wrote:
> Ville, I used FILE * as a theoretical example of an embedded system
> in which the code stands alone with no operating system. A C runtime
> library in this sort of environment could use the FILE structure to
> manage its file I/O if it wanted, rather than have a more-traditional
> file descriptor model with stdio above it. I just thought that FILE
> * could resolve the issue of having no reasonable answer to
> native_handle.

I think there *is* a reasonable answer to `native_handle`: an int on
POSIX systems and a HANDLE on Windows systems. (And, since Windows also
lets you refer to a file via a file descriptor, the Windows fdstream
could also have a `fileno` member function. It might be harder to
provide room in the standard for that, though.)

However, I think it's *more* important to be able to create an iostream
object from a native file handle. Especially on a POSIX system, it's
very common to create pipes, dup file descriptors, and so on. Having a
way to turn these into iostreams would make it easier to write POSIX
code in C++, and it's a lot narrower in scope than trying to add a C++
I/O streams equivalent to pipe(2) and friends.

> Vadim(*), for the question posed in that thread, I think that the
> definition ought to be that the handle remains owned by the object
> from which it came. Only during the lifetime should it be valid.
> Users would have to duplicate the handle if they need one they own.
> I suppose that there could be a way to destroy/deactivate the object
> and return the handle into the caller's possession, like a move
> constructor and the thread thing.

I think it would be sufficient for fstream::native_handle (note: not
fdstream) to work pretty much like fileno(3). That said, I don't feel as
strongly about there being an fstream::native_handle as I do about an
fdstream class.

- Jim

[Ville Voutilainen]

On 25 October 2014 00:08, Jim Porter <jvp...@g.rit.edu> wrote:
> However, I think it's *more* important to be able to create an iostream
> object from a native file handle. Especially on a POSIX system, it's very
> common to create pipes, dup file descriptors, and so on. Having a way to
> turn these into iostreams would make it easier to write POSIX code in C++,
> and it's a lot narrower in scope than trying to add a C++ I/O streams
> equivalent to pipe(2) and friends.

Precisely so, fully agreed. There are also things like system-specific
flags given
to open() and functions like openat() that would become much more convenient
to wrap.

[Myriachan]

Yes, I agree that it's important to be able to make a stream object from a native handle type.

But what to do about environments like Windows NT, where there are two layers of file handles? Even fileno and fdopen are not standardized.

Melissa

[Jim Porter]

On 10/24/2014 5:46 PM, Myriachan wrote:
> But what to do about environments like Windows NT, where there are
> two layers of file handles? Even fileno and fdopen are not
> standardized.

If we ignore the `native_handle` member function for the time being, the
reasonable thing to do would be to allow an fdstream on Windows to be
constructed with either an int (file descriptor) or a HANDLE. This is,
in fact, what Boost.IOStreams does.

- Jim

[Matheus Izvekov]

On Friday, October 24, 2014 8:46:20 PM UTC-2, Myriachan wrote:

Yes, I agree that it's important to be able to make a stream object from a native handle type.

But what to do about environments like Windows NT, where there are two layers of file handles?  Even fileno and fdopen are not standardized.

Maybe instead of, or in addition to, supporting native handles, it would be desirable to support a C++ stream equivalent of GNU's fopencookie.
 

[Thiago Macieira]

On Friday 24 October 2014 16:00:20 Matheus Izvekov wrote:
> Maybe instead of, or in addition to, supporting native handles, it would be
> desirable to support a C++ stream equivalent of GNU's fopencookie.

The C++ stream equivalent of fopencookie is std::iostream itself. Just derive
from istream or ostream.

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center
PGP/GPG: 0x6EF45358; fingerprint:
E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358

[Matheus Izvekov]

On Friday, October 24, 2014 9:39:21 PM UTC-2, Thiago Macieira wrote:

The C++ stream equivalent of fopencookie is std::iostream itself. Just derive
from istream or ostream.

Sure, but it's not as simple to use, and I am not sure it works as well in practice.

[Jim Porter]

What you're suggesting seems significantly more complex than just adding
a std::fdstream class. Still, Boost.IOStreams does a lot to try to make
the creation of new streams easier. You could probably pull out some of
the pieces from that to make a proposal.

- Jim

[Thiago Macieira]

I'm not disputing that. I personally think the iostream API is horrible. But
it's what we have.

Unless you want to suggest adding a new API and deprecating iostreams. I
wouldn't mind.

[Olaf van der Spek]

Op vrijdag 24 oktober 2014 08:35:49 UTC+2 schreef Jim Porter:

I think this would make writing C++ that interacts with POSIX (or Win32)
code a fair bit easier. If this sounds reasonable to others, I can work
on a proposal for it.

Are you looking for a iostream-like interface or a binary-IO posix-like interface (just read/write, etc)?

I'd love to see the latter as a thin wrapper around native file IO.