[Boost-users] [Asio] Making custom std::streambuf for async_read

[kna...@online.de]

At the moment I'm using async_read with boost::asio::streambuf and this
works fine but I need a custom stream where every invoke to async_read
appends the data to my stream. The stream should be based on
std::streambuf because I want to keep the asio interface hidden to the
user.

My first idea was to search for the problem and I ended up with reading
about MutableBufferSequence but I don't know if this is the right way
and how to implement this in my streambuf object. Any help is higly
appreciated.
_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users

[Gavin Lambert]

On 9/08/2016 17:13, kna...@online.de wrote:
> At the moment I'm using async_read with boost::asio::streambuf and this
> works fine but I need a custom stream where every invoke to async_read
> appends the data to my stream.

Asio streambuf already does this, so I'm not sure why you'd want to make
a custom one on that basis.

> The stream should be based on std::streambuf because I want to keep
> the asio interface hidden to the user.

While I understand the motivation, note that some of the performance of
Asio relies on the compiler being able to inline things. There's
certainly nothing stopping you from hiding it but you'll lose some of
the benefits this way. Some of the special features such as strands and
custom allocators also rely on the way that the templates interact.

And Asio's streambuf is already based on std::streambuf; to make a
custom one you'd end up completely duplicating it anyway.

> My first idea was to search for the problem and I ended up with reading
> about MutableBufferSequence but I don't know if this is the right way
> and how to implement this in my streambuf object. Any help is higly
> appreciated.

Asio does support using many other data sequences as buffers, so you can
just use something else other than streambuf if you wish to.



http://www.boost.org/doc/libs/1_61_0/doc/html/boost_asio/overview/core/buffers.html

[kna...@online.de]

Thank you for the response so far. My idea is to prevent unnecessary
copy operations when receiving large files with tcp. For this reason I
thought about implementing my own stream, which writes direct to file or
memory, depending on the situation.

I know that I can connect std::iostream to the asio streambuf but what
about moving the received memory to a std::ofstream or std::stringstream
without having the data twice in memory. Is that possible?

Maybe I'm concerned too much about that problem and a simple copy
operation to the std stream works in most cases.

[Gavin Lambert]

On 11/08/2016 07:16, kna...@online.de wrote:
> Thank you for the response so far. My idea is to prevent unnecessary
> copy operations when receiving large files with tcp. For this reason I
> thought about implementing my own stream, which writes direct to file or
> memory, depending on the situation.
>
> I know that I can connect std::iostream to the asio streambuf but what
> about moving the received memory to a std::ofstream or std::stringstream
> without having the data twice in memory. Is that possible?

ASIO does provide posix::stream_descriptor and windows::stream_handle,
which can wrap native file descriptors and handles (resp.) and provide
async operations on them. Thus you can read from the network into one
asio streambuf and then pass the same streambuf to the file write
operation. This is probably as close to zero-copy as you can get.

Though other than scalability and blocking concerns there's not much
difference between this and just doing a blocking write to a regular
fstream -- at the point you're hitting disk the cost of copying memory
is negligible.

For writing direct to memory, you can simply supply the target memory as
a buffer to the initial read (making sure to keep track of the origin
and size correctly as data blocks incrementally arrive, remembering that
short reads are likely). This will quite literally be zero-copy (bar
any copies that occur inside the network stack); no special streams
required.

[kna...@online.de]

> ASIO does provide posix::stream_descriptor and windows::stream_handle,
> which can wrap native file descriptors and handles (resp.) and provide
> async operations on them. Thus you can read from the network into one
> asio streambuf and then pass the same streambuf to the file write
> operation. This is probably as close to zero-copy as you can get.
>
> Though other than scalability and blocking concerns there's not much
> difference between this and just doing a blocking write to a regular
> fstream -- at the point you're hitting disk the cost of copying memory
> is negligible.
>
> For writing direct to memory, you can simply supply the target memory
> as a buffer to the initial read (making sure to keep track of the
> origin and size correctly as data blocks incrementally arrive,
> remembering that short reads are likely). This will quite literally
> be zero-copy (bar any copies that occur inside the network stack); no
> special streams required.

I'm using my library for HTTP handling and receiving HTTP POST requests.
When the transfer is chunked then every call to the complete handler can
append a small bit of the stream to a stringstream or a file with the
standard c++ features. This would be the most portable way than using
stream_descriptor or stream_handle.