Re: [Boost-users] [asio] Sending large packages
Nicola Michael Gutberlet
Doesn’t anyone have any ideas? Did anyone receive this mail?
Best regards and a happy new year ;-)
From: Nicola
Michael Gutberlet [mailto:nicola.g...@hhi.fraunhofer.de]
Sent: 15 December 2009 17:28
To: 'boost...@lists.boost.org'
Subject: RE: [asio] Sending large
packages
Some testing showed that the maximum package (load) size of boost::asio::write/read() for a TCP socket seems to be 1007 kB
From: Nicola
Michael Gutberlet [mailto:nicola.g...@hhi.fraunhofer.de]
Sent: 15 December 2009 17:11
To: 'boost...@lists.boost.org'
Subject: [asio] Sending large
packages
Dear all,
Is there a well-defined way of sending large packages over TCP? Or do I have to split the packages on my own? For explanation: I want to send images (size around 6MB).
Best regards,
Nicola
Igor R
>
>
>
> Best regards and a happy new year ;-)
>
> ________________________________
>
> From: Nicola Michael Gutberlet [mailto:nicola.g...@hhi.fraunhofer.de]
> Sent: 15 December 2009 17:28
> To: 'boost...@lists.boost.org'
> Subject: RE: [asio] Sending large packages
>
>
>
> Some testing showed that the maximum package (load) size of
> boost::asio::write/read() for a TCP socket seems to be 1007 kB
I guess it's not an asio limitation, but the OS one - you just
exceeded the maximum message size for the underlying protocol.
On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.
_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users
Jason Felice
>> Doesn’t anyone have any ideas? Did anyone receive this mail?
>
> exceeded the maximum message size for the underlying protocol.
> On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.
I've several times encountered a faulty assumption about how TCP
works, actually in multiple code bases I've maintained. I'm not sure
if this might be the same faulty assumption, but it sounds like it
might be.
TCP networking is a stream-based protocol and it makes NO guarantees
that the receiving end's read sizes are coincident with the sending
send's write sizes. In fact, it has a few mechanisms which will
actively thwart this, including Nagle's algorithm (see
http://en.wikipedia.org/wiki/Nagle%27s_algorithm ). All TCP networks
have an MTU, and packets larger than this will be sliced up for
delivery (without affecting the data contained within or the
guaranteed order of delivery). There is some handshaking to discover
the minimum MTU along a route, though this is intended to allow client
systems to adapt for maximum throughput, it is not a guarantee that
slicing won't happen. It's more like a guarantee that it won't happen
frequently. Additionally, loaded routers may well choose to split
transmit units or collect and concatenate them in order to meet
quality of service and throughput requirements.
Unfortunately, though, making this assumption will apparently work
often enough with small packet sizes to lull some developers into
believing they've confirmed that read and write sizes are coincident.
IF you are making this assumption, you need to redesign your wire
protocol to include within it delineation of the data, such as sending
the data size up front, or using some sort of demarcation or
end-of-data marker. You then need to collect the reads on the
receiving side until you've reached the demarcation or expected data
Oliver Seiler
> On Mon, Jan 4, 2010 at 5:41 AM, Igor R <boost...@gmail.com> wrote:
>>> Doesn’t anyone have any ideas? Did anyone receive this mail?
>>
>> I guess it's not an asio limitation, but the OS one - you just
>> exceeded the maximum message size for the underlying protocol.
>> On windows you can use SO_MAX_MSG_SIZE socket option to get the maximum size.
>
> have an MTU, and packets larger than this will be sliced up for
> delivery (without affecting the data contained within or the
> guaranteed order of delivery).
And herein lies what the original poster is hitting. The MTU is way
smaller than the ~1MB limit the poster sees, so that isn't likely the
problem (and would typically only be an issue if using UDP).
My guess is the TCP read/write buffers have been exceeded. Original
poster should include more information:
- OS that this is on,
- The actual error codes being encountered.
If you're exceeding the TCP buffer sizes and trying to write in one
chunk, you may be hitting an error like ENOBUFS (no_buffer_space in
boost:error_code). Just a guess though.
Nicola Michael Gutberlet
Thank you for your replies. I wanted to tell you the error message thrown
when trying to send larger packages but it seems to be solved. I don't know
why or how...
Anyway, it is sensible to check the maximum message size for the
socket/OS/whatever. How do I get the information of the maximum size of
messages I can write/read over my TCP socket? I noticed the option mentioned
by Jason Felice "SO_MAX_MSG_SIZE" but the TCP socket doesn't support it. I
tried to use an own template:
boost::asio::detail::socket_option::integer<SOL_SOCKET, SO_MAX_MSG_SIZE>
cOptionMaxMessageSize;
socket->get_option( cOptionMaxMessageSize );
But this also did not work - resulting in the option value -1.
Best regards,
Nicola
Matheus Araújo Aguiar
Dear all,
Thank you for your replies. I wanted to tell you the error message thrown
when trying to send larger packages but it seems to be solved. I don't know
why or how...
Anyway, it is sensible to check the maximum message size for the
socket/OS/whatever. How do I get the information of the maximum size of
messages I can write/read over my TCP socket? I noticed the option mentioned
by Jason Felice "SO_MAX_MSG_SIZE" but the TCP socket doesn't support it. I
tried to use an own template:
boost::asio::detail::socket_option::integer<SOL_SOCKET, SO_MAX_MSG_SIZE>
cOptionMaxMessageSize;
socket->get_option( cOptionMaxMessageSize );
But this also did not work - resulting in the option value -1.
Hi Nicola,
Note that in the code you posted, you wrote the option as "SOL_SOCKET, SO_MAX_MSG_SIZE". You have an extra "L" there which might be causing your value -1. Also, remember that as stated before, this option goes for microsoft windows. There is no maximum size you can send through a socket, although there is a MTU(Maximum Transfer Unity) and you should read more about it if you are not familiarized with that concept. As said before, TCP being a stream protocol, you have to ensure in a way or another (by putting the size in front of data, or using an "END OF STREAM" mark) that your read get all the bytes you expect.
Hope that helps!
--
Matheus Araújo Aguiar
Computer Scientist
mathe...@gmail.com
Matheus Araújo Aguiar
On Tue, Jan 5, 2010 at 8:27 AM, Nicola Michael Gutberlet <nicola.g...@hhi.fraunhofer.de> wrote:
Dear all,
Thank you for your replies. I wanted to tell you the error message thrown
when trying to send larger packages but it seems to be solved. I don't know
why or how...
Anyway, it is sensible to check the maximum message size for the
socket/OS/whatever. How do I get the information of the maximum size of
messages I can write/read over my TCP socket? I noticed the option mentioned
by Jason Felice "SO_MAX_MSG_SIZE" but the TCP socket doesn't support it. I
tried to use an own template:
boost::asio::detail::socket_option::integer<SOL_SOCKET, SO_MAX_MSG_SIZE>
cOptionMaxMessageSize;
socket->get_option( cOptionMaxMessageSize );
But this also did not work - resulting in the option value -1.
Hi Nicola,
Note that in the code you posted, you wrote the option as "SOL_SOCKET, SO_MAX_MSG_SIZE". You have an extra "L" there which might be causing your value -1. Also, remember that as stated
Sorry for that, the option is not written wrong, i confused things about that.
Tim Couper
class A
{
public
A(const std::vector<std::vector double >>& my_array);
};
and would intuitively write the wrapper:
BOOST_PYTHON_MODULE(foo)
{
using namespace boost::python
class_<A>("A")
.def(init(std::vector<std::vector<double> >())
;
but get the error "a call to a constructor cannot appear in a constant expression"
Can anyone supply any guidance as to how to wrap this? Thanks
Tim
Simon Pickles
>
> Can anyone supply any guidance as to how to wrap this? Thanks
>
of Python/C++ integration'
Si
> ------------------------------------------------------------------------
Tim Couper
Tim
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.725 / Virus Database: 270.14.127/2603 - Release Date: 01/06/10 07:35:00