I would be very curious to know what Boost Serialization does in this
situation, if anyone knows.

-Le Chaud Lapin-

The problem is when the link is insecure.  It ruins the entire
serialization framework.  Note that ruin happens not just for strings,
but for any situation where there is a vector of elements, and the
source of an object is about to convey to the target that size of that
vector before serializing the individual elements of that vector.

Note again that this is a framework here, not a specific application,
so I cannot, for example, in the context of each serializable class,
specific an arbitrary limit on the number of elements involved,
because it would be, well...arbitrary. This is true especially if the
class contains a vector template, as it would not be known the size of
each element in the vector, so even if some arbitrary limit were set
for the size of the array, say 65,536, if each element of the array is
an object with multiple members, it is conceivable that one of those
members would be an array itself.  This problem presents itself
recursively, so that, if N is limit on number of elements allowed to
be serialized for vector V,  then recursively L levels, there would be
an exponential explosion in memory space required for new[] against a
Foo vector[N], equal to N^L, so that even for L=4 and N=65,536, N^L is
2^64, and we're back where we started.

But aside from the details, it should be intuitively apparent that
trying to put these artificial limits ruins the regularity of the
entire model, which again, is a framework and not a specific
application.  As we all know, arbitrariness is a red-flag in good
design principles.

Consider defining the serialization function for a List<>:

Socket s;

List<Foo> l;

s << l;

One would not be able to specify a limit on the count of s without
knowing how much space Foo will take up.  Big Foo, small limit.  Small
Foo, large limit. Foo itself could contain members that contain
List<>, and so on, recursively.

The more I think about this problem, the more I am beginning to
believe that it is better to leave the classes themselves alone and
focus on the memory management itself.  At least the regularity would
be preserved.

In that case, there are two possible "solutions", one that will not
work, the other that might:

The solution of putting a limit on the "archive" object (Socket in
this case) won't work because that will be meaningless for a long-
duration application that was meant to acquire and release terabytes
of memory throughout its natural life.

That leaves memory allocation against the thread itself.  At any given
instant, on a server machine with 4GB of ram and 500 client connects,
if one server thread is hogging 2.5GB for itself, there is probably a
breach in progress.  In that case, the memory allocation should fail
with an exception, the server thread will hard-abort, the evil client
connection will be broken, and the only entity unhappy at that point
will be the evil client.

Unfortunately, memory allocation quotas on on most OS's, if I am not
mistaken, are applied on a per-process, not per-thread, basis.

-Le Chaud Lapin-

--
  Nevin ":-)" Liber  <mailto:ne...@eviloverlord.com>  773 961-1620

These lines are your problem:

Socket s(...);
s << l;

You're conflating serialization with transmission, and on the other
end, deserialization with reception. You need to serialize your data:

std::vector<char> data;
serialize(l, data);

, and then send it

s.send(data);

And on the other end, receive the data:

std::vector<char> data;
s.receive(data);

, and then deserialize it:

deserialize(l, data);

That gives you control over how much data moves in and out of your
application.

In the Socket::receive() function you put a limit on the number of
bytes you are willing to read off the network, and if you're still
suspicious, you allocate memory in smaller chunks and realloc as the
data comes in.

Jarl.

I am surprised at some of the responses here to be honest.  It is
intuitively obvious to me that "validation" is not possible, and the
grab-and-realloc method facings the same issues - when is too much
memory too much?  And given that class objects can be nested, per-
object limitations on allocated memory are completely arbitrary.  This
should be evident from a container of objects.

map<int, map<int, map<int, map<int, Foo> > > > valid_object;

Socket s;

s << valid_object;

This is a reasonable piece of code in terms of the model it implies.
One can imagine how serialization for map<> might be implemented.  If
Foo contains a string that is say, 800 bytes long, that is a
reasonable value for some strings.  If the count of elements in a
map<> is 3500, that is a reasonable value for some map<>'s. But I can
take this structure and easily make its overall memory consumption on
the order of Gigabytes.

Then what?  Should the internal map<>'s be artificially-intelligent
and say, "Uh oh...I detected that I am inside of something big and bad
going on..."  Obviously they cannot.  This will blow up at the server.

Furthermore, there is another problem that is insurmountable, I
think.  It is highly reasonable that one legitimate client might
induce the server to allocate, say, 20 megabytes, on behalf of the
client.  That 20 megabytes does not have to be allocated in one array-
chunk.  It could be distributed over, say, the last 2000 objects
created by the server on behalf of the client.

I am going to repeat myself here because I can feel that, when I write
the last sentence, there are some reading this who are thinking, "Just
put limits on what's done."

I cannot do that. :)  This is a serialization framework.  I must be
able to provide a means of serializing an object, then hands off.  If
I parameterize the allocation size, the size becomes arbitrary.  And
given the nested map<>'s up above, it should be evident that, if I
make the per-array size too small, I deny legitimate clients.  If I
make it too large, the malicious client can successfully attack.  If I
choose something "reasonable", a malicious client can still attack.

An to reiterate, I have a secure-mode of operation where this issue is
not a problem.

The problem is when the link is insecure.  And there are cases where
it is a legitimate necessity that the link be insecure.

I am beginning to think that a poor "solution" might be for the kernel
of the OS to allow per-thread quotas on allocated pages.

For those who keep saying, "Put a limit..", I encourage you to write C+
+ code to show how you would serialize a List<> template, and tell me
what limits you would use.

-Le Chaud Lapin-

Greetings, Branimir.

This leads us to a simple conclusion, was somewhat sure of when I
wrote the OP, but now I am certain of: one cannot have his cake and
eat it.  Generalized serialization frameworks, the kind that many C++
programmers write, fail in the face of insecure IPC channels.

Being a researcher in computer networking, this is very troubling to
me.  It means that the most wonder of feature of serialization,
obviation of microscopic attention to marshalling of data across the
channel, fails completely.  On an insecure channel, every single
element just be range-checked, etc.

This means that if one wants to avoid DoS attacks, either through over
memory allocation or simple causing the server to choke on bad data,
one really should not use serialization at all over an insecure
channel.

-Le Chaud Lapin-

If this is not clear, think about it some more. :)

Best,

-Le Chaud Lapin-

It it is evident to me that, with no authentication, you cannot have
your cake and eat it.  What you wrote above is inevitable.

What this means is that, any serialization framework, not just mine,
that claims that, "you can use it against sockets just as well as
files", is actually being somewhat dishonest.  Again, I am curious to
know how Boost handles serialization of strings.  What happens if I
want to serialize a 10,000-character string over a socket using
Boost's archive method.

Why is this important?

I means that, for all the applications on the Internet that uses
unprotected serialization of the kind provided by Boost,/etc...they
are all vulnerable to DoS attack.

All one has to do is super-saturate the server with bogus resource
consumption (memory allocation), and linger.

The most important observation, which I keep repeating, is that it
should also be evident that anything beyond a secure (authenticated)
connection won't work.  It will result in quick and massive
degradation of the framework itself.   For example, someone might
propose that the IP address of the server be checked, and if it makes
too many connections within a specified period, limit its memory
allocation.  Or whatever.

It should be obvious that:

1.  You are back to the original problem, which is "How much is too
much?"
2.  There are legitimate cases to multiple connections.

One cannot have his cake and eat it without authentication.

If I were an evil person, I'd go hunting around the Internet finding
servers that use serialization against general-public links and do
naughty things to them. ;)

-Le Chaud Lapin

Greetings, Branimir.

Brian Wood
Ebenezer Enterprises

For the mods: just like the rest of this thread, this post concerns
the application of generalized C++ serialization frameworks, like the
one in Boost, to IPC applications.

IPC systems commonly use a concept of a message, with a header and a
payload. Among other things, the header would contain the length of
the payload. When the server receives a message from a client, it
reads the header, and checks the payload length against a preset
limit. After that, it proceeds with deserialization of the payload,
and because it already knows the length of the payload, eg 4196 bytes,
it knows that it should not accept eg a single byte string claiming a
length of 5000, or a double byte string claiming a length of 2500,
etc.

Now if your serialization code blindly allocates buffers of arbitrary
size, then you obviously have a problem in your serialization code.
You need to improve it to be aware of the payload length of the
current message being processed. I'm curious as to why you think that
is a big deal?

Jarl.