Buffer Overflow

[Hsaka]

Hi Nate,

I got this error tonight during normal play. There were only 4 players
online at the time and they were in the same room.

java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Unknown Source)
at java.nio.DirectByteBuffer.put(Unknown Source)
at com.esotericsoftware.kryo.serialize.IntSerializer.put
(IntSerializer.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:228)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject
(Kryo.java:262)
at com.esotericsoftware.kryonet.TcpConnection.send
(TcpConnection.java:167)
at com.esotericsoftware.kryonet.Connection.sendTCP
(Connection.java:56)
at com.esotericsoftware.kryonet.Server.sendToAllTCP
(Server.java:389)
at server.AllFoursServer.sendConstantMessage
(AllFoursServer.java:133)
at server.AllFoursServer.access$4(AllFoursServer.java:123)
at server.AllFoursServer$3.run(AllFoursServer.java:108)

Any ideas?

[Hsaka]

sendConstantMessage() sends the following object every 200
milliseconds:

public class ConstantMessage()
{
public long time;
}

ConstantMessage is registered as follows:

kryo.register(ConstantMessage.class, new BlowfishCompressor(new
FieldSerializer(kryo), key));

[Nate]

Hi Hsaka,

The buffer that overflowed is the TCP write buffer for a connection.
This buffer holds serialized objects that are waiting to be written to
the connection's socket. Normally objects are written to this buffer,
then immediately written to the socket. Sometimes the socket isn't
writable, in which case the bytes stay in the buffer until the socket
becomes writable.

That is how it is *supposed* to work.

If for some reason, quite possibly a valid reason, the connection's
socket isn't writable for a while, the write buffer could overflow.
Did your server crash when this happened? I'll make sure that it
doesn't crash the server and that it just closes the connection, since
this situation can occur under valid (but poor) conditions. If this is
the case, then when you create the server, you can specific a larger
buffer size. The default is 2048, which in hindsight is probably a bit
small. I'll increase the default buffer size on the server.

However, it is possible something else went wrong. From your
exception, the write that failed was the first byte for an object. For
this is happen, the buffer would have to become *exactly* full after
the previous object was written. This is possible, but not likely.
Maybe there is some code path that causes the write buffer to not get
flipped. If the limit is being left at the position then the next
write would cause an overflow. I'll trace through the code and look
for anything suspicious. When I add handling to make sure a valid
buffer overflow doesn't crash the server, I'll also add some logging
for the current state of the buffer (position, limit, etc). If the
exception happens again, we'll be better prepared.

How large is a serialized ConstantMessage after blowfish encryption?
Should show in the logs if you enable the debug level. The encryption
seems a bit excessive, especially for something sent so often. I
assume the key for decryption is provided inside your game code
somewhere, so the blowfish is not really providing any security.

If you are sending a message very often, you could do some
optimization. Eg, you could remember the first time sent, then just
send deltas. The deltas could be int. If the delta is < 16384 then it
will only take 2 bytes to send, versus the long which will always take
8 bytes. When a new client connects, send a long to everyone, then you
can go back to sending deltas. Then again, this optimization may not
be worth the effort/complexity. :)

-Nate

[Hsaka]

Thanks for that explanation Nate.

The server did not crash, it kept running and throwing exceptions.
Chat messages could still be sent as they don't rely on the
ConstantMessage.

You are right about encrypting the ConstantMessage, I've since changed
that to serialize normally.

[Hsaka]

Nate, what is the downside to having a large buffer size for the
server?

[Nate]

Only that it uses more memory. It allocates two times (one for write,
one for read) the specified buffer size per connected client. If you
were also using UDP, it would allocate two more buffers for a total of
four.

-Nate

[Nate]

Some new changes are checked in.

Default buffer sizes have been increased. See no-arg constructor
javadocs on Server and Client.

Both Server and Client constructors have changed. To be more
efficient, I split the buffer size into separate write and read sizes.
Basically using TCP uses writeBufferSize+readBufferSize bytes and
using both TCP and UDP uses writeBufferSize+(readBufferSize*3) bytes.
readBufferSize can be much smaller than writeBufferSize. See the
appropriate constructor javadocs for details about how the parameters
are used and what values you should pass.

There is now better handling for when a TCP write buffer overflows due
to being unable to write to the socket. The exception is logged as a
warning and the connection is closed.

-Nate

[Hsaka]

Thanks Nate. The problem has not yet occurred since I increased the
buffer size.

[Hsaka]

Just got it again:

64:01 WARN: Write buffer overflow, position/limit/capacity:
16380/16384/16384
java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:
501)
at java.nio.DirectByteBuffer.putLong(DirectByteBuffer.java:
736)
at
com.esotericsoftware.kryo.serialize.LongSerializer.writeObjectData
(LongSerializer.java:24)
at com.esotericsoftware.kryo.Serializer.writeObject
(Serializer.java:38)
at
com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData
(FieldSerializer.java:141)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject
(Kryo.java:263)
at com.esotericsoftware.kryonet.TcpConnection.send
(TcpConnection.java:168)
at com.esotericsoftware.kryonet.Connection.sendTCP
(Connection.java:58)
at com.esotericsoftware.kryonet.Server.sendToAllTCP
(Server.java:403)
at server.AllFoursServer.sendConstantMessage
(AllFoursServer.java:130)
at server.AllFoursServer.access$3(AllFoursServer.java:
120)
at server.AllFoursServer$3.run(AllFoursServer.java:
105)

The server closed right after. I'm using the no-arg constructor for
the server.

[Nate]

Because the buffer limit == capacity, we can be sure that the buffer
is legitimately full. For whatever reason, the socket for a client
could not be written to. Objects that needed to be sent to that client
got queued up until the buffer was full.

I'm not sure why the TCP connection is coming back unwritable.
Probably the client's socket went bad for some reason and stopped
reading data. On the server, once whatever buffer the OS has for the
socket is full, the socket stops being writable and we do buffering in
KryoNet until we run out of buffer space. Normally when something goes
wrong TCP should tell us the connection has been closed. I'm not sure
why the client socket goes bad.

Once the KryoNet buffer overflows, the connection with the client
should be closed. You said the server closed right after? Did it
crash? Do you have a stacktrace? The server should have just closed
the connection and carried on without any problems, as if the socket
was terminated on the client's side.

-Nate

[Hsaka]

Sorry Nate, I assumed that the server had closed down after I saw that
exception in the server console, and I restarted it right away.

So only a specific client should be affected? I'll try to verify this
if/when it happens again. I assume that increasing the buffer size
further won't help me much..

[Nate]

Yes, I believe it happens because a client's connection has gone sour.
The server should be fine otherwise. It would be great to know what
puts the client socket in a bad state.

I log all normal connection problems, like when a client forcibly
terminates a connection (ie, force quits the client app process), at
the debug level because the reason the connection is gone isn't
terribly useful. I log the buffer overflow as a warning because it
might be an indication that the buffer size needs to be increased. I'm
thinking I should log this at debug though, to match the rest. If
someone is rapidly sending large amounts of data, they may notice the
clients getting closed, in which case they'll turn on debug and see
the reason is buffer overflow.

-Nate

[Hsaka]

Ok, it happened again and I verified that only one client is affected
and the server continues running normally. I'll build the next version
with logging at a lower level to see what's going on.

[Hsaka]

Nate, I think I figured out why the overflow happens. For some reason,
the server does not close the connection to a client who disconnects
due to a loss of internet connectivity (the server closes the
connection properly if the client closes the app/tab/browser).

Because of this, the server keeps sending data to the socket for the
disconnected client until the buffer eventually overflows.

[xQuasar]

I've been getting a BufferIOverflowException too, except it's Client-
side.

Nate hasn't been replying to anything at all lately and he hasn't been
on IRC in AGES, I wonder if anything's up? :/

[Hsaka]

I hope he's ok :x

[Nate]

On Fri, Oct 16, 2009 at 3:03 AM, xQuasar <xag...@hotmail.com> wrote:
> I've been getting a BufferIOverflowException too, except it's Client-
> side.

Can you post the exceptions?

> Nate hasn't been replying to anything at all lately and he hasn't been
> on IRC in AGES, I wonder if anything's up? :/

Nothing's up. :) I answered your last email within 14 hours. We may be
on different sides of the Earth.

-Nate

[Nate]

Interesting! Takes two computers to test. I'll try this out soon.

-Nate

[Bharathi]

Hi,

I'm using the kryonet libraries to write a simple RMI application.

The prototype of the method in the remote interface is
String RemoteMethod(int len);

The return value is a String, whose length is equal to len (the
argument to the method).

If len is more than 2048 bytes, the String object returned is more
than 2048 bytes; this makes the server crash with a
BufferOverflowException as given below:
-----------------
Exception in thread "Server" java.nio.BufferOverflowException
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:311)
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:290)
at com.esotericsoftware.kryo.serialize.StringSerializer.put
(StringSerializer.java:50)
at
com.esotericsoftware.kryo.serialize.StringSerializer.writeObjectData
(StringSerializer.java:39)
at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData
(FieldSerializer.java:136)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:263)
at com.esotericsoftware.kryonet.TcpConnection.send(TcpConnection.java:
167)
at com.esotericsoftware.kryonet.Connection.sendTCP(Connection.java:
56)
at com.esotericsoftware.kryonet.rmi.ObjectSpace.invoke
(ObjectSpace.java:182)
at com.esotericsoftware.kryonet.rmi.ObjectSpace$1.received
(ObjectSpace.java:59)
at com.esotericsoftware.kryonet.Connection.notifyReceived
(Connection.java:246)
at com.esotericsoftware.kryonet.Server.update(Server.java:174)
at com.esotericsoftware.kryonet.Server.run(Server.java:305)
at java.lang.Thread.run(Thread.java:595)
----------------
I understand looking through the code that the write buffer has a
default size of 2048 bytes only and I'm trying to write more than that
and hence the exception.

Does this mean return objects cannot be more than 2048 bytes ? How
does one handle such cases ?

I was able to make it work by initializing the Server() with more than
2048 bytes, but then it is not possible for me to predict the maximum
size of my return value.

I would appreciate any suggestions here.

Thanks.

[Nate]

On Nov 30, 3:59 pm, Bharathi <bharathi.sesha...@gmail.com> wrote:
> I understand looking through the code that the write buffer has a
> default size of 2048 bytes only and I'm trying to write more than that
> and hence the exception.
>
> Does this mean return objects cannot be more than 2048 bytes ? How
> does one handle such cases ?
>
> I was able to make it work by initializing the Server() with more than
> 2048 bytes, but then it is not possible for me to predict the maximum
> size of my return value.

Data for an object is buffered as it comes in. When all the data is
received, then the object can be created. Increasing the buffer size
is the correct answer. At some point, there is a limit to the size of
your return value. Eg, the object has to fit in the JVM's memory,
which has limits that you can configure. Like the JVM's max heap size,
set your buffer size to something you should not normally exceed.

If you really have the need to pull in huge amounts of data, it
probably won't be a string or an object, probably you'd be sending
large files. You could break the data into chunks and send it
piecemeal.

-Nate