Any incoming messages limits?

[13Homer]

I'm trying send (and receive) very long messages (LinkedList with
1.000.000 elements). Server builds message and sends it without
problem, but client throws exception:

Exception in thread "Client" com.esotericsoftware.kryo.KryoException:
java.lang.ArrayIndexOutOfBoundsException: 512
Serialization trace:
ts (IQFeedKryo.messages.IQTickHistory)
ticks (IQFeedKryo.messages.IQTickHistoryResponse)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:
241)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:714)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:
97)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:
18)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:628)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:
224)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:714)
at
com.esotericsoftware.kryonet.KryoSerialization.read(KryoSerialization.java:
57)
at
com.esotericsoftware.kryonet.TcpConnection.readObject(TcpConnection.java:
138)
at com.esotericsoftware.kryonet.Client.update(Client.java:239)
at com.esotericsoftware.kryonet.Client.run(Client.java:317)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 512
at com.esotericsoftware.kryo.io.Input.readLong_slow(Input.java:544)
at com.esotericsoftware.kryo.io.Input.readLong(Input.java:514)
at com.esotericsoftware.kryo.serializers.DefaultSerializers
$DateSerializer.create(DefaultSerializers.java:250)
at com.esotericsoftware.kryo.serializers.DefaultSerializers
$DateSerializer.create(DefaultSerializers.java:244)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:626)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:
224)
... 11 more

So I have split this big message to 40 "submessages" (25.000 elements
each). 4 of them are handled correctly, but 5th (always fifth!) failed
with exactly the same exception.
Receive and send buffers are set for 10M bytes, kryonet is 2.08 (one-
jar), all classes are registered manually in kryo.

When I tried track down how many elements can be sent without error
the transfer start failing when list had about 37.500 elements.
I'm sending rather complicated objects so no code this time, but I can
prepare simpler example if it would be helpful (tomorrow, I'm very
tired right now).

[Nate]

It explodes in readLong_slow. I took a close look at that method again, and it looks ok except it should be require(count+1) and not require(count). This won't fix your problem, it will only tell you "buffer underflow" instead of ArrayIndexOutOfBounds. Otherwise the method checks out and should be working properly for all input.

KryoNet message bytes are a length, then class id, then serialized bytes. It doesn't attempt deserialization until all the bytes are ready. The only limit is the size of the buffers.

Note if you send a LOT of data very rapidly, KryoNet will buffer it up and you won't be able to send other data until that goes through. You could break your data into smaller pieces and use TcpIdleSender to send a piece only when the connection is idle, so you can still send other data. This also has the benefit of not needing a large write buffer.

Your problem is probably that bytes got mangled somehow, classes are registered with Kryo differently on the client/server, or an error in deserialization not matching up to serialization. Debugging it is a pain, sorry.

You could edit the server so that after it serializes an object, it deserializes those bytes before sending them. If deserialization fails there, we've narrowed it down to a bug in Kryo serialization. If so, write out the bytes to a file to make a simpler test case.

You could capture the actual bytes on your client. For that, breakpoint on ArrayIndexOutOfBounds on the client then cause the failure. Go up the stacktrace to TCPConnection#readObject, where it calls KryoSerialization#read. Write the bytes from readBuffer to a file, starting at startPosition and ending at startPosition + length. Easy way to do this is with the debugger stopped and the TCPConnection#readObject stack frame selected, paste the following code anywhere, select the code, right click and choose Execute.

java.io.FileOutputStream output = new java.io.FileOutputStream("fubar");
for (int i = startPosition; i < startPosition + length; i++)
   output.write(readBuffer.get(i));

Now you should be able to write a simpler test using just Kryo to deserialize the bytes from that file. Code to do that looks like:

Kryo kryo = new Kryo();
// Configure Kryo here.
Object object = kryo.readClassAndObject(new Input(new FileInputStream("fubar")));
System.out.println(object);

Most likely it will fail. Send me the file and the classes and Java code you use to configure Kryo and I'll look at fixing it, or you can debug it from there. Coming up with a simpler test case may be difficult.

-Nate

--
You received this message because you are subscribed to the "kryonet-users" group.
http://groups.google.com/group/kryonet-users

[13Homer]

I started debug it with much simpler app. Here it is:


import java.math.BigDecimal;
import java.util.Date;
import java.util.LinkedList;

import com.esotericsoftware.kryo.Kryo;
import
com.esotericsoftware.kryo.serializers.DefaultSerializers.BigDecimalSerializer;
import com.esotericsoftware.kryonet.Client;
import com.esotericsoftware.kryonet.Connection;
import com.esotericsoftware.kryonet.Listener;
import com.esotericsoftware.kryonet.Server;

public class KryonetTest {

public static void main(String[] args) {
final Object lock = new Object();
final int bufSize = 10000000;
final Client c = new Client(bufSize, bufSize);
final Server s = new Server(bufSize, bufSize);
registerMessages(c.getKryo());
registerMessages(s.getKryo());
try {
s.start();
c.start();
s.addListener(new Listener() {
public void connected(Connection c) {
IQTickHistoryResponse r = new IQTickHistoryResponse("@EUM12");
Date ts = new Date();
for (int i = 0; i < 100000; ++i) {
// FAILED
//r.ticks.add(new IQTickHistory(new Date(), new
BigDecimal("1.2345"), 7, new BigDecimal("2.3456"), new
BigDecimal("9.8765")));
// FAILED
//r.ticks.add(new IQTickHistory(ts, new BigDecimal("12345"), 7,
new BigDecimal("23456"), new BigDecimal("98765")));
// OK
r.ticks.add(new IQTickHistory(ts, new BigDecimal("12345"), 7,
new BigDecimal("23456"), new BigDecimal("9876")));
// OK
//r.ticks.add(new IQTickHistory(ts, BigDecimal.TEN, 7,
BigDecimal.ONE, BigDecimal.ZERO));
}
c.sendTCP(r);
}
});
c.addListener(new Listener() {
public void received(Connection c, Object o) {
IQTickHistoryResponse r = (IQTickHistoryResponse)o;
System.out.println("r="+r.ticks.size());
synchronized (lock) {
lock.notify();
}
}
});
s.bind(60606);
c.connect(60606, "localhost", 60606);
synchronized (lock) {
lock.wait();
}
} catch (Throwable ex) {
ex.printStackTrace();
} finally {
c.stop();
s.stop();
c.close();
s.close();
}
}

public static void registerMessages(Kryo kryo) {
kryo.register(BigDecimal.class, new BigDecimalSerializer());
//kryo.register(IQWatchRequest.class);
//kryo.register(IQWatchResponse.class);
//kryo.register(IQTickHistoryRequest.class);
//kryo.register(IQTickHistoryDayRequest.class);
//kryo.register(IQUnwatchRequest.class);
kryo.register(Date.class);
//kryo.register(Quote.class);
//kryo.register(HashSet.class);
kryo.register(LinkedList.class);
//kryo.register(IQQuotesRequest.class);
//kryo.register(IQQuotesResponse.class);
kryo.register(IQTickHistory.class);
kryo.register(IQTickHistoryResponse.class);
/*
kryo.register(LocalDate.class, new SimpleSerializer<LocalDate>() {
@Override
public LocalDate read(ByteBuffer b) {
return LocalDate.fromDateFields(new Date(b.getLong()));
}
@Override
public void write(ByteBuffer b, LocalDate d) {
b.putLong(d.toDate().getTime());
}
});
*/
}

static public class IQTickHistoryResponse {

public final String symbol;
public final LinkedList<IQTickHistory> ticks = new
LinkedList<IQTickHistory>();

public boolean last;

public IQTickHistoryResponse(String symbol) {
this.symbol = symbol;
last = false;
}

public IQTickHistoryResponse() { this(null); }
}

static public class IQTickHistory {

public final Date ts;
public final BigDecimal last;
public final BigDecimal bid;
public final BigDecimal ask;
public final int lastSize;

public IQTickHistory(Date ts, BigDecimal last, int lastSize,
BigDecimal bid, BigDecimal ask) {
this.ts = ts;
this.last = last;
this.lastSize = lastSize;
this.bid = bid;
this.ask = ask;
}

public IQTickHistory() { this(null, null, -1, null, null); }
}
}

The strange thing is in the lines ~30 (4 versions of adding elements
to the list). Look carefully at the second FAILED and first OK. Seems
like some BigDecimal values causes crash.
I have no idea if this behaviour is my computer specific (system?
java?), but it fails/pass every time I run app, so I think it is not
accidental.
I've tried serialize BigDecimal as is in the code:

kryo.register(BigDecimal.class, new BigDecimalSerializer());

and in the default way:

kryo.register(BigDecimal.class);//, new BigDecimalSerializer());

Result is exactly the same.
I'm still using official 2.08 (not build from repo).

Now I'm going to try deserialize message on the server just before
send.

[13Homer]

I've tried to deserialize message:

//ByteArrayOutputStream out = new ByteArrayOutputStream();
FileOutputStream out = new FileOutputStream("x.bin");
s.getKryo().writeClassAndObject(new Output(out), r);
//System.out.println(out.size());
//ByteArrayInputStream in = new
ByteArrayInputStream(out.toByteArray());
FileInputStream in = new FileInputStream("x.bin");
Object rr = s.getKryo().readClassAndObject(new Input(in));
System.out.println(rr.getClass());

and I've got "Buffer underflow" exception (using ByteArrays makes no
difference). So I've downloaded sources from svn (found 2 places with
require(count + 1); so I think it is fixed), compiled and replaced
kryo part in all-jar than rebuild test app and run. No changes - still
exeception.

[Nate]

Thanks for the test. I did a bad thing when updating KryoNet to v2, a buffer was flipped when it shouldn't, causing the first write to succeed but subsequent writes to fail. Fixed in SVN and I did a new release of Kryo and KryoNet, both to 2.09. Sorry for the trouble it caused you!

-Nate

[13Homer]

Thank you!
Now app is sending over 1m elements in 25k chunks without any problem.
Although serialization/deserialization using file still fails, but
this is not the way I'll using it, so I can live with that ;)
Thanks again.

[Nate]

Sweet! :)

Oh, sorry I forgot to respond to your second email about deserializing from a file failing. You need to flush the Output before reading the file. Call Output#close() to flush and close the file.

-Nate

[13Homer]

Yeah, now it rocks!
Thanks.