org.vertx.java.core.json.DecodeException: Failed to decode:Unexpected character (Clustering)

[Oje Preradovic]

Hi.

Everything is working great is an all in one Vertx app. But when using clustering getting the above exception in the event bus (no flow through errors to handlers) when using Hazelcast clustering. Since some of the data being passed by JSON is LARGE I am assuming that data is getting truncated and causing this. Since then same error happens in HTTP if you don't set Receive/Send buffer size big enough I though that setting them in Hazelcast like:

System.setProperty("hazelcast.socket.receive.buffer.size", "128000");

        System.setProperty("hazelcast.socket.send.buffer.size", "128000");

Would fix things it does not. Error looks like:

SEVERE: Unhandled exception

org.vertx.java.core.json.DecodeException: Failed to decode:Unexpected character ('c' (code 99)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

 at [Source: java.io.StringReader@4d7f405e; line: 1, column: 18]

at org.vertx.java.core.json.impl.Json.decodeValue(Json.java:63)

at org.vertx.java.core.json.JsonObject.<init>(JsonObject.java:67)

at org.vertx.java.core.eventbus.impl.JsonObjectMessage.readBody(JsonObjectMessage.java:55)

at org.vertx.java.core.eventbus.impl.BaseMessage.<init>(BaseMessage.java:306)

at org.vertx.java.core.eventbus.impl.JsonObjectMessage.<init>(JsonObjectMessage.java:43)

at org.vertx.java.core.eventbus.impl.MessageFactory.read(MessageFactory.java:70)

at org.vertx.java.core.eventbus.impl.DefaultEventBus$2$1.handle(DefaultEventBus.java:587)

at org.vertx.java.core.eventbus.impl.DefaultEventBus$2$1.handle(DefaultEventBus.java:580)

at org.vertx.java.core.parsetools.RecordParser.parseFixed(RecordParser.java:200)

at org.vertx.java.core.parsetools.RecordParser.handleParsing(RecordParser.java:158)

at org.vertx.java.core.parsetools.RecordParser.handle(RecordParser.java:214)

at org.vertx.java.core.parsetools.RecordParser.handle(RecordParser.java:50)

at org.vertx.java.core.net.impl.DefaultNetSocket.handleDataReceived(DefaultNetSocket.java:267)

at org.vertx.java.core.net.impl.VertxNetHandler.channelRead(VertxNetHandler.java:48)

at org.vertx.java.core.net.impl.VertxNetHandler.channelRead(VertxNetHandler.java:32)

at org.vertx.java.core.net.impl.VertxHandler.channelRead(VertxHandler.java:156)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:332)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:318)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:125)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

at java.lang.Thread.run(Thread.java:745)

Any ideas?? And yes the Json objects are LARGE (thank you hashed values).

[Tim Fox]

We don't use hazelcast for cluster traffic, so it's not related to hazelcast.

Do you have a reproducer?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oje Preradovic]

Well that would explain why mucking around with hazelcast does nothing. I will create a test case to show..

Tim...

How does one event loop send messages to another one?? Is Hazelcast then merely used as a CORBA like Naming/Event service? And then nodes send event traffic directly to one another??

Again we had the same error until we increased the HTTP Receive buffer size, so I am assuming that whatever transport is sendind event data around needs the same setting tweeked.

Cheers,

Oje

[Oje Preradovic]

Tim,

We created a test app which of course works perfectly!

Basically:

a. Incoming HTTP Vertical takes in POST Buffer converts to JSON Data;

b. A worker Verticle in same process (therefore same event bus??) takes in message posts message to another verticle in another process (works)

c. Verticle in another process creates a reply and sends back (works)

d. Verticle from b receives response and extracts data (works)

e. At this point that Verticle creates a response to Verticle in A and sends it. This WORKS when everything is in one process, but we get the following error when the Verticle c is another process:

org.vertx.java.core.json.DecodeException: Failed to decode:Unexpected character ('c' (code 99)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

 at [Source: java.io.StringReader@6a56e21d; line: 1, column: 18]

at org.vertx.java.core.json.impl.Json.decodeValue(Json.java:63)

Which is strange because it is between 2 Verticles in the same process! We tried to create a small test app sending LOTS of data in JsonObject, but it works!!

Any ideas?? We are stumped!

[Oje Preradovic]

I wonder if it has something to do with the value of the string (hash)

WjNWcFpEMWlZV3hzZVdaaVh6RTBOVFpqTVRKbVpEQmpYMlZoWkdFd05qVmlYemxtWWpRM1ptUTBKQ1JtYjNKdFlYUTlhbk52YmlRa1pYaHdhWEpsY3owME5UQXdNREF3TURBd01EQXdKQ1J0WVhKclpYSTlNVEkyTkRnME16QT0uLi5NMlptTWpNME4yTXlOVE5rWkdVeVpXUm1NVFUwTlRKbVpXRTVOR1ZpT1dJelpqSmxaak5qTlRFNE1XTTNaRGcxWkRrelpUa3pORFkwTm1NeU9EWmtPUT09

Changing the above to byte[] works!

So..

String hash = WjNWcFpEMWlZV3hzZVdaaVh6RTBOVFpqTVRKbVpEQmpYMlZoWkdFd05qVmlYemxtWWpRM1ptUTBKQ1JtYjNKdFlYUTlhbk52YmlRa1pYaHdhWEpsY3owME5UQXdNREF3TURBd01EQXdKQ1J0WVhKclpYSTlNVEkyTkRnME16QT0uLi5NMlptTWpNME4yTXlOVE5rWkdVeVpXUm1NVFUwTlRKbVpXRTVOR1ZpT1dJelpqSmxaak5qTlRFNE1XTTNaRGcxWkRrelpUa3pORFkwTm1NeU9EWmtPUT09;

JsonObject resp = new JsonObject();

// this fails with decode exception

resp.putString("user_hash", hash);

jsonObjectRxMessage.reply(resp); // -> final RxMessage<JsonObject> jsonObjectRxMessage

// this works!

resp.putBinary("user_hash", hash.getBytes());

jsonObjectRxMessage.reply(resp); // -> final RxMessage<JsonObject> jsonObjectRxMessage

When we change the value to something random or null the message goes through

[Oje Preradovic]

My mistake, making everything binary still gets same exception!

[Tim Fox]

Really hard to tell without seeing a reproducer

[Jez P]

This should be relatively simple to create a reproducer for, since you should be able to take out most of the processing logic from your verticles (so essentially it's just the messaging between them), this should give you 3 small (in code terms) verticles, then just share those with instructions to start them up and reproduce. 

[Oje Preradovic]

Yeah I think this is the way to go...

I thought there was something obvious, but I will try to reproduce!

[Jez P]

Just one thought. When you're running clustered, are the separate JVMs on the same box or on different boxes? I'm wondering if on different boxes, whether decoding of byte streams might be being done using different codepages to those with which they're encoded (this would be more likely if going between different OSs I suspect). Presumably in clustered mode, the Strings get transmitted as bytestreams. If on the same box this would be unlikely, but it's conceivable that that could cause corruption. I know your failing transfer is happening between two verticles in the same JVM, but is it possible that the data extracted is corrupt?

All speculation, but until you have a reproducer, hard to do much other than speculate. 

[Oje Preradovic]

Hi!

So we figured it out. It turns out the culprit is a quick and dirty implementation of java.lang.Number that we had baked so we could easily shove numbers into a JsonObject since there is no putDouble, putInteger etc. In non cluster mode I am sure it is all in memory and no serialization happens. In cluster mode, setting a break point in the  public static <T> T decodeValue method yielded this as the string to be deserialized into our own number class:

{"user_balance":com.genesis.m4.vertx.utils.M4Number@36903ce8}

Which of course would blow up the deserializer! and account for the message: Failed to decode:Unexpected character ('c' (code 99)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

Now.. Is there an EASY way to pass Numbers around in vertx inside those  Json Objects??

Can't seem to find it in the documentation...

[Oje Preradovic]

Hahahah

Boy do we feel Dumb :)

JsonObject.putNumber("val", 45);

So why did the in memory stuff work and throw an exception??

No worries all working great now of course!

[Nick Scavelli]

If you're not going across the cluster, then there is no need to serialize the message (hence no call to encode the json).

[Oje Preradovic]

Makes sense!

Thanks for you all help Nick, are clusters are all running smoothly and we will be going to production on EC2 in a few weeks!

Vert.X is just amazing! Once we found the guice mod it was relatively easy to switch from an Jetty 9 /Asynch Servlet / RabbitMq infrastructure to Vert.x

Old system:

a. Main App (Jetty) over 700 threads

b. RabbitMq

c. Stand alone Queue Processors

d. Stand alone Containers accessed over RPC via RabbitMQ

On an med EC2 instance used about 80% RAM at low loads. On a SOAP UI load tests we could handle about 50 000 concurrent users SLAMMING the system hard, decent performance.. Keep scaling up the number of users and eventually started getting connection refused errors. CPU = 100% (mostly beam). After load removed Memory SLOWLY going down..

Vert.X:

a. Several Verticles in a cluster;

b. No more than 70 threads per process

c. Memory at doing nothing about 45% (lots of allocated RAM)

Load test:

a Under same load above, CPU around 65%, yes memory climbs, but after load removed nicely drops back down. Performance is equivalent to above system BUT we kept adding more and more connections and NEVER got a connection refused! So we have orders of magnitude greater scalability (again can't generate enough users to get a connection refused) and way less stress on the system!

The great thing is adding more and more nodes to the cluster is very easy. Haven't even played with the HA stuff...

Thank you Tim and all! This is the best thing to happen to Java in ages, and I DESPISE Node.js for many reasons and am sick of heating about it. You want to write a whole server in JS, then use Vert.X! 

[Tim Fox]

Thanks Oje!

I'm glad Vert.x is working well for you :)