Issue With JSON Serialization/Deserialization Between Node.js (Node-Kafka), KafkaSpout and Bolt

[Chris Alexander]

Hi All,

First my environment:

Storm: 0.8.1

Kafka: 0.7.2

Storm-Kafka: storm-kafka-0.7.2-20120517.033754-1.jar

Zk: 3.3.4

I have a Node.js producer sending JSON payloads to Kafka using Node-Kafka (https://github.com/marcuswestin/node-kafka). I am consuming these messages from Kafka using a KafaSpout-based topology. I have not been able to successfully deserialize the JSON object in my bolt. I have used json-simple (1.1.1) and jackson (2.1.4) libaries to no success. So my question is what is the Type written to disk by Kafka (docs say binary) vs. Type retrieved by KafkaSpout as a tuple. Here is the code I am using in my bolt to coerce type:

// System.out.print()

System.out.print("Tuple Received = " + tuple.toString());     // Tuple Received = [object Object]

// Jackson

ObjectMapper mapper = new ObjectMapper();

Map.Entry<String, String> promoterRegData = mapper.readValue(tuple.getString(0), Map.Entry.class);

Of course it fails even if I use the byte[]. I would appreciate some guidance here: parser, another compliant version of Storm, storm-kafka spout, Node.js client (Node-Kafka), etc.

Thanks,

Chris

[Chris Alexander]

Solved issue: was not using JSON.stringify() before sending message onto Kafka bus. Hope this oversight helps someone else.

Chris

[Sumant Sankaran]

We have used the same before ,to pump in json objects using nodejs and read using kafka spout .The only thing I can think of is ,are you using the string scheme for Kafka spout ? 

Thanks

Sumant

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.