Re: [storm-user] update storm form 0.7.1 to 0.8.0, get java.io.NotSerializableException

532 views
Skip to first unread message

Nathan Marz

unread,
Aug 23, 2012, 4:21:56 AM8/23/12
to storm...@googlegroups.com
For starters, you should provide us with the exception stack trace you're seeing. 

On Wed, Aug 22, 2012 at 11:42 PM, celix <cj.m...@gmail.com> wrote:
hi, i update my storm from  form 0.7.1 to 0.8.0, and i submitt the same job, but got  java.io.NotSerializableException, who knows why?

ps: i recompile the job by new jar



--
Twitter: @nathanmarz
http://nathanmarz.com

陈竞

unread,
Aug 23, 2012, 11:33:37 AM8/23/12
to storm...@googlegroups.com
the exception belows:
  
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:495)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
        at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
        at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
        at backtype.storm.daemon.worker$mk_transfer_fn$fn__9587$fn__9591.invoke(worker.clj:99)
        at backtype.storm.util$fast_list_map.invoke(util.clj:770)
        at backtype.storm.daemon.worker$mk_transfer_fn$fn__9587.invoke(worker.clj:99)
        at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__9175.invoke(executor.clj:205)
        at backtype.storm.disruptor$clojure_handler$reify__6931.onEvent(disruptor.clj:43)
        at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:79)
        ... 6 more

it seems that the topology.kryo.register doesn't work well


2012/8/23 Nathan Marz <natha...@gmail.com>



--
陈竞，中科院计算技术研究所，高性能计算机中心
Jing Chen HPCC.ICT.AC China

Nathan Marz

unread,
Aug 24, 2012, 1:43:08 AM8/24/12
to storm...@googlegroups.com
Please show the full exception. If possible, please provide a reproducible test case as well. You aren't providing enough information for us to help you.

陈竞

unread,
Aug 24, 2012, 2:35:33 AM8/24/12
to storm...@googlegroups.com

2012-08-23 14:50:37 worker [INFO Worker 7795ed12-2b4f-47de-8d2e-6faea5d0b8f0 for storm CJTest-7-1345703487 on d6df63e0-68b7-49c8-af99-5e08fde34569:6700 has finished loading
2012-08-23 14:50:39 STDIO [INFO] Reader thread, emit one chunk 2979
2012-08-23 14:50:39 util [ERROR] Async loop died!
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: java.lang.RuntimeException: java.io.NotSerializableException: com.xxxx.topology.logdata.EchoTest
Serialization trace:
ads (com.xxxx.topology.logdata.XXXCJLogEntry)
        at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:82)
        at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55)
        at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56)
        at backtype.storm.disruptor$consume_loop_STAR_$fn__6947.invoke(disruptor.clj:67)
        at backtype.storm.util$async_loop$fn__5734.invoke(util.clj:377)
        at clojure.lang.AFn.run(AFn.java:24)
        at java.lang.Thread.run(Thread.java:662)
Caused by: com.esotericsoftware.kryo.KryoException: java.lang.RuntimeException: java.io.NotSerializableException: com.xxxx.topology.logdata.EchoTest
Serialization trace:
ads (com.xxxx.topology.logdata.XXXCJLogEntry)
        at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:495)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
        at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
        at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
        at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
        at backtype.storm.daemon.worker$mk_transfer_fn$fn__9587$fn__9591.invoke(worker.clj:99)
        at backtype.storm.util$fast_list_map.invoke(util.clj:770)
        at backtype.storm.daemon.worker$mk_transfer_fn$fn__9587.invoke(worker.clj:99)
        at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__9175.invoke(executor.clj:205)
        at backtype.storm.disruptor$clojure_handler$reify__6931.onEvent(disruptor.clj:43)
        at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:79)
        ... 6 more
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: com.xxxx.topology.logdata.EchoTest
        at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
        at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:535)
        at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:480)
        ... 22 more
Caused by: java.io.NotSerializableException: com.xxxx.topology.logdata.EchoTest
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
        at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1346)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1154)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
        at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
        ... 24 more
2012-08-23 14:50:39 executor [ERROR] 
java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: java.lang.RuntimeException: java.io.NotSerializableException: com.xxxx.topology.logdata.EchoTest



2012/8/24 Nathan Marz <natha...@gmail.com>

陈竞

unread,
Aug 24, 2012, 2:36:46 AM8/24/12
to storm...@googlegroups.com
i can't get the source code,  i only have the log, can you help me

2012/8/24 陈竞 <cj.m...@gmail.com>

Nathan Marz

unread,
Aug 24, 2012, 2:52:12 AM8/24/12
to storm...@googlegroups.com
The error is telling you exactly what the problem is. You have an object that Storm doesn't know how to serialize. See this wiki page: https://github.com/nathanmarz/storm/wiki/Serialization

陈竞

unread,
Aug 24, 2012, 2:56:25 AM8/24/12
to storm...@googlegroups.com
but the same code works well in storm 0.7.1,  and i see the changelog that there is no upgrade in serialization from 0.7.1 to 0.8.0

2012/8/24 Nathan Marz <natha...@gmail.com>

Nathan Marz

unread,
Aug 24, 2012, 3:01:16 AM8/24/12
to storm...@googlegroups.com
Kryo was upgraded to 2.17, which means that serialization *completely* changed. Likely, Kryo was serializing your object before and now it isn't. So you probably just have to register it, as detailed on that wiki page.

陈竞

unread,
Aug 24, 2012, 3:07:32 AM8/24/12
to storm...@googlegroups.com
topology.kryo.register:
  - java.util.ArrayList
  - java.security.MessageDigest$Delegate
  - com.xxxx.topology.TTSpout$Report
  - com.xxxx.topology.RTTSpout$Report
  - com.xxxx.topology.CounterBolt$Report
  - com.xxxx.topology.logdata.Time
  - com.xxxx.topology.logdata.EchoTest

i have register it in a conf.yaml, but i seems doesn't work. and i saw that if class is not registered, it will get "class is not registered " in log




2012/8/24 Nathan Marz <natha...@gmail.com>

Nathan Marz

unread,
Aug 27, 2012, 5:48:18 AM8/27/12
to storm...@googlegroups.com
Storm will not recognize anything called "conf.yaml". Please be precise in explaining exactly where you registered your serializations. Or even better, please send a reproducible test case.

陈竞

unread,
Aug 29, 2012, 12:40:30 AM8/29/12
to storm...@googlegroups.com
i create a feedback.yaml in $STORM/conf, and i get add such code 
conf.putAll(Utils.findAndReadConfigFile("feedback.yaml"));
then i see the configuration in work's log , the log is:

worker [INFO] Worker has topology config {
             ....
         topology.kryo.register {xxxx nil, xxxx nil, ...}
....
}
xxxx means the class i register in feedback.yaml, forgive me not showing the class name because of privacy.

so what is the real reason? thankyou very much

2012/8/27 Nathan Marz <natha...@gmail.com>
Reply all
Reply to author
Forward
0 new messages