Storm-0.9.0-wip21 failed for topology with Hadoop and Hbase dependencies

156 views
Skip to first unread message

Zheng Xue

unread,
Mar 6, 2014, 10:46:45 PM3/6/14
to storm...@googlegroups.com
Hi, all:

    I was trying to build a Storm topology with Hadoop and Hbase dependencies. And I want to run this topology with Storm-on-Yarn. The version of Storm in it is 0.9.0-wip21. I created the Jar file with Maven, and the pom.xml file is attached.

    I submitted the topology (with-dependencies), and there was no exceptions. But it did't run at all. I checked the supervisor logs (see bottom), which shows that the workers failed to start.

    To trace the reason of this issue, I add a test topology (WordCountTopology) in the jar file. The problems remains when trying to submit the WordCountTopology with dependencies, but it works well when trying to submit the topology without dependencies. 

    To make it clear that where is this issue from, I ran the topology without Storm-on-Yarn. I tried Storm-0.8.1 and Storm-0.9.0-wip21. The results show that:
    1) Storm-0.8.1 is okay for the jar topology, no matter with or without dependencies.
    2) Storm-0.9.0-wip21 is okay for the jar topology without dependencies, but the same issue (workers failed to start) appears for the "with-dependencies" jar file.

    

    I am a newer for Storm, and I don't know the differences between these two Storm version. What will result in the worker failure when trying to submit the "with-dependencies" topology: Is the "with-dependencies" jar file too big? or something conflicting?

    Any idea? 
    Will Storm-0.8.1 work with Storm-on-Yarn?
    Thanks~






2014-03-06 15:08:59 b.s.d.supervisor [INFO] Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "      WordCount-1-1394089715", :executors ([2 2] [5 5] [8 8] [11 11] [14 14] [17 17] [20 20] [23 23] [26 26])} for this supervisor 70932268-c723      -4693-b1d6-9e5ef2cc6b2b on port 6702 with id 4514c1c1-8213-44fc-81ba-441f21e48262
   53 2014-03-06 15:08:59 b.s.d.supervisor [INFO] Launching worker with command: /usr/lib/jvm/java-7-openjdk-amd64//bin/java -server -Xmx768m  -      Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=/home/xuezh/hadoop-2.2.0/logs/userlogs/application_1394075359032_      0003/container_1394075359032_0003_01_000002/worker-6702.log -Dstorm.home=./storm/storm-0.9.0-wip21/ -Dstorm.id=WordCount-1-1394089715 -Dwo      rker.id=4514c1c1-8213-44fc-81ba-441f21e48262 -Dworker.port=6702 -cp conf:./storm/storm-0.9.0-wip21/:./storm/storm-0.9.0-wip21/storm-consol      e-logging-0.9.0-wip21.jar:./storm/storm-0.9.0-wip21/lib/jgrapht-0.8.3.jar:./storm/storm-0.9.0-wip21/lib/commons-fileupload-1.2.1.jar:./sto      rm/storm-0.9.0-wip21/lib/httpclient-4.1.1.jar:./storm/storm-0.9.0-wip21/lib/minlog-1.2.jar:./storm/storm-0.9.0-wip21/lib/libthrift7-0.7.0-      2.jar:./storm/storm-0.9.0-wip21/lib/json-simple-1.1.jar:./storm/storm-0.9.0-wip21/lib/slf4j-api-1.6.5.jar:./storm/storm-0.9.0-wip21/lib/ri      ng-core-1.1.5.jar:./storm/storm-0.9.0-wip21/lib/clj-time-0.4.1.jar:./storm/storm-0.9.0-wip21/lib/asm-4.0.jar:./storm/storm-0.9.0-wip21/lib      /ring-devel-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/curator-client-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/jetty-6.1.26.jar:./storm/storm      -0.9.0-wip21/lib/httpcore-4.1.jar:./storm/storm-0.9.0-wip21/lib/hiccup-0.3.6.jar:./storm/storm-0.9.0-wip21/lib/commons-codec-1.4.jar:./sto      rm/storm-0.9.0-wip21/lib/ring-servlet-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/logback-core-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/reflec      tasm-1.07-shaded.jar:./storm/storm-0.9.0-wip21/lib/curator-framework-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/math.numeric-tower-0.0.1.jar:      ./storm/storm-0.9.0-wip21/lib/commons-io-1.4.jar:./storm/storm-0.9.0-wip21/lib/junit-3.8.1.jar:./storm/storm-0.9.0-wip21/lib/carbonite-1.5      .0.jar:./storm/storm-0.9.0-wip21/lib/snakeyaml-1.11.jar:./storm/storm-0.9.0-wip21/lib/servlet-api-2.5.jar:./storm/storm-0.9.0-wip21/lib/co      re.incubator-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/disruptor-2.10.1.jar:./storm/storm-0.9.0-wip21/lib/commons-exec-1.1.jar:./storm/storm      -0.9.0-wip21/lib/clout-1.0.1.jar:./storm/storm-0.9.0-wip21/lib/netty-3.6.3.Final.jar:./storm/storm-0.9.0-wip21/lib/clojure-1.4.0.jar:./sto      rm/storm-0.9.0-wip21/lib/tools.cli-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/clj-stacktrace-0.2.2.jar:./storm/storm-0.9.0-wip21/lib/tools.ma      cro-0.1.0.jar:./storm/storm-0.9.0-wip21/lib/jetty-util-6.1.26.jar:./storm/storm-0.9.0-wip21/lib/kryo-2.17.jar:./storm/storm-0.9.0-wip21/li      b/servlet-api-2.5-20081211.jar:./storm/storm-0.9.0-wip21/lib/log4j-over-slf4j-1.6.6.jar:./storm/storm-0.9.0-wip21/lib/objenesis-1.2.jar:./      storm/storm-0.9.0-wip21/lib/jline-0.9.94.jar:./storm/storm-0.9.0-wip21/lib/guava-13.0.jar:./storm/storm-0.9.0-wip21/lib/commons-lang-2.5.j      ar:./storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar:./storm/storm-0.9.0-wip21/lib/compojure-1.1.3.jar:./storm/storm-0.9.0-wip21/lib      /ring-jetty-adapter-0.3.11.jar:./storm/storm-0.9.0-wip21/lib/mockito-all-1.9.5.jar:./storm/storm-0.9.0-wip21/lib/commons-logging-1.1.1.jar      :./storm/storm-0.9.0-wip21/lib/zookeeper-3.3.3.jar:./storm/storm-0.9.0-wip21/lib/tools.logging-0.2.3.jar:./storm/storm-0.9.0-wip21/lib/jzm      q-2.1.0.jar:./storm/storm-0.9.0-wip21/lib/joda-time-2.0.jar:./storm/storm-0.9.0-wip21/storm-netty-0.9.0-wip21.jar:./storm/storm-0.9.0-wip2      1/storm-core-0.9.0-wip21.jar:storm-local/supervisor/stormdist/WordCount-1-1394089715/stormjar.jar backtype.storm.daemon.worker WordCount-1      -1394089715 70932268-c723-4693-b1d6-9e5ef2cc6b2b 6702 4514c1c1-8213-44fc-81ba-441f21e48262
   54 2014-03-06 15:08:59 b.s.d.supervisor [INFO] f2ae26ec-54d4-4429-8597-6745c206c162 still hasn't started
   55 2014-03-06 15:08:59 b.s.d.supervisor [INFO] f2ae26ec-54d4-4429-8597-6745c206c162 still hasn't started
   56 2014-03-06 15:09:00 b.s.d.supervisor [INFO] f2ae26ec-54d4-4429-8597-6745c206c162 still hasn't started
   57 2014-03-06 15:09:00 b.s.d.supervisor [INFO] f2ae26ec-54d4-4429-8597-6745c206c162 still hasn't started
   58 2014-03-06 15:09:01 b.s.d.supervisor [INFO] f2ae26ec-54d4-4429-8597-6745c206c162 still hasn't started
   59 2014-03-06 15:09:01 b.s.d.supervisor [INFO] f2ae26ec-54d4-4429-8597-6745c206c162 still hasn't started
   .....
  293 2014-03-06 15:10:58 b.s.d.supervisor [INFO] f2ae26ec-54d4-4429-8597-6745c206c162 still hasn't started
  294 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Worker f2ae26ec-54d4-4429-8597-6745c206c162 failed to start
  295 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Worker b7b16a5b-2429-4a4c-8100-0d6ac83af954 failed to start
  296 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Worker 4514c1c1-8213-44fc-81ba-441f21e48262 failed to start
  297 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Shutting down and clearing state for id f2ae26ec-54d4-4429-8597-6745c206c162. Current supervis      or time: 1394089859. State: :disallowed, Heartbeat: nil
  298 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Shutting down 70932268-c723-4693-b1d6-9e5ef2cc6b2b:f2ae26ec-54d4-4429-8597-6745c206c162
  299 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Shut down 70932268-c723-4693-b1d6-9e5ef2cc6b2b:f2ae26ec-54d4-4429-8597-6745c206c162
  300 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Shutting down and clearing state for id b7b16a5b-2429-4a4c-8100-0d6ac83af954. Current supervis      or time: 1394089859. State: :disallowed, Heartbeat: nil
  301 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Shutting down 70932268-c723-4693-b1d6-9e5ef2cc6b2b:b7b16a5b-2429-4a4c-8100-0d6ac83af954
  302 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Shut down 70932268-c723-4693-b1d6-9e5ef2cc6b2b:b7b16a5b-2429-4a4c-8100-0d6ac83af954
  303 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Shutting down and clearing state for id 4514c1c1-8213-44fc-81ba-441f21e48262. Current supervis      or time: 1394089859. State: :disallowed, Heartbeat: nil
  304 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Shutting down 70932268-c723-4693-b1d6-9e5ef2cc6b2b:4514c1c1-8213-44fc-81ba-441f21e48262
  305 2014-03-06 15:10:59 b.s.d.supervisor [INFO] Shut down 70932268-c723-4693-b1d6-9e5ef2cc6b2b:4514c1c1-8213-44fc-81ba-441f21e48262
pom-hbase.xml

Bobby Evans

unread,
Mar 7, 2014, 10:04:37 AM3/7/14
to Zheng Xue, storm...@googlegroups.com
My guess is log4j.  Storm switched logging frameworks from log4j in 0.8.2 to logback in 0.9.0.  As part of this they included a slf4j library that implements the log4j APIs, but having log4j  or slf4j-log4j12 in the classpath can really mess things up.  Try to take log4j out of your dependencies, and add in excludes to your other dependencies that use log4j. Here are some of the dependencies that I have included in a pom that works with 0.9.0.

    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase</artifactId>
      <version>0.94.3</version>
      <exclusions>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>0.23.3</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <groupId>org.apache.avro</groupId>
          <artifactId>avro</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-annotations</artifactId>
      <version>0.23.3</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <groupId>org.apache.avro</groupId>
          <artifactId>avro</artifactId>
        </exclusion> 
      </exclusions>
    </dependency>
    <dependency>
      <groupId>storm</groupId>
      <artifactId>storm</artifactId>
      <version>0.9.0-wip15</version>
      <!-- keep storm out of the jar-with-dependencies -->
      <scope>provided</scope>
    </dependency>

—Bobby

--
You received this message because you are subscribed to the Google Groups "storm-yarn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-yarn+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Zheng Xue

unread,
Mar 8, 2014, 6:15:25 AM3/8/14
to storm...@googlegroups.com
Thanks a lot Bobby. It works with your information. I excluded the log4j dependency in the pom file. The topology can run in Storm-0.9.0-wip21 now.

But another error appears in the last bolt, which is responsible for writing statistics into HBase. Here is the exception information:

java.lang.RuntimeException: java.lang.NullPointerException
	at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
	at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
	at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
	at backtype.storm.daemon.executor$fn__3483$fn__3495$fn__3542.invoke(executor.clj:715)
	at backtype.storm.util$async_loop$fn__441.invoke(util.clj:396)
	at clojure.lang.AFn.run(AFn.java:24)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
	at com.mycompany.app.MobileNetLogThresholdBolt.execute(MobileNetLogThresholdBolt.java:81)
	at backtype.storm.daemon.executor$fn__3483$tuple_action_fn__3485.invoke(executor.clj:610)
	at backtype.storm.daemon.executor$mk_task_receiver$fn__3406.invoke(executor.clj:381)
	at backtype.storm.disruptor$clojure_handler$reify__2948.onEvent(disruptor.clj:43)
	at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:79)
Line 81 in MobileNetLogThresholdBolt.java is "table.put(put)" (see the java code of this bolt below). Referred to https://github.com/nathanmarz/storm/wiki/Troubleshooting, multiple threads using the OutputCollector leads to this exception. But how can I fix it? Thanks.

public class MobileNetLogThresholdBolt implements IRichBolt {
    private OutputCollector outputCollector;
    public static Configuration configuration;
    public static String tablename = "t_mobilenet_threshold";
    public static HTable table;
    static {
    configuration = HBaseConfiguration.create();
    configuration.set("hbase.zookeeper.property.clientPort","2181");
    configuration.set("hbase.zookeeper.quorum", "xx.xx.xx.xx");
    configuration.set("hbase.master", "xx.xx.xx.xx:60000");
    }
   
    
    private Log log = LogFactory.getLog(MobileNetLogThresholdBolt.class);

    @Override
    public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
        this.outputCollector = outputCollector;
        try {
table = new HTable(configuration, tablename);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
        
    }


    @Override
public void execute(Tuple tuple) {
        
        log.info("deal data " + tuple.getString(0) + "=" + tuple.getInteger(1));
        if (tuple.getInteger(1) > 2) {
       
            Put put = new Put(Bytes.toBytes(tuple.getString(0)));
            put.add(Bytes.toBytes("STAT_INFO"), Bytes.toBytes("COUNT"), Bytes.toBytes(String.valueOf(tuple.getInteger(1))));
            try {
table.put(put);
} catch (IOException e) {  
           e.printStackTrace();  
       }  
        }
        this.outputCollector.emit(tuple, tuple.getValues());
        this.outputCollector.ack(tuple);
    }

    @Override
    public void cleanup() {

    }
    
    @Override
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
    }

    @Override
    public Map<String, Object> getComponentConfiguration() {
        return null;
    }
}

Bobby Evans

unread,
Mar 8, 2014, 8:25:27 AM3/8/14
to Zheng Xue, storm...@googlegroups.com
The caused by of the exception is coming from your code not deep inside storm like troubleshooting talks about. If line 81 is really table.put then table must be null. My guess is you got an IOException creating the table, which is ignored by your code. 

--Bobby 
On my phone 

Zheng Xue

unread,
Mar 10, 2014, 10:34:03 PM3/10/14
to Bobby Evans, storm...@googlegroups.com
Sorry for the late reply. This exception is not from storm. It is caused by the version issue of Hbase.

Thanks a lot for your help.
--
------------------
Best Regards!

Zheng Xue
http://netlab.sysu.edu.cn/~zhxue
Computer Science and Technology college,Sun Yat-sen University
E-mail: xuez...@gmail.com, xuez...@acm.org
Reply all
Reply to author
Forward
0 new messages