Error while running OpenSOC Pcap topology

[vidhur...@gmail.com]

Hi,

    I get a ClassNotFoundException while trying to run the Pcap topology from OpenSOC_Topologies project.

java.lang.ClassNotFoundException: com.opensoc.indexing.adapters.ESTimedRotatingAdapter
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:191)
        at com.opensoc.topology.runner.TopologyRunner.initializeIndexingBolt(TopologyRunner.java:818)
        at com.opensoc.topology.runner.TopologyRunner.initTopology(TopologyRunner.java:314)
        at com.opensoc.topology.Pcap.main(Pcap.java:38)

Has anybody else come across this issue ?

Below are the logs and the version of storm and mvn that i'm using.

[root@sandbox opensoc-streaming]# storm jar ./OpenSOC-Topologies/target/OpenSOC-Topologies-0.5BETA.jar com.opensoc.topology.Pcap -local_mode true -config_path ./OpenSOC-Topologies/target/classes/OpenSOC_Configs/ -generator_spout true
Running: java -client -Dstorm.options= -Dstorm.home=/usr/hdp/2.2.0.0-2041/storm -Dstorm.log.dir=/usr/hdp/2.2.0.0-2041/storm/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.2.0.0-2041/storm/lib/crypto-equality-1.0.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jetty-io-7.6.13.v20130916.jar:/usr/hdp/2.2.0.0-2041/storm/lib/clojure-1.5.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ranger-plugins-audit-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/storm/lib/hadoop-common-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ring-anti-forgery-1.0.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/tools.cli-0.2.4.jar:/usr/hdp/2.2.0.0-2041/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.2.0.0-2041/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.2.0.0-2041/storm/lib/commons-fileupload-1.2.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jgrapht-core-0.9.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/mysql-connector-java.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ranger-plugins-impl-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jetty-servlets-7.6.13.v20130916.jar:/usr/hdp/2.2.0.0-2041/storm/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.2.0.0-2041/storm/lib/logback-classic-1.0.6.jar:/usr/hdp/2.2.0.0-2041/storm/lib/clj-time-0.4.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/joda-time-2.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.2.0.0-2041/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jetty-client-7.6.13.v20130916.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jetty-servlet-7.6.13.v20130916.jar:/usr/hdp/2.2.0.0-2041/storm/lib/guava-11.0.2.jar:/usr/hdp/2.2.0.0-2041/storm/lib/snakeyaml-1.11.jar:/usr/hdp/2.2.0.0-2041/storm/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/clj-stacktrace-0.2.4.jar:/usr/hdp/2.2.0.0-2041/storm/lib/objenesis-1.2.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jetty-server-7.6.13.v20130916.jar:/usr/hdp/2.2.0.0-2041/storm/lib/commons-configuration-1.10.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ranger-plugins-common-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jetty-continuation-7.6.13.v20130916.jar:/usr/hdp/2.2.0.0-2041/storm/lib/clout-1.0.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.2.0.0-2041/storm/lib/json-simple-1.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.2.0.0-2041/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.2.0.0-2041/storm/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/kryo-2.21.jar:/usr/hdp/2.2.0.0-2041/storm/lib/minlog-1.2.jar:/usr/hdp/2.2.0.0-2041/storm/lib/math.numeric-tower-0.0.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jersey-bundle-1.17.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/gson-2.2.4.jar:/usr/hdp/2.2.0.0-2041/storm/lib/zookeeper.jar:/usr/hdp/2.2.0.0-2041/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.2.0.0-2041/storm/lib/hadoop-auth-2.4.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/crypto-random-1.2.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jetty-util-7.6.13.v20130916.jar:/usr/hdp/2.2.0.0-2041/storm/lib/commons-io-2.4.jar:/usr/hdp/2.2.0.0-2041/storm/lib/commons-exec-1.1.jar:/usr/hdp/2.2.0.0-2041/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.2.0.0-2041/storm/lib/chill-java-0.3.5.jar:/usr/hdp/2.2.0.0-2041/storm/lib/commons-logging-1.2.jar:/usr/hdp/2.2.0.0-2041/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.2.0.0-2041/storm/lib/storm-core-0.9.3.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/storm/lib/tools.macro-0.1.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/asm-4.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/slf4j-api-1.6.5.jar:/usr/hdp/2.2.0.0-2041/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.2.0.0-2041/storm/lib/carbonite-1.4.0.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ranger-storm-plugin-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/storm/lib/ranger-plugins-cred-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/storm/lib/logback-core-1.0.6.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jetty-security-7.6.13.v20130916.jar:/usr/hdp/2.2.0.0-2041/storm/lib/jetty-http-7.6.13.v20130916.jar:/usr/hdp/2.2.0.0-2041/storm/lib/commons-lang-2.5.jar:./OpenSOC-Topologies/target/OpenSOC-Topologies-0.5BETA.jar:/usr/hdp/2.2.0.0-2041/storm/conf:/usr/hdp/2.2.0.0-2041/storm/bin -Dstorm.jar=./OpenSOC-Topologies/target/OpenSOC-Topologies-0.5BETA.jar com.opensoc.topology.Pcap -local_mode true -config_path ./OpenSOC-Topologies/target/classes/OpenSOC_Configs/ -generator_spout true
[OpenSOC] Starting topology deployment...
[OpenSOC] Debug mode set to: true
[OpenSOC] Local mode set to: true
[OpenSOC] Setting config path to external config path: ./OpenSOC-Topologies/target/classes/OpenSOC_Configs/
[OpenSOC] Looking for environment identifier: ./OpenSOC-Topologies/target/classes/OpenSOC_Configs//topologies/environment_identifier.conf
[OpenSOC] Looking for topology identifier: ./OpenSOC-Topologies/target/classes/OpenSOC_Configs//topologies/pcap/topology_identifier.conf
[OpenSOC] Looking for topology config: ./OpenSOC-Topologies/target/classes/OpenSOC_Configs//topologies/pcap/topology.conf
[OpenSOC] Initializing Topology: mtd_allen_dev_pcap_P001
[OpenSOC] Initializing Spout: mtd_allen_dev_pcap_P001
[OpenSOC] Initializing Test Spout
[OpenSOC] ------Component TestSpout initialized with the following settings:
[OpenSOC] Key: spout.test.name -> TestSpout
[OpenSOC] Key: spout.test.enabled -> true
[OpenSOC] Key: spout.test.num.tasks -> 1
[OpenSOC] Key: spout.test.parallelism.hint -> 1
[OpenSOC] Key: spout.test.parallelism.repeat -> false
[OpenSOC] ------ParserBolt is initializing from TestSpout
491  [main] INFO  com.opensoc.parsing.PcapParserBolt - Configured for MICRO, setting timePrecisionDivisor to 1000L
[OpenSOC] ------Component ParserBolt initialized with the following settings:
[OpenSOC] Key: parser.bolt.name -> ParserBolt
[OpenSOC] ------IndexBolt is initializing from ParserBolt
java.lang.ClassNotFoundException: com.opensoc.indexing.adapters.ESTimedRotatingAdapter
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:191)
        at com.opensoc.topology.runner.TopologyRunner.initializeIndexingBolt(TopologyRunner.java:818)
        at com.opensoc.topology.runner.TopologyRunner.initTopology(TopologyRunner.java:314)
        at com.opensoc.topology.Pcap.main(Pcap.java:38)

[root@sandbox opensoc-streaming]# storm version
0.9.3.2.2.0.0-2041

[root@sandbox opensoc-streaming]# mvn -version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T17:29:23+00:00)
Maven home: /usr/local/apache-maven-3.2.5
Java version: 1.7.0_71, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-504.1.3.el6.x86_64", arch: "amd64", family: "unix"
[root@sandbox opensoc-streaming]#


Vidhuran

[vidhur...@gmail.com]

As a workaround for this issue , i set the <minimizeJar> option to false in the topologies project pom file. 

Vidhuran

[colombo...@gmail.com]

Thank you for your solution! Works perfectly for me too.


Pierre