Having issues running Cucumber testing in Scala/Spark - NoClassDefFoundError: cucumber/api/cli/Main

[Jeffrey Kang]

I am writing a scala application using a maven build that i would like to run in spark. I have no issues running this locally and just hitting play will run the tests fine. However, when i run a spark-submit I get the error displayed below.  

Exception in thread "main" java.lang.NoClassDefFoundError: cucumber/api/cli/Main

at cucumberFunctions.myTest$.main(myTest.scala:16)

at cucumberFunctions.myTest.main(myTest.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:733)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:177)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:202)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:116)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.ClassNotFoundException: cucumber.api.cli.Main

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 11 more

These are the dependencies in my pom.xml

    <dependency>

            <groupId>info.cukes</groupId>

            <artifactId>cucumber-scala_2.11</artifactId>

            <version>1.2.4</version>

        </dependency>

        <dependency>

            <groupId>info.cukes</groupId>

            <artifactId>cucumber-junit</artifactId>

            <version>1.2.4</version>

            <scope>test</scope>

        </dependency>

<dependency>

        <groupId>info.cukes</groupId>

        <artifactId>cucumber-core</artifactId>

        <version>1.2.4</version>

</dependency>

Jeff

[Paolo Ambrosio]

Hi Jeff,

Technically, it looks like you are not packaging cucumber-core in your fat jar. All dependencies needed by your job's main method need to be bundled in the fat jar you upload for the Spark cluster, and sometimes you need shading if Spark's own dependencies have conflicts with yours (the Kafka client is a well-known example).

On the approach, I can't think of a reason why submitting a Spark job with Cucumber tests in it would make sense. I'm happy to learn something new if you have one: Spark jobs in the industry suffer from lack of testing and I have not seen really good patterns yet. Let me explain...

You could be doing Integrated Acceptance Tests (black box), so you would set up input data, submit your job, verify the output. In this kind of test you'll have to painfully generate input and expected output for a large amount of data.. and for each case described in Cucumber by pre- and post-conditions (Given/Then). I find that Cucumber is not well suited for this kind of Spark testing.

Otherwise you could be doing Decoupled Acceptance Tests (as in Aslak's excellent talk [1]). In this case, you would run Spark locally (no submit) and use something like Spark Testing Base [2], that lets you define and make assertions on Datasets/RDDs/... without reading and writing from and to your real data sources. I never tried, but I see how someone could use Cucumber for this.

Cheers,

Paolo

Links:

 [1] https://youtu.be/uwH83PjR98g

 [2] https://github.com/holdenk/spark-testing-base

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.