kafka-streams depedency problem

[Darrel Riekhof]

I've been trying to run one of the kafka-streams tutorials out there.  Currently trying https://kafka.apache.org/10/documentation/streams/tutorial

But I'm having problems getting my maven dependencies right:

java -cp target/streams.examples-0.1-shaded.jar myapps.Pipe
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/streams/StreamsBuilder
 at myapps.Pipe.main(Pipe.java:29)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.streams.StreamsBuilder
 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

I've included this, and various other kafka dependencies in my POM but it seems to be missing some required dep deep under the covers that StreamsBuilder and other classes in the kafka-streams jar need.  Has anyone had this problem?  OS is RH 7.3.

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-streams</artifactId>
            <version>1.0.0</version>
        </dependency>

[Matthias J. Sax]

Does your jar contain the corresponding files?

You can check via `unzip -l target/streams.examples-0.1-shaded.jar |
grep StreamsBuilder`

If not, it seems your maven build is not set up correctly to assembly a
fat jar including all dependencies.

As your code compiles the dependency itself is specified correctly.
Otherwise, you would get an compile error on `mvn compile`

As an alternative, you can add the dependencies to you classpath
manually. Or you try our `bin/kafka-run-class` that helps setting up the
classpath.


-Matthias

On 2/12/18 10:10 AM, Darrel Riekhof wrote:
> I've been trying to run one of the kafka-streams tutorials out there.
>  Currently trying https://kafka.apache.org/10/documentation/streams/tutorial
>
> But I'm having problems getting my maven dependencies right:
>
> |
> java -cp target/streams.examples-0.1-shaded.jar myapps.Pipe
> Exceptioninthread
> "main"java.lang.NoClassDefFoundError:org/apache/kafka/streams/StreamsBuilder
>  at myapps.Pipe.main(Pipe.java:29)

> Causedby:java.lang.ClassNotFoundException:org.apache.kafka.streams.StreamsBuilder

>  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
> |
>
> I've included this, and various other kafka dependencies in my POM but
> it seems to be missing some required dep deep under the covers that
> StreamsBuilder and other classes in the kafka-streams jar need.  Has
> anyone had this problem?  OS is RH 7.3.
>
> |
>         <dependency>
>             <groupId>org.apache.kafka</groupId>
>             <artifactId>kafka-streams</artifactId>
>             <version>1.0.0</version>
>         </dependency>
>
> |
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/b4421cc5-24e9-4291-9696-26571fab51bf%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/b4421cc5-24e9-4291-9696-26571fab51bf%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Darrel Riekhof]

I'd messed around with several things before to get it to compile and build that shaded jar.  When I start over fresh and just follow the tutorial exactly I get this.

$ mvn clean package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Kafka Streams Quickstart :: Java 0.1
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ streams.examples ---
[INFO] Deleting /home/driekhof/dev/gitProjects/streams.examples/target
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ streams.examples ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ streams.examples ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/driekhof/dev/gitProjects/streams.examples/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/foo/dev/gitProjects/streams.examples/src/main/java/myapps/Pipe.java:[20]
 import org.apache.kafka.streams.KafkaStreams;
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The import org.apache.kafka.streams.KafkaStreams cannot be resolved
[ERROR] /home/foo/dev/gitProjects/streams.examples/src/main/java/myapps/Pipe.java:[44]
 builder.stream("streams-plaintext-input").to("streams-pipe-output");
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The type org.apache.kafka.streams.kstream.KStream cannot be resolved. It is indirectly referenced from required .class files
[ERROR] /home/foo/dev/gitProjects/streams.examples/src/main/java/myapps/Pipe.java:[44]
 builder.stream("streams-plaintext-input").to("streams-pipe-output");
        ^^^^^^
The method stream(String) from the type StreamsBuilder refers to the missing type KStream
3 problems (3 errors)
[INFO] 3 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.256 s
[INFO] Finished at: 2018-02-12T10:42:40-08:00
[INFO] Final Memory: 13M/210M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project streams.examples: Compilation failure: Compilation failure:
[ERROR] /home/foo/dev/gitProjects/streams.examples/src/main/java/myapps/Pipe.java:[20]
[ERROR] import org.apache.kafka.streams.KafkaStreams;
[ERROR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] The import org.apache.kafka.streams.KafkaStreams cannot be resolved
[ERROR] /home/foo/dev/gitProjects/streams.examples/src/main/java/myapps/Pipe.java:[44]
[ERROR] builder.stream("streams-plaintext-input").to("streams-pipe-output");
[ERROR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] The type org.apache.kafka.streams.kstream.KStream cannot be resolved. It is indirectly referenced from required .class files
[ERROR] /home/foo/dev/gitProjects/streams.examples/src/main/java/myapps/Pipe.java:[44]
[ERROR] builder.stream("streams-plaintext-input").to("streams-pipe-output");
[ERROR] ^^^^^^
[ERROR] The method stream(String) from the type StreamsBuilder refers to the missing type KStream
[ERROR] 3 problems (3 errors)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

So either something is wrong with the Maven project template I'm pulling, or something is messed up with my environment, or something else?

[Darrel Riekhof]

Mystery solved.  The kafka-streams jar in my local maven repo was corrupted.  Deleting it and pulling down a fresh copy fixed all my problems.

On Monday, February 12, 2018 at 11:06:24 AM UTC-8, Darrel Riekhof wrote:

I'd messed around with several things before to get it to compile and build that shaded jar.  When I start over fresh and just follow the tutorial exactly I get this.

snip