RocksDB Linkage error with Kafka Stream 0.10.1.1 on Windows Platform

[Nitin Gupta]

Dear All,

Would like to know if the RocksDB linkage error is resolved in the latest version of Kafka Streams for the Windows platform. The latest version of Kafka Streams (0.10.1.1) does use the RocksDB 4.9.0  but I am still facing the problem when I run the basic example using Eclipse on Windows.

All my developers are on Windows platform so it will be big shift if we want to move them to Linux to use Kafka Streams. Please suggest.

[Eno Thereska]

We just upgraded RocksDb to 5.0.1 in the upcoming 0.10.2 release (currently being voted) and also in trunk. However, I haven't tried in Windows and it would be good to have someone try and confirm.

Eno

[Matthias J. Sax]

It is strange that it does not work with 0.10.1.1

According to this blog post, RocksDB 4.9.0 should work with windows:
https://technology.amis.nl/2017/02/11/getting-started-with-kafka-streams-building-a-streaming-analytics-java-application-against-a-kafka-topic/#comment-8930

Can you elaborate on the problem you are facing with 0.10.1.1 and
RocksDB 4.9.0 ?

Btw: Windows is officially still not supported (even if we want to make
it work on Windows, too).


-Matthias

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/7e48df45-3400-4bc7-8345-10f70eabaa28%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/7e48df45-3400-4bc7-8345-10f70eabaa28%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Nitin Gupta]

Hi Matthias,

I am receiving the below error.  I am not sure why it check in my Temp dir for the dll. It appends some timestamp in the name of the dll and it changes everytime I run the code. 
I check the rockdb jar associated with my code and it has the dll .

Any help with be appreciable.

Exception in thread "StreamThread-1" java.lang.UnsatisfiedLinkError: C:\Users\nitin.gupta\AppData\Local\Temp\librocksdbjni6618673850139271275.dll: Can't find dependent libraries
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1929)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1814)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1083)
    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
    at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:47)
    at org.rocksdb.RocksDB.<clinit>(RocksDB.java:23)
    at org.rocksdb.Options.<clinit>(Options.java:21)
    at org.apache.kafka.streams.state.internals.RocksDBStore.<init>(RocksDBStore.java:127)
    at org.apache.kafka.streams.state.internals.RocksDBStore.<init>(RocksDBStore.java:112)
    at org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreSupplier.get(RocksDBKeyValueStoreSupplier.java:56)
    at org.apache.kafka.streams.processor.TopologyBuilder.build(TopologyBuilder.java:791)
    at org.apache.kafka.streams.processor.TopologyBuilder.build(TopologyBuilder.java:765)
    at org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:631)
    at org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:660)
    at org.apache.kafka.streams.processor.internals.StreamThread.access$100(StreamThread.java:69)
    at org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:124)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:228)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:313)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:277)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:259)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:407)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)

Below is the entry in my pom file .

<dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-streams</artifactId>
            <version>0.10.1.1-cp1</version>
        </dependency>

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

[Nitin Gupta]

I also tried to exclude the 4.9 version of RockDB from the  KafkaStream dependency in the pom file and explicitly added the dependency to use the 5.1.2 version of Rockdb .
But still the same problem

<dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-streams</artifactId>
            <version>0.10.1.1-cp1</version>

            <exclusions>
                <exclusion>
                    <groupId>org.rocksdb</groupId>
                    <artifactId>rocksdbjni</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.rocksdb</groupId>
            <artifactId>rocksdbjni</artifactId>
            <version>5.1.2</version>
        </dependency>

[Matthias J. Sax]

What Windows version do you use? It might be related to this... Also
32bit or 64bit?

I guess, it's not a Streams issue, but more or less a RocksDB issue. For
example, I did find this: https://github.com/facebook/rocksdb/issues/1302

Maybe it helps.


-Matthias

> > an email to confluent-platf...@googlegroups.com
> > <mailto:confluent-platf...@googlegroups.com>.

> > To post to this group, send email to
> confluent...@googlegroups.com
> > <mailto:confluent...@googlegroups.com>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/7e48df45-3400-4bc7-8345-10f70eabaa28%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/7e48df45-3400-4bc7-8345-10f70eabaa28%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/confluent-platform/7e48df45-3400-4bc7-8345-10f70eabaa28%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/confluent-platform/7e48df45-3400-4bc7-8345-10f70eabaa28%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/2cae5310-8f8d-489c-85ef-b61d63369bd8%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/2cae5310-8f8d-489c-85ef-b61d63369bd8%40googlegroups.com?utm_medium=email&utm_source=footer>.