UnsatisfiedLinkError on Lib rocks DB dll when developing with Kafka Streams

1,850 views
Skip to first unread message

Giulio Vito de Musso

unread,
May 3, 2017, 3:00:43 AM5/3/17
to Confluent Platform
Copy from here: http://stackoverflow.com/questions/43742423/unsatisfiedlinkerror-on-lib-rocks-db-dll-when-developing-with-kafka-streams

I'm writing a Kafka Streams application on my development Windows machine. If I try to use the leftJoin and branch features of Kafka Streams I get the error below when executing the jar application:

Exception in thread "StreamThread-1" java.lang.UnsatisfiedLinkError: C:\Users\user\AppData\Local\Temp\librocksdbjni325337723194862275.dll: Can't find dependent libraries
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
    at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
    at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
    at org.rocksdb.Options.<clinit>(Options.java:22)
    at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:115)
    at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:38)
    at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:75)
    at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:72)
    at org.apache.kafka.streams.state.internals.ChangeLoggingSegmentedBytesStore.put(ChangeLoggingSegmentedBytesStore.java:54)
    at org.apache.kafka.streams.state.internals.MeteredSegmentedBytesStore.put(MeteredSegmentedBytesStore.java:101)
    at org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:109)
    at org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:101)
    at org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:65)
    at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:48)
    at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
    at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:134)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:83)
    at org.apache.kafka.streams.kstream.internals.KStreamFlatMapValues$KStreamFlatMapValuesProcessor.process(KStreamFlatMapValues.java:43)
    at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:48)
    at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
    at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:134)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:83)
    at org.apache.kafka.streams.kstream.internals.KStreamFilter$KStreamFilterProcessor.process(KStreamFilter.java:44)
    at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:48)
    at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
    at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:134)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:83)
    at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:70)
    at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:197)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:641)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)

It seems like Kafka does not find a DLL, but wait...I'm developing a Java application! Why is the Java Maven application trying to access an external DLL? Should it try to access only classpath stuff?

What could be the problem? And why this error doesn't show off if I try to do simpler streaming operations like only a filter?

Thank you

Giulio Vito de Musso

unread,
May 3, 2017, 5:11:01 AM5/3/17
to Confluent Platform

I think I found the problem. In my local Maven repository I had two versions of RocksDB, version 4.4.1 and 5.0.1 (which is the one used by Kafka Streams 0.10.2 I'm using). I deleted the 4.4.1 version and the problem went away. The strange thing is that Maven was using the old version of the library.. 

Giulio Vito de Musso

unread,
May 3, 2017, 12:01:28 PM5/3/17
to Confluent Platform

UPDATE:


The solution I posted is not correct. In fact, the problem raises only when a message is present in the broker (and there were no messages in the broker when I posted the "solution").

I'm using Kafka Streams version 0.10.2.1.


This is the piece of code which raises the problem


public class KafkaStreamsMainClass {

    private KafkaStreamsMainClass() {
    }

    public static void main(final String[] args) throws Exception {
        Properties streamsConfiguration = new Properties();
        streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "kafka-streams");
        streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-server:9092");
        streamsConfiguration.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "schema-registry:8082");
        streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10 * 1000);
        streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
        streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, GenericAvroSerde.class);
        streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, GenericAvroSerde.class);
        KStreamBuilder builder = new KStreamBuilder();
        KStream<GenericRecord, GenericRecord> sourceStream = builder.stream(SOURCE_TOPIC);

        KStream<GenericRecord, GenericRecord> finishedFiltered = sourceStream
                .filter((GenericRecord key, GenericRecord value) -> value.get("endTime") != null);

        KStream<GenericRecord, GenericRecord>[] branchedStreams = sourceStream
                .filter((GenericRecord key, GenericRecord value) -> value.get("endTime") == null)
                .branch((GenericRecord key, GenericRecord value) -> value.get("firstField") != null,
                        (GenericRecord key, GenericRecord value) -> value.get("secondField") != null);

        branchedStreams[0] = finishedFiltered.join(branchedStreams[0],
                (GenericRecord value1, GenericRecord value2) -> {
                    return value1;
                }, JoinWindows.of(TimeUnit.SECONDS.toMillis(2)));

        branchedStreams[1] = finishedFiltered.join(branchedStreams[1],
                (GenericRecord value1, GenericRecord value2) -> {
                    return value1;
                }, JoinWindows.of(TimeUnit.SECONDS.toMillis(2)));

        KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
        streams.setUncaughtExceptionHandler((Thread thread, Throwable throwable) -> {
            throwable.printStackTrace();
        });
        streams.start();

        Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
    }

}


I opened the rocksdbjni-5.0.1.jar archive downloaded by Maven and it includes the librocksdbjni-win64.dll library. It seems that it is trying to retrieve the library from the outside (from C:\Users\user\AppData\Local\Temp\) of the RocksDB instead from the inner.


I'm developing on a Windows 7 machine.

Have you ever experienced this problem?



Il giorno mercoledì 3 maggio 2017 09:00:43 UTC+2, Giulio Vito de Musso ha scritto:

Matthias J. Sax

unread,
May 3, 2017, 11:19:37 PM5/3/17
to confluent...@googlegroups.com
Some background information:

Kafka Stream uses RocksDB under the hood to manage state. RocksDB is
written in C++ and provides a Java binding to be used within Java. Thus,
if RocksDB gets used, it needs to load a dll on Windows.

If you only use `filter`, there is no store, and thus RocksDB is not
used and you don't see the error. Stores are only used for aggregations
and joins (or if you read a topic as a KTable).

We got multiple error report for Windows with regard to RocksDB and just
want point out, the Windows is officially not yet supported (we are
working on Windows support though...)

Maybe you can try to change the RocksDB version? Just update the
corresponding dependency in your project setup. The latest version is 5.2.1.

Hope this helps.


-Matthias
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

signature.asc

Giulio Vito de Musso

unread,
May 3, 2017, 11:48:09 PM5/3/17
to confluent...@googlegroups.com
Ho Matthias,

Thanks for the explanation. I already tried changing che RocksDB version but nothing new happened...I also noticed that the parere version (5.2.1 if I remember good) does not embed the windows DLL in the jar.

So also if I was developing in a Linux machine I would have had to install locally the RocksDB libraries (e.g. via yum or apt-get)?

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.
> To post to this group, send email to confluent-platform@googlegroups.com
> <mailto:confluent-platform@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/5YFNQtdBuf4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/5abb761a-d85d-1319-7e40-f76eef6e89b7%40confluent.io.

Matthias J. Sax

unread,
May 4, 2017, 12:26:24 PM5/4/17
to confluent...@googlegroups.com
Hi,

if you run on Linux (same as Windows -- after the issues got fixed) you
will not need to install anything. Your application jar should be self
contained.


-Matthias

On 5/3/17 8:48 PM, Giulio Vito de Musso wrote:
> Ho Matthias,
>
> Thanks for the explanation. I already tried changing che RocksDB version
> but nothing new happened...I also noticed that the parere version (5.2.1
> if I remember good) does not embed the windows DLL in the jar.
>
> So also if I was developing in a Linux machine I would have had to
> install locally the RocksDB libraries (e.g. via yum or apt-get)?
>
> Il 4 mag 2017 5:19 AM, "Matthias J. Sax" <matt...@confluent.io
> <mailto:matt...@confluent.io>> ha scritto:
> > an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platform%2Bunsu...@googlegroups.com>
> > <mailto:confluent-platf...@googlegroups.com
> <mailto:confluent-platform%2Bunsu...@googlegroups.com>>.
> > To post to this group, send email to
> confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>
> > <mailto:confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com>
> >
> <https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "Confluent Platform" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/confluent-platform/5YFNQtdBuf4/unsubscribe
> <https://groups.google.com/d/topic/confluent-platform/5YFNQtdBuf4/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to
> confluent-platf...@googlegroups.com
> <mailto:confluent-platform%2Bunsu...@googlegroups.com>.
> To post to this group, send email to
> confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/5abb761a-d85d-1319-7e40-f76eef6e89b7%40confluent.io
> <https://groups.google.com/d/msgid/confluent-platform/5abb761a-d85d-1319-7e40-f76eef6e89b7%40confluent.io>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/CAJN_W1RaXSpmscP9YFmbx5X_dEGDhfa1h6L69wTUB03w8kk39w%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAJN_W1RaXSpmscP9YFmbx5X_dEGDhfa1h6L69wTUB03w8kk39w%40mail.gmail.com?utm_medium=email&utm_source=footer>.
signature.asc

qi yao

unread,
Sep 25, 2017, 5:12:21 AM9/25/17
to Confluent Platform

Hi gvdm,  

you can install the vc++2005 runtime to resolve this issue, url : https://www.microsoft.com/en-us/download/details.aspx?id=48145.


在 2017年5月4日星期四 UTC+8上午11:48:09，gvdm写道：
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/5YFNQtdBuf4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

qi yao

unread,
Sep 25, 2017, 5:12:23 AM9/25/17
to Confluent Platform

Hi gvdm,  

you can install the vc++2005 runtime to resolve this issue, url : https://www.microsoft.com/en-us/download/details.aspx?id=48145.


在 2017年5月4日星期四 UTC+8上午11:48:09，gvdm写道：
Ho Matthias,
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/78917597-940d-40be-b5b0-6bd7d1d76bd5%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/5YFNQtdBuf4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Salvatore Milite

unread,
Nov 16, 2018, 9:00:49 AM11/16/18
to Confluent Platform
Hi Dean,
I searched the whole internet about this error. Everyone said "new version does not have the issue". Your fix is really the right solution.
Thank you a lot

Greetz
Reply all
Reply to author
Forward
0 new messages