Error for /dataflow org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null

[sudheer Aki]

Hi All,

I am using pipeline code to read a file from Cloud storage.

Code :

DataflowPipelineOptions options = PipelineOptionsFactory.create().as(DataflowPipelineOptions.class);
options.setRunner(BlockingDataflowPipelineRunner.class);
options.setProject("projectId");
options.setStagingLocation("StagingPath");

    //PipelineOptions options = PipelineOptionsFactory.create();
    // Then create the pipeline.
    Pipeline p = Pipeline.create(options);
        PCollection<String> lines = p.apply(TextIO.Read.named("ReadDataFromStorage").from("source_Path"));

When am running from local eclipse am able to read a file.

But when i deployed to App engine i am getting below error.I am using snappy-java-1.1.2.1.jar.Please see the error below and help me ASAP.

Error for /dataflow
org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:159)
at org.xerial.snappy.Snappy.(Snappy.java:47)
at org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:97)
at org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:89)
at org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:79)
at com.google.cloud.dataflow.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:49)
at com.google.cloud.dataflow.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:84)
at com.google.cloud.dataflow.sdk.io.Read$Bounded.(Read.java:108)
at com.google.cloud.dataflow.sdk.io.Read$Bounded.(Read.java:103)
at com.google.cloud.dataflow.sdk.io.Read.from(Read.java:62)
at com.google.cloud.dataflow.sdk.io.TextIO$Read$Bound.apply(TextIO.java:319)

[Evan Jones]

You cannot use anything that uses JNI on App Engine. You'll have to see if you can configure this Snappy library to use a "pure Java" version, instead of trying to load a native library. See:

https://cloud.google.com/appengine/docs/java/runtime

[Nicholas (Google Cloud Support)]

As mentioned here previously, you cannot use JNI on App Engine due to sandbox security limitations.  Though I'm not too familiar with the Snappy library, I've not found much in its documentation about a pure Java implementation.  I would have suggested you file a new issue on its github discussion group to request a possible pure Java port but found that you have already done so.  That is the appropriate forum for that native Java support specifically.

If you must use Snappy, you could technically use it in a custom runtime on App Engine flexible but would lose access to some key GAE features and be using a beta product.  I would advise against it and recommend seeking out a native Java compression/decompression library.  GAE-compatible compression/decompression libraries would be an excellent idea for a new thread on this discussion group and invite you to start one.  It may also be a good idea to mention what you are planning to compress/decompress and why.

Hope this helps and happy coding!

On Wednesday, August 3, 2016 at 9:42:23 AM UTC-4, sudheer Aki wrote: