Java Class Loading in CDAP

r.a.b...@gmail.com

unread,

Nov 18, 2019, 9:34:04 AM11/18/19

to CDAP User

I have a UDD which needs to load a Java class. I've defined it in part of the initialize() step.

localLibraryPath = "/home/my_udd_code/lib";

JAVA_LIBRARY_PATH = "java.library.path";

System.setProperty(JAVA_LIBRARY_PATH, localLibraryPath );

Field fieldSysPath = null;

try {

fieldSysPath = ClassLoader.class.getDeclaredField( "sys_paths" );

fieldSysPath.setAccessible( true );

fieldSysPath.set( null, null );

} catch (NoSuchFieldException | IllegalAccessException e) {

System.out.println("Error setting local library");

}

// Load the JNI library

System.loadLibrary("myjavaclass");

This works perfectly when I run the process first time round. However, subsequent runs throw an error (below). I thought about ignoring the class load on subsequent runs, which is fine unless any other process uses the same UDD - then I get the same error.

I've looked in the manual here https://docs.cask.co/cdap/6.0.0/en/developer-manual/advanced/class-loading.html and here https://docs.cask.co/cdap/6.0.0/en/developer-manual/building-blocks/program-lifecycle.html but I can't quite find what I'm looking for. Ideally I need the class to be loaded once and once only when first used by any specific UDD (or maybe loaded by CDAP at start-up time). Is this possible?

Here's the error:

java.lang.UnsatisfiedLinkError: Native Library /home/my_udd_code/lib/myjavaclass.so already loaded in another classloader

at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1907) ~[na:1.8.0_222]

at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1857) ~[na:1.8.0_222]

at java.lang.Runtime.loadLibrary0(Runtime.java:870) ~[na:1.8.0_222]

at java.lang.System.loadLibrary(System.java:1122) ~[na:1.8.0_222]

at org.example.directives.Anonymise.initialize(Anonymise.java:104) ~[1574081722252-0/:na]

at io.cdap.wrangler.parser.GrammarBasedParser.parse(GrammarBasedParser.java:120) ~[wrangler-core-4.0.0.jar:na]

at io.cdap.wrangler.executor.RecipePipelineExecutor.initialize(RecipePipelineExecutor.java:65) ~[wrangler-core-4.0.0.jar:na]

at io.cdap.wrangler.Wrangler.initialize(Wrangler.java:364) ~[1574081722257-0/:na]

at io.cdap.cdap.etl.common.plugin.WrappedTransform.lambda$initialize$3(WrappedTransform.java:72) ~[cdap-etl-core-6.0.0.jar:na]

at io.cdap.cdap.etl.common.plugin.Caller$1.call(Caller.java:30) ~[cdap-etl-core-6.0.0.jar:na]

at io.cdap.cdap.etl.common.plugin.StageLoggingCaller.call(StageLoggingCaller.java:40) ~[cdap-etl-core-6.0.0.jar:na]

at io.cdap.cdap.etl.common.plugin.WrappedTransform.initialize(WrappedTransform.java:71) ~[cdap-etl-core-6.0.0.jar:na]

at io.cdap.cdap.etl.common.plugin.WrappedTransform.initialize(WrappedTransform.java:34) ~[cdap-etl-core-6.0.0.jar:na]

at io.cdap.cdap.etl.batch.mapreduce.MapReduceTransformExecutorFactory.getInitializedTransformation(MapReduceTransformExecutorFactory.java:319) ~[cdap-etl-batch-6.0.0.jar:na]

at io.cdap.cdap.etl.batch.mapreduce.MapReduceTransformExecutorFactory.getTransformation(MapReduceTransformExecutorFactory.java:189) ~[cdap-etl-batch-6.0.0.jar:na]

at io.cdap.cdap.etl.batch.mapreduce.MapReduceTransformExecutorFactory.getPipeStage(MapReduceTransformExecutorFactory.java:303) ~[cdap-etl-batch-6.0.0.jar:na]

at io.cdap.cdap.etl.batch.mapreduce.MapReduceTransformExecutorFactory.create(MapReduceTransformExecutorFactory.java:220) ~[cdap-etl-batch-6.0.0.jar:na]

at io.cdap.cdap.etl.batch.mapreduce.TransformRunner.<init>(TransformRunner.java:108) ~[cdap-etl-batch-6.0.0.jar:na]

at io.cdap.cdap.etl.batch.mapreduce.ETLMapReduce$ETLMapper.initialize(ETLMapReduce.java:224) ~[cdap-etl-batch-6.0.0.jar:na]

at io.cdap.cdap.etl.batch.mapreduce.ETLMapReduce$ETLMapper.initialize(ETLMapReduce.java:210) ~[cdap-etl-batch-6.0.0.jar:na]

at io.cdap.cdap.internal.app.runtime.batch.MapperWrapper.run(MapperWrapper.java:122) ~[na:na]

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.8.0.jar:na]

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.8.0.jar:na]

at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270) ~[org.apache.hadoop.hadoop-mapreduce-client-common-2.8.0.jar:na]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_222]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_222]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_222]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_222]

at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_222]

alia...@google.com

unread,

Nov 21, 2019, 11:42:25 AM11/21/19

to CDAP User

Already answered here: https://groups.google.com/a/google.com/d/msg/cdap-discuss/HuLHvjRG0G4/Ddb16u-FBgAJ

r.a.b...@gmail.com

unread,

Nov 21, 2019, 3:46:53 PM11/21/19

to CDAP User

Thanks for responding. Unfortunately I can't access that group. Are you able to summarise the answer?

Ali Anwar

unread,

Nov 26, 2019, 1:39:10 PM11/26/19

to cdap...@googlegroups.com

Sure, I'll forward the thread to this mailing list.

--
You received this message because you are subscribed to a topic in the Google Groups "CDAP User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cdap-user/MVE5fghHJKA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/ca236078-1220-40d9-a0cc-2ccb3e3a7e95%40googlegroups.com.

Ali Anwar

unread,

Nov 26, 2019, 1:39:40 PM11/26/19

to cdap...@googlegroups.com

---------- Forwarded message ---------
From: Terence Yim <tere...@google.com>
Date: Mon, Nov 25, 2019 at 9:14 AM
Subject: Re: Java Class Loading in CDAP
To: CDAP Discuss <cdap-d...@google.com>
Cc: Anat Perly <ape...@google.com>, tere...@google.com <tere...@google.com>

Hi,

I don't quite understand the question. Can you clarify it? Do you mean CDAP doesn't unload the library if the pipeline execution failed? What is the runtime environment?

Terence

On Thursday, November 21, 2019 at 10:44:55 AM UTC-8 Anat Perly wrote:

after more digging it seems when a pipeline finishes successfully CDAP presumably unloads the library, whereas when an error occurs it drops out without doing this. Do you have any idea how I could force a known starting point each time where the library wasn’t loaded, or get it to unload properly?

On Wednesday, November 20, 2019 at 5:47:02 PM UTC+2, Anat Perly wrote:
also if i will move and run the code on GCP ( not in the sandbox) will it work ? or will i face the same issue?

On Wed, Nov 20, 2019 at 5:43 PM Anat Perly <ape...@google.com> wrote:
Hi Terence
Thanks for your reply, it is very helpful
I can confirm i can the job in CDAP Sandbox.
Can you please point me to the right bugnizzer link (if there is any) or if this limitation is document somewhere

Thanks.
Anat

On Wed, Nov 20, 2019 at 5:24 PM Anat Perly <ape...@google.com> wrote:
correct i'm CDAP locally

On Wed, Nov 20, 2019 at 5:19 PM Terence Yim <tere...@google.com> wrote:
Hi,

I believe you are running the job in CDAP sandbox, right? There is a problem in using native library in such case, due to native library loading in Java is not aware of different classloaders that CDAP creates for each run. If you run your workload in a cluster (e.g. using Dataproc compute profile), you won’t see the issue.

Terence

On Wed, Nov 20, 2019 at 14:39 Anat Perly <ape...@google.com> wrote:
Hi
Hope it is the correct channel to raise the issue i have.

I have a UDD which needs to load a Java class. I've defined it in part of the initialize() step.

localLibraryPath = "/home/my_udd_code/lib";
JAVA_LIBRARY_PATH = "java.library.path";

System.setProperty(JAVA_LIBRARY_PATH, localLibraryPath );

Field fieldSysPath = null;
try {
fieldSysPath = ClassLoader.class.getDeclaredField( "sys_paths" );
fieldSysPath.setAccessible( true );
fieldSysPath.set( null, null );
} catch (NoSuchFieldException | IllegalAccessException e) {
System.out.println("Error setting local library");
}

// Load the JNI library
System.loadLibrary("myjavaclass");

This works perfectly when I run the process first time round. However, subsequent runs throw an error (below). I thought about ignoring the class load on subsequent runs, which is fine unless any other process uses the same UDD - then I get the same error.

I've looked in the manual here https://www.chilkatsoft.com/java-loadLibrary-Linux.asp and here https://docs.cask.co/cdap/6.0.0/en/developer-manual/building-blocks/program-lifecycle.html but I can't quite find what I'm looking for. Ideally I need the class to be loaded once and once only when first used by any specific UDD (or maybe loaded by CDAP at start-up time). Is this possible?

Here's the error:

Thanks in advace,
Anat

--
You received this message because you are subscribed to the Google Groups "CDAP Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-discuss...@google.com.
To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/cdap-discuss/CANdBRBYYiRu_ud1998QzL7eJGh9rD3hvTQvb0Leh0Qr%3DPDAVXg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "cdap-team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-team+...@google.com.
To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/cdap-team/CANdBRBYYiRu_ud1998QzL7eJGh9rD3hvTQvb0Leh0Qr%3DPDAVXg%40mail.gmail.com.

--
Terence Yim | Staff Software Engineer | tere...@google.com |

--
You received this message because you are subscribed to the Google Groups "CDAP Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-discuss...@google.com.
To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/cdap-discuss/c965b26b-33a2-49ef-ace9-a5f6b4c7dd49%40google.com.

--
You received this message because you are subscribed to the Google Groups "cdap-team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-team+...@google.com.
To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/cdap-team/c965b26b-33a2-49ef-ace9-a5f6b4c7dd49%40google.com.

Reply all

Reply to author

Forward