CDAP 4.3.4 - Different version of javax.ws.rs.core.MultivaluedMap class in cdap lib causing java.lang.NoSuchMethodError

[Lav Mudgal]

Hi,

I am getting below exception while Running my custom plugin.

2018-10-18 12:13:07,654 - WARN  [Thread-48:o.a.h.m.LocalJobRunner@587] - job_local1835827613_0001
java.lang.Exception: java.lang.NoSuchMethodError: javax.ws.rs.core.MultivaluedMap.addAll(Ljava/lang/Object;[Ljava/lang/Object;)V
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489) ~[org.apache.hadoop.hadoop-mapreduce-client-common-2.8.0.jar:na]
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549) ~[org.apache.hadoop.hadoop-mapreduce-client-common-2.8.0.jar:na]
java.lang.NoSuchMethodError: javax.ws.rs.core.MultivaluedMap.addAll(Ljava/lang/Object;[Ljava/lang/Object;)V
	at org.glassfish.jersey.client.ClientRequest.accept(ClientRequest.java:326) ~[jersey-client-2.17.jar:na]
	at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:229) ~[jersey-client-2.17.jar:na]
	at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:59) ~[jersey-client-2.17.jar:na]
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient$15.run(SchemaRegistryClient.java:1070) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient$15.run(SchemaRegistryClient.java:1067) ~[schema-registry-client-0.5.1.jar:na]
	at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_31]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[na:1.8.0_31]
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.postEntity(SchemaRegistryClient.java:1067) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.doRegisterSchemaMetadata(SchemaRegistryClient.java:410) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.addSchemaMetadata(SchemaRegistryClient.java:393) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.registerSchemaMetadata(SchemaRegistryClient.java:385) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.addSchemaVersion(SchemaRegistryClient.java:438) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.addSchemaVersion(SchemaRegistryClient.java:426) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.serde.AbstractSnapshotSerializer.serialize(AbstractSnapshotSerializer.java:64) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.serdes.avro.kafka.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:86) ~[schema-registry-serdes-0.5.1.jar:na]
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:463) ~[kafka-clients-0.10.2.0.jar:na]
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:440) ~[kafka-clients-0.10.2.0.jar:na]
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:360) ~[kafka-clients-0.10.2.0.jar:na]
	at co.cask.hydrator.plugin.sink.KafkaRecordWriter.sendMessage(KafkaRecordWriter.java:44) ~[1539844977953-0/:na]
	at co.cask.hydrator.plugin.sink.KafkaRecordWriter.write(KafkaRecordWriter.java:55) ~[1539844977953-0/:na]
	at co.cask.hydrator.plugin.sink.KafkaRecordWriter.write(KafkaRecordWriter.java:32) ~[1539844977953-0/:na]
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:658) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.8.0.jar:na]
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.8.0.jar:na]

On closer inspection, I found that this class is coming from 4 different sources.

1. {cdaphomefolder}/lib/com.sun.jersey.jersey-core-1.9.jar

2.{cdaphomefolder}/lib/javax.ws.rs.javax.ws.rs-api-2.0.jar

3.{cdaphomefolder}/lib/org.jboss.resteasy.jaxrs-api-3.0.8.Final.jar

4. My custom plugin assembly jar (javax.ws.rs-api-2.0.1.jar is one of the jar in my assembly jar)

MultivaluedMap class of 2,3 and 4th jar has the addAll method (javax.ws.rs.core.MultivaluedMap.addAll) but not present in {cdaphomefolder}/lib/com.sun.jersey.jersey-core-1.9.jar.

Removing jar com.sun.jersey.jersey-core-1.9.jar and restarting cdap resolves this issue for me on my local sandbox.

Please advice how to proceed from here and resolve this issue. Is there a way to add my jars before cdap jars in the class-path ?

--

Thanks,

Lav Mudgal

[Nitin Motgi]

You should include it with your plugin and expose it in the Felix configuration ( 

E.g. https://github.com/data-integrations/wrangler/blob/develop/wrangler-transform/pom.xml )

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To post to this group, send email to cdap...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAD8dLjwmCOzzQnusQG7XSZf-_j-ewxVeo2BCnyVYmgYOLmcoYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Nitin Motgi

"We reap what we sow. We are the makers of
our own fate. The wind is blowing; those
vessels whose sails are unfurled catch it, and
go forward on their way, but those which have
their sails furled do not catch the wind. Is that
the fault of the wind?....... We make our own
destiny.”
                -- Swami Vivekananda

[Lav Mudgal]

Hi Nitin,

My pom already has a entry Felix configuration.

<dependency>
            <groupId>javax.ws.rs</groupId>
            <artifactId>javax.ws.rs-api</artifactId>
            <version>2.0.1</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <version>3.3.0</version>
                <configuration>
                    <instructions>
                        <_exportcontents>co.cask.hydrator.plugin.*;org.apache.spark.streaming.kafka010.*;
                            org.apache.kafka.*;com.google.common.base.*;kafka.serializer.*;
                            org.glassfish.jersey.*;com.hortonworks.*;javax.ws.*;org.apache.avro.*;
                        </_exportcontents>
                        <Embed-Dependency>*;inline=false;scope=compile</Embed-Dependency>
                        <Embed-Transitive>true</Embed-Transitive>
                        <Embed-Directory>lib</Embed-Directory>
                    </instructions>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>bundle</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

Thanks,

Lav Mudgal

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAAZMJ_WbYn7z4rYGzja50EqfkMHj3G34OOZk3K%2BacJpu9ddbqA%40mail.gmail.com.

[Albert Shau]

Hi Lav,

CDAP provides classloader isolation for most classes, to only expose the classes in the CDAP API. The javax.ws.rs classes are part of the CDAP API because of the HTTP service handler annotations for GET, PUT, etc. So certain classes might always come from the system and we would have to see if that's the case here.

What type of plugin are you creating? Is it a Kafka source that uses Hortonworks' schema registry? You should only be exporting the packages that your plugin lives in and those the framework needs in order to be able to instantiate the Spark source. In other words, I'm guessing that most of the last line of exports ('org.glassfish.jersey.*;javax.ws.*;org.apache.avro.*;) isn't required. Can you try removing those and see if that changes anything?

Regards,

Albert

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAD8dLjzV2O6vWF3qAfrARLV34Rja-uKO8DUyeHqV6tFuYdMFFw%40mail.gmail.com.

[Lav Mudgal]

Hi Albert,

Thanks for the reply. Yes we are trying to add Hortonworks' schema registry support in kafka plugin. Schema registry code internally uses org.glassfish.jersey.client.ClientConfig class which is not present in cdap-sandbox-4.3.4/lib/com.sun.jersey.jersey-client-1.9.jar. Below is the exception trace

Caused by: java.lang.NoClassDefFoundError: org/glassfish/jersey/client/ClientConfig
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.createClientConfig(SchemaRegistryClient.java:364) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.<init>(SchemaRegistryClient.java:207) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.serde.AbstractSerDes.init(AbstractSerDes.java:54) ~[schema-registry-client-0.5.1.jar:na]
	at com.hortonworks.registries.schemaregistry.serdes.avro.kafka.KafkaAvroSerializer.configure(KafkaAvroSerializer.java:73) ~[schema-registry-serdes-0.5.1.jar:na]

Thanks,

Lav Mudgal

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAOtBAgiw3rwdiiRekAWytn6DxfvN2cukWYrKJ9pxve%2BmjADxUA%40mail.gmail.com.

[Albert Shau]

Hi Lav,

I see. Can you try going back to your previous exports but instead of javax.ws.* try javax.ws.core.*? If that doesn't work I think the CDAP platform would need to be modified in order to prevent these classloading issues with javax.ws classes.

Another workaround would be to use the maven shade plugin to rename the relevant packages names to prevent clashes. The shade plugin doesn't work well with the maven-bundle-plugin, so you would have to replace maven-bundle-plugin with shade and use its ManifestResourceTransformer (http://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ManifestResourceTransformer) to create a Manifest that exports the relevant packages. Unfortunately that doesn't support the '*' syntax so you would have to list all of them out. (See https://stackoverflow.com/questions/18280608/using-maven-bundle-plugin-with-the-maven-shade-plugin for more info about that).

Best,

Albert

To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CAD8dLjz6Sozeoz1svLduwScisYUU6mS%3DzjUZsE1qMvDEzHLRDA%40mail.gmail.com.