Getting java.lang.NoSuchMethodError while starting connectors in distributed mode.

[Renukaradhya H D]

Hi,

I am using 3.2.0 version of Confluent Platform.

I am running the worker in distributed mode and I started my connector using REST Service and i got below error.

Is this the issue with some jar versions?

Logs:

2017-03-10 05:56:40,372] WARN  (org.eclipse.jetty.servlet.ServletHandler:620)

javax.servlet.ServletException: java.lang.NullPointerException

        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)

        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)

        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)

        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)

        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.Server.handle(Server.java:499)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.NullPointerException

        at org.apache.kafka.connect.connector.Connector.validate(Connector.java:134)

        at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:252)

        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:510)

        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$6.call(DistributedHerder.java:507)

        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:250)

        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:200)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        ... 1 more

[2017-03-10 05:56:40,375] WARN /connectors (org.eclipse.jetty.server.HttpChannel:384)

java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:684)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.Server.handle(Server.java:499)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:745)

[2017-03-10 05:56:40,377] WARN Could not send response error 500: java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z (org.eclipse.jetty.server.HttpChannel:482)

I am using below jars:

                        <dependency>

<groupId>org.apache.kafka</groupId>

<artifactId>connect-api</artifactId>

                                                <version>0.10.2.0</version>

<scope>compile</scope>

</dependency>

<dependency>

<groupId>org.apache.avro</groupId>

<artifactId>avro</artifactId>

<version>1.7.7</version>

</dependency>

<dependency>

<groupId>org.elasticsearch</groupId>

<artifactId>elasticsearch</artifactId>

<version>5.2.1</version>

</dependency>

<dependency>

<groupId>org.elasticsearch.client</groupId>

<artifactId>transport</artifactId>

<version>5.2.1</version>

</dependency>

<dependency>

<groupId>com.fasterxml.jackson.core</groupId>

<artifactId>jackson-core</artifactId>

<version>2.6.3</version>

</dependency>

<dependency>

<groupId>com.fasterxml.jackson.core</groupId>

<artifactId>jackson-databind</artifactId>

<version>2.6.3</version>

<exclusions>

<exclusion>

<artifactId>com.fasterxml.jackson.core</artifactId>

<groupId>jackson-core</groupId>

</exclusion>

</exclusions>

</dependency>

<dependency>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>4.12</version>

</dependency>

<dependency>

<groupId>org.elasticsearch.test</groupId>

<artifactId>framework</artifactId>

<version>5.2.1</version>

<scope>test</scope>

</dependency>

<dependency>

<groupId>com.google.guava</groupId>

<artifactId>guava</artifactId>

<version>19.0</version>

</dependency>

<dependency>

<groupId>org.immutables</groupId>

<artifactId>value</artifactId>

<version>2.3.9</version>

<scope>provided</scope>

</dependency>

                               <dependency>

<groupId>org.slf4j</groupId>

<artifactId>slf4j-api</artifactId>

<version>1.7.21</version>

<scope>compile</scope>

</dependency>

<dependency>

<groupId>org.slf4j</groupId>

<artifactId>slf4j-log4j12</artifactId>

<version>1.7.21</version>

</dependency>

                                <dependency>

<groupId>commons-lang</groupId>

<artifactId>commons-lang</artifactId>

<version>2.6</version>

</dependency>

    

[Konstantine Karantasis]

This is caused by conflicting dependencies and jetty packages are a common source of such conflicts.

Things you can try for now:
- Reorder packages in your CLASSPATH.
- Figure out the version of certain dependencies in Kafka (and Kafka Connect in specific) and try to peg your connector's dependency versions to match the ones in Connect. (A few I can quickly see are guava, for which connect pulls version 18.0 and jackson which in Connect is 2.8.5. But these are just examples of version mismatches. I'm not sure which of your packages is pulling an incompatible jetty version). 

A way to figure out the dependencies pulled by Connect is to run for instance:

  $ ./gradlew connect:runtime:dependencies

(or substitute "connect:runtime" to find deps for another module)

To view your connectors dependencies you would run something like (I understand you use maven by looking the xml above):

  $ mvn dependency:tree -Dverbose

Note: Class loading isolation between the framework and the connectors is an issue that's expected to be addressed in one of the upcoming versions of Kafka Connect. 

Konstantine

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f6ce84d4-536a-4763-947e-348d38c4ee3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Renukaradhya H D]

Thanks for the guidance Konstantine.

I will try and let you know the results.

Konstantine

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f6ce84d4-536a-4763-947e-348d38c4ee3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/KO4Q89zxCxI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CABqDk2MUuYLvg9R5UVaoVc_omts8SSqH%2BNGa3L90r23cJ_Mqkg%40mail.gmail.com.

[Renukaradhya H D]

Hi Konstantine,

I am still struggling to resolve this.

• I removed all maven dependency and included only these below 3 jars. My code now contains only Sinkconnector and Sinktask classes and i removed all other business functionalities. Still the issue is seen.
  

                         <dependency>

<groupId>org.apache.logging.log4j</groupId>

<artifactId>log4j-api</artifactId>

<version>2.6.2</version>

<scope>compile</scope>

</dependency>

<dependency>

<groupId>org.apache.kafka</groupId>

<artifactId>connect-api</artifactId>

<version>0.10.2.0</version>

<scope>compile</scope>

</dependency>

<dependency>

<groupId>org.apache.avro</groupId>

<artifactId>avro</artifactId>

<version>1.7.7</version>

<scope>compile</scope>

</dependency>

      The way i am starting the connector is below :

   1) KAFKA_HEAP_OPTS="-Xmx4G" sh $KAFKA_SERVICE_HOME/connect-distributed $KAFKA_CONFIG_HOME/connect-avro-distributed.properties -> this is to start the worker.

  2) Fired a POST call (http://xyz.com:8083/connectors) with the required configuration.

Is there any thing that I am doing wrong? Since with just Sinkconnector and SinkTask classes I am still not able to start the connector.

Regards,

Aradhya

Konstantine

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f6ce84d4-536a-4763-947e-348d38c4ee3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shannon Ma]

I am having the same issue. I am not sure about the jars, this is a clean vm, we only installed java and kafka related.

java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z

i can post the sample local-file-sink connector, but not jdbc connector.

Thanks
Shannon

Konstantine

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/f6ce84d4-536a-4763-947e-348d38c4ee3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/KO4Q89zxCxI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

[Shannon Ma]

there are different versions of serverlet-api/jetty/jersey under rest-utils and /kafka-connect-hdfs and /kafka, for the connect rest api, which version of servlet/jetty/jersey should we use?

[Shannon Ma]

After i remove the servlet-api-2.5.jar from  kafka-connect-hdfs, it passed that error, now 

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token

 at [Source: org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream@7b2ea8db; line: 1, column: 26] (through reference chain: java.util.LinkedHashMap["config"])

        at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148)

is this the right format?

{"name": "traffic-source", 

 "config":  {"connector.class":"JdbcSourceConnector", 

  "tasks.max":"1", 

  "topic.prefix":"traffic_input",

  "connection.url": "jdbc:oracle:thin:xxx",

  "query" : "SELECTxxx",

  "mode": "timestamp+incrementing",

  "timestamp.column.name" : "xxx",

  "incrementing.column.name" : "xxx"  }

[Shannon Ma]

Weird it errors using postman, no error using curl but it times out, and after it times out any call will hang.

Not sure is it only happening in 3.2?

[Shannon Ma]

If call  connector-plugins/jdbcSourceConnector/config/validate, from curl i am getting the same

 Can not deserialize instance of java.lang.String out of START_OBJECT token

error

it is weird that if i put the wrong url (wrong port or db name) i get config error right away (and ask me to use validate to get more detailed error). With the correct url it just hangs.

[Shannon Ma]

Got it working once i added/used the new configs http://docs.confluent.io/3.2.0/connect/connect-jdbc/docs/source_config_options.html.