kafka connect with sqlserver source

[richter.b...@gmail.com]

Hello i really need help with Kafka Connect. Yesterday I tried the file and sqlite3 example and they worked great. Then I tried my acctual goal connecting Kafka with a SQLServer.

I run Kafka and the connector within a container everything in one just for testing purposes.

CLASSPATH is set to:

root@68367e9b813f:/home# echo $CLASSPATH

/usr/share/java/kafka-connect-jdbc/sqljdbc4-4.1.jar

Programms are running:

root@68367e9b813f:/home# supervisorctl

kafka                            RUNNING    pid 42, uptime 0:56:12

kafka-rest                       RUNNING    pid 183, uptime 0:56:08

schema-registry                  RUNNING    pid 176, uptime 0:56:08

zookeeper                        RUNNING    pid 41, uptime 0:56:12

Everything I tried sofar failed. If I understand the log correctly I guess there is an error in my .properties file ( there are placeholders included < > )

name=test-sqlserver-jdbc-autoincrement

connector.class=io.confluent.connect.jdbc.JdbcSourceConnector

tasks.max=1

connection.url=jdbc:sqlserver://<ip>\\<instance>;database=<dbname>;user=<user>;password=<password>

query=select da.id , ...<EXTREME LONG QUERY>

mode=incrementing

incrementing.column.name=da.id

topic.prefix=test-sqlserver-jdbc-

The error says: 

[2017-04-05 05:45:06,304] ERROR Failed to create job for sqlserver-source.properties (org.apache.kafka.connect.cli.ConnectStandalone:88)

[2017-04-05 05:45:06,305] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:99)

I checked thosed lines it is a loop that iterates through the properties. But in any case I give you the whole log here:

root@68367e9b813f:/home# connect-standalone /etc/schema-registry/connect-avro-standalone.properties sqlserver-source.properties

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-elasticsearch/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/share/java/kafka-connect-hdfs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/share/java/kafka/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

[2017-04-05 05:54:45,974] INFO StandaloneConfig values:

        access.control.allow.methods =

        access.control.allow.origin =

        bootstrap.servers = [localhost:9092]

        internal.key.converter = class org.apache.kafka.connect.json.JsonConverter

        internal.value.converter = class org.apache.kafka.connect.json.JsonConverter

        key.converter = class io.confluent.connect.avro.AvroConverter

        offset.flush.interval.ms = 60000

        offset.flush.timeout.ms = 5000

        offset.storage.file.filename = /tmp/connect.offsets

        rest.advertised.host.name = null

        rest.advertised.port = null

        rest.host.name = null

        rest.port = 8083

        task.shutdown.graceful.timeout.ms = 5000

        value.converter = class io.confluent.connect.avro.AvroConverter

 (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:180)

[2017-04-05 05:54:46,140] INFO Logging initialized @738ms (org.eclipse.jetty.util.log:186)

[2017-04-05 05:54:46,455] INFO AvroConverterConfig values:

        schema.registry.url = [http://localhost:8081]

        max.schemas.per.subject = 1000

 (io.confluent.connect.avro.AvroConverterConfig:169)

[2017-04-05 05:54:46,677] INFO AvroDataConfig values:

        schemas.cache.config = 1000

        enhanced.avro.schema.support = false

        connect.meta.data = true

 (io.confluent.connect.avro.AvroDataConfig:169)

[2017-04-05 05:54:46,680] INFO AvroConverterConfig values:

        schema.registry.url = [http://localhost:8081]

        max.schemas.per.subject = 1000

 (io.confluent.connect.avro.AvroConverterConfig:169)

[2017-04-05 05:54:46,681] INFO AvroDataConfig values:

        schemas.cache.config = 1000

        enhanced.avro.schema.support = false

        connect.meta.data = true

 (io.confluent.connect.avro.AvroDataConfig:169)

[2017-04-05 05:54:46,697] INFO Kafka Connect starting (org.apache.kafka.connect.runtime.Connect:50)

[2017-04-05 05:54:46,697] INFO Herder starting (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:72)

[2017-04-05 05:54:46,697] INFO Worker starting (org.apache.kafka.connect.runtime.Worker:119)

[2017-04-05 05:54:46,698] INFO Starting FileOffsetBackingStore with file /tmp/connect.offsets (org.apache.kafka.connect.storage.FileOffsetBackingStore:60)

[2017-04-05 05:54:46,780] INFO Worker started (org.apache.kafka.connect.runtime.Worker:124)

[2017-04-05 05:54:46,781] INFO Herder started (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:74)

[2017-04-05 05:54:46,782] INFO Starting REST server (org.apache.kafka.connect.runtime.rest.RestServer:98)

[2017-04-05 05:54:46,936] INFO jetty-9.2.15.v20160210 (org.eclipse.jetty.server.Server:327)

Apr 05, 2017 5:54:48 AM org.glassfish.jersey.internal.Errors logErrors

WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.

WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.

WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.

WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

[2017-04-05 05:54:48,333] INFO Started o.e.j.s.ServletContextHandler@466cf502{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744)

[2017-04-05 05:54:48,361] INFO Started ServerConnector@8bef923{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:266)

[2017-04-05 05:54:48,365] INFO Started @2965ms (org.eclipse.jetty.server.Server:379)

[2017-04-05 05:54:48,366] INFO REST server listening at http://172.17.0.4:8083/, advertising URL http://172.17.0.4:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:150)

[2017-04-05 05:54:48,367] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:56)

[2017-04-05 05:55:00,977] INFO Reflections took 14065 ms to scan 271 urls, producing 12642 keys and 83594 values  (org.reflections.Reflections:229)

[2017-04-05 05:55:17,875] ERROR Failed to create job for sqlserver-source.properties (org.apache.kafka.connect.cli.ConnectStandalone:88)

[2017-04-05 05:55:17,878] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:99)

java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid (use the endpoint `/{connectorType}/config/validate` to get a full list of errors)

        at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:80)

        at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:67)

        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:96)

Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid (use the endpoint `/{connectorType}/config/validate` to get a full list of errors)

        at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:161)

        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:93)

[2017-04-05 05:55:17,889] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:66)

[2017-04-05 05:55:17,890] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:154)

[2017-04-05 05:55:17,903] INFO Stopped ServerConnector@8bef923{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:306)

[2017-04-05 05:55:17,927] INFO Stopped o.e.j.s.ServletContextHandler@466cf502{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:865)

[2017-04-05 05:55:17,932] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:165)

[2017-04-05 05:55:17,932] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:78)

[2017-04-05 05:55:17,933] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:131)

[2017-04-05 05:55:17,935] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:68)

[2017-04-05 05:55:17,935] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:151)

[2017-04-05 05:55:17,936] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:88)

[2017-04-05 05:55:17,937] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:71)

[Robin Moffatt]

Does it work if you specify a table instead of a query, using table.whitelist=? 

Could be worth a try, to rule out the "extreme long query" from somehow causing a problem :)

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/28436d07-2bb6-4d97-8f1d-c71a06b3899c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[richter.b...@gmail.com]

Robin Moffatt you made my day!! that was it!

my college just finished the table and it worked. Thanks a lot