connector task fails falsely complaining about missing db history topic

[suda...@confluent.io]

Below outputs shows, 

a)connect errors complaining about missing history topic

b) outputs of the history topic existing with infinite retention 

c) outputs of the config

I am not able to recover until I remove all topics created by the connector.

Originally used 0.7.2, currently using debezium-connector-mysql-0.7.5.jar.

In a separate cycle I had used SCHEMA_ONLY_RECOVERY as snapshotting mode too.

Would like to know if there are known workarounds for this issue.

curl -s 127.0.0.1:8083/connectors/mysql-debez-connector/status | jq

{

  "name": "mysql-debez-connector",

  "connector": {

    "state": "RUNNING",

    "worker_id": "172.31.4.138:8083"

  },

  "tasks": [

    {

      "state": "FAILED",

      "trace": "org.apache.kafka.connect.errors.ConnectException: The db history topic is missing. You may attempt to recover it by reconfiguring the connector to SCHEMA_ONLY_RECOVERY\n\tat io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:95)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:164)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n",

      "id": 0,

      "worker_id": "xxxxxxxxx:8083"

    }

  ],

  "type": "source"

kafka-topics --zookeeper ip-xx-xx-xx-xx.us-east-2.compute.internal:2181 --describe --topic dbhistory.demo

Topic:dbhistory.demo PartitionCount:1 ReplicationFactor:3 Configs:retention.bytes=-1

Topic: dbhistory.demo Partition: 0 Leader: 2 Replicas: 2,1,0 Isr: 0,1,2

curl -s 127.0.0.1:8083/connectors/mysql-debez-connector | jq

[2018-06-14 00:04:10,894] INFO 127.0.0.1 - - [14/Jun/2018:00:04:10 +0000] "GET /connectors/mysql-debez-connector HTTP/1.1" 200 583  3 (org.apache.kafka.connect.runtime.rest.RestServer:60)

{

  "name": "mysql-debez-connector",

  "config": {

    "connector.class": "io.debezium.connector.mysql.MySqlConnector",

    "database.user": "user",

    "database.server.id": "223344",

    "database.hostname": "xxxxxxxxxxxxxxx.us-east-2.compute.internal",

    "database.password": "user123",

    "database.history.kafka.bootstrap.servers": "xxxxxxxxxxxxxxxxx.us-east-2.compute.internal:9092",

    "database.history.kafka.topic": "dbhistory.demo",

    "name": "mysql-debez-connector",

    "database.server.name": "demo",

    "database.port": "3306",

    "include.schema.changes": "true"

  },

  "tasks": [

    {

      "connector": "mysql-debez-connector",

      "task": 0

    }

  ],

  "type": "source"

[Jiri Pechanec]

Hi,

is there anything in the history topic?

J.

[suda...@confluent.io]

Yes. Few records. Schema has not changed.

Thanks

[Jiri Pechanec]

And what are your time-based retention settings?

J.

[Sudarshan Pathalam]

I have set retention.bytes=-1 expecting this will translate  to time based retention of -1.

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+unsubscribe@googlegroups.com.
To post to this group, send email to debe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/37f290c1-78a2-4b2b-a6e4-0c48f25749da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jiri Pechanec]

AFAIK, the default retention is 7 days.

We are using these settings

retention.ms=9223372036854775807,cleanup.policy=delete,retention.bytes=-1

J.

To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

[Sudarshan Pathalam]

I see that you have set both "retention.ms" and retention.bytes to a high number (clean up policy is default) That might explain the periodicity of the issue. Thanks. Appreciate it.

To unsubscribe from this group and stop receiving emails from it, send an email to debezium+unsubscribe@googlegroups.com.

To post to this group, send email to debe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/1d76168a-953f-4a25-a264-b9d9634b0a05%40googlegroups.com.