Migrate Kafka Topic to new Cluster (and impact on Druid).

[Mrugesh Kadia]

Hi All,

I am ingesting data into Druid from Kafka's topic. Now I want to migrate my Kafka Topic to the new Kafka Cluster. What are the possible ways to do this without duplication of data and without downtime?

I have considered below possible ways to migrate Topic to the new Kafka Cluster.

1. Manually Migration:
   
   • Create a topic with the same configuration in the new Kafka cluster.
     
   • Stop pushing data in the Kafka cluster.
   • Start pushing data in the new cluster.
   • Stop consuming from the old cluster.
   • Start consuming from the new cluster.
2. Produce data in both Kafka clusters:
   • Create a topic with the same configuration in the new Kafka cluster.
   • Start producing messages in both Kafka clusters.
   • Change Kafka topic configration in Druid.
   • Reset Kafka topic offset in Druid.
   • Start consuming from the new cluster.
   • after successful migration, stop producing in the old Kafka cluster.
3. Use Mirror Maker 2:
   • MM2 creates Kafka's topic in a new cluster.
   • Start replicating data in both clusters.
   • Move producer and consumer to the new Kafka cluster.
   • The problem with this approach:
   • 1. Druid manages Kafka topic's offset in its metadata.
     
   • 2. MM2 will create two topics with the same name(with prefix) in the new cluster.
     
   • 3. Does druid support the topic name with regex?

Druid Version: 0.22.1

Old Kafka Cluster Version: 2.0

[Mark Herrera]

Maybe a slight modification of your number 1:

1) Wait for the current supervisor to catch up all the data in the old topic.
2) Suspend the supervisor. This will force all the tasks to write and publish the segments. Wait for all the tasks for this supervisor to succeed.
3) All the offset information of the current datasource are stored in Metadata Storage. Delete those records.
4) Optionally terminate the current supervisor.

Now on to the new topic and new server:

5) If you have done step 4 - submit the new spec with the new topic and new server information.
6) If you have not done step 4 - open the supervisor spec in the data loader and change the topic and server information, then submit it again.

Let us know what you end up doing and how it goes.

[Mrugesh Kadia]

Hello mark,

Can we Hard reset Supervisor after suspending it, instead of deleting data manually from metadata storage?

as per documentation hard reset will clears stored offsets.

[Mark Herrera]

Apologies for my late reply, but I was offsite. You can, but be aware of this warning from the same doc:

Use care when using this operation! Resetting the supervisor may cause Kafka messages to be skipped or read twice, resulting in missing or duplicate data.

Maybe set up a test cluster to try it out first?

[Renato Cron]

Hello,

If you follow all steps, 1, 2 on the 3th:

3) All the offset information of the current datasource are stored in Metadata Storage. Delete those records.

I can't see what's the difference between issuing a hard reset and delete from druid_pendingsegments (is this the correct table?)

I suspended the task, waited from PENDING to SUSPEND, and then did a hard-reset and resume the job (I saw no change on the pendingsegments table)

when I resumed the job, the supervisor started the task with:

  "ioConfig": {
    "type": "kafka",
    "taskGroupId": 0,
    "baseSequenceName": "index_kafka_GL_NETWORK_STATUS_ROLLUP_7ff3643011c9174",
    "startPartitions": {
      "type": "end",
      "stream": "GL_NETWORK_STATUS",
      "topic": "GL_NETWORK_STATUS",
      "partitionSequenceNumberMap": {
        "0": 4962
      },
      "partitionOffsetMap": {
        "0": 4962
      }
    },

The 4962 offset is the easiest offset I had on the stream, I think that's the desired output if I had changed the topic between those runs, and would not lead to duplicates