We are trying to monitor the Kafka Topics and use the data to drive Analytical dashboards.
We can create the tables in clickhouse using the Kafka Engine or use the JDBC Driver and send the event info to Clickhouse using either Custom Kafka Consumer or Apache NiFi etc.
What are the downsides of using the Kafka Engine. We want to monitor any topic created on the Kafka Cluster and the no.of topics are only going to grow. How does that impact the Clickhouse server/Cluster if we use the Kafka Engine. If we don't use Kafka Engine then the retrieval of info from kafka is externalized and hence no impact on the Cluster.
If we use the JDBC Driver and send the data in batches what are the optional batch sizes.
Are there any best practices w.r.to