CREATE TABLE queue ( timestamp UInt64, level String, message String ) ENGINE = Kafka('localhost:9092', 'topic', 'group1', 'JSONEachRow'); CREATE TABLE daily ( day Date, level String, total UInt64 ) ENGINE = SummingMergeTree(day, (day, level), 8192); CREATE MATERIALIZED VIEW consumer TO daily AS SELECT toDate(toDateTime(timestamp)) AS day, level, count() as total FROM queue GROUP BY day, level; SELECT level, sum(total) FROM daily GROUP BY level;
To stop receiving topic data or to change the conversion logic, detach the materialized view:
DETACH TABLE consumer; ATTACH MATERIALIZED VIEW consumer; (it doesn't work, ATTACH TABLE consumer works, but it is not necessary if server is restarted, because consumer is automatically attached).
We have tried to deatach consumer table before restarting clickhouse-server, but after clickhouse-server is up again, we have noticed that we have lost some of the messages.
We also tried to deatach consumer and queue tables, but the problem still remains.
What would bi the best way to restart clickhouse-server node and not to lose any kafka messages?
Do we have to set up something on clickhouse side? Kafka side? Or both?
Regards.