How to restart clickhouse server with kafka engine tables gracefully

537 views
Skip to first unread message

kriticar

unread,
May 6, 2019, 12:32:24 PM5/6/19
to ClickHouse
Hi,


is the following example:

  CREATE TABLE queue (
    timestamp UInt64,
    level String,
    message String
  ) ENGINE = Kafka('localhost:9092', 'topic', 'group1', 'JSONEachRow');

  CREATE TABLE daily (
    day Date,
    level String,
    total UInt64
  ) ENGINE = SummingMergeTree(day, (day, level), 8192);

  CREATE MATERIALIZED VIEW consumer TO daily
    AS SELECT toDate(toDateTime(timestamp)) AS day, level, count() as total
    FROM queue GROUP BY day, level;

  SELECT level, sum(total) FROM daily GROUP BY level;

To stop receiving topic data or to change the conversion logic, detach the materialized view:

  DETACH TABLE consumer;
  ATTACH MATERIALIZED VIEW consumer; (it doesn't work, ATTACH TABLE consumer works, but it is not necessary if server is restarted, because consumer is automatically attached). 


We have tried to deatach consumer table before restarting clickhouse-server, but after clickhouse-server is up again, we have noticed that we have lost some of the messages.
We also tried to deatach consumer and queue tables, but the problem still remains.

What would bi the best way to restart clickhouse-server node and not to lose any kafka messages?
Do we have to set up something on clickhouse side? Kafka side? Or both?

Regards.

Reply all
Reply to author
Forward
0 new messages