Disk space used by Distributed table in ReplicatedMergeTree

[Matt K]

I have a 4 node cluster running v1.1.54327-stable using Distributed and ReplicatedMergeTree tables, with a simple insert loading data into the Distributed table:

CREATE TABLE test_shard ( ...)

ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/cds', '{replica}', date, (pk), 8192);

CREATE TABLE test (shard_key Int64, ...)

ENGINE = Distributed(test_cluster, testdb, test_shard, shard_key);

Everything appears to be working as expected except there is a significant amount of disk space consumed by the Distributed table on the node accepting the inserts: 29GB for the Distributed table when each shard table has 21GB disk used. This is many hours after a batch load completed, with no new inserts during that period.

Should the Distributed table be consuming disk space?

[Alex Zatelepin]

Hi Matt,

When the directory of a Distributed table consumes significant amount of disk space in most cases it means that there is some problem with inserting data to remote nodes (INSERTSs into a Distributed table are async by default and the inserted data is first written to disk and only then sent to remote nodes). Please check server logs for errors containing the string "DirectoryMonitor".

[Matt K]

It looks like the .bin files left in directory for the Distributed table are not getting touched at all?

Picking some at random with a file time of 0000 GMT today (19 hours ago), 

    -rw-r----- 1 root root 205332 Jan 12 00:00 6350.bin

they do not appear as processed in the log:

    grep '/6350.bin`' clickhouse-server.log

... returns nothing for the current date/

When that same filename was part of a previous load yesterday, I see it being processed:

    2018.01.11 19:27:43.243860 [ 107 ] <Trace> cds.Distributed.DirectoryMonitor: Started processing `/localdata2/ch/data/logstore/cds/default@209%2E197%2E7%2E135:9000/6350.bin`

    2018.01.11 19:27:43.516764 [ 107 ] <Trace> cds.Distributed.DirectoryMonitor: Finished processing `/localdata2/ch/data/logstore/cds/default@209%2E197%2E7%2E135:9000/6350.bin`

Is there anything else I can look at to diagnose?

[Matt K]

It may be that these bin files are getting loaded into the shards based on row count in the table, but the backlog continues to grow. Opened https://github.com/yandex/ClickHouse/issues/1758