Clickhouse just silently died

[Саги Усаги]

Idea was to store netflow(ipfix) data in clickhouse.

I have a table:

CREATE TABLE IF NOT EXISTS default.netflow (
      FlowDate Date DEFAULT toDate(FlowDateTime),
     FlowDateTime DateTime,
     SourceIPV4 UInt32,
     DestinationIPV4 UInt32,
     SourcePort UInt16,
     DestinationPort UInt16,
     tcpControlBits UInt16,
     tcpOptions UInt64,
     protocolIdentifier UInt8,
     octetDeltaCount UInt64,
     packetDeltaCount UInt64
) ENGINE=MergeTree(FlowDate, (FlowDateTime, SourceIPV4, DestinationIPV4, SourcePort, DestinationPort), 8192);

It was working great with small amount of netflow data, however when we tried it with production data sizes/rates clickhouse just died

It suddenly stopped listening on ports, while still running.

It wont stop (only by SIGKILL), and when started again behaves the same.

And there's nothing suspicious in logs.

Also, there's over 6M files in data dir and, imho, that's a really big number of small files.

I'm pretty sure i'm doing something wrong (keys? granularity?)

[Саги Усаги]

Okay, looks like clickhouse was killed by OOM, so we bumped testbench specs to 8 cores and 16Gb RAM

Sadly, it still can't start (well, it does, but it spends tons of time loading data)

With empty database it seems to work (well, better than before), but looks like it fails to catch up with background merges, so it still creates tons of directories, and it's only a matter of time before it fails again.

Given that, what are the minimum requirements for running clickhouse?

[Alex Zatelepin]

Minimum system requirements are highly dependent on table definitions, stored data and typical queries. Bare minimum RAM requirements are around 8Gb.

How did you insert your data? Because a single INSERT creates a new part (each part corresponds to a single data directory), it is better to do INSERTs in big batches. Anyway, normally there shouldn't be that many directories because when the number of parts per partition (per month) reaches 150, the server starts throttling INSERTs, and when it reaches 300, the server forbids INSERTs.

[Саги Усаги]

Yep, the problem was in single inserts. With batch inserts it seems to work relatively fine (relatively, because there's still ~25k parts, but it doesn't seem to be throttling of forbidding inserts)

Another problem, however, remains - i'm sometimes getting empty reply from clickhouse (e.g. unexpected EOF)

[Alex Zatelepin]

i'm sometimes getting empty reply from clickhouse (e.g. unexpected EOF)

Could you provide more details? What is are the queries, what happens in server logs etc.

[Саги Усаги]

Well, i'm getting Post http://localhost:8123/: EOF with go-clickhouse (sadly, it's not very informative)

There's nothing in error log, and nothing suspicious in server log. Clickhouse continues to work normal (and better compressing and merging parts, obviously)

Here's last log entry before client died:

2017.06.15 04:28:04.634546 [ 369 ] <Debug> executeQuery: (from [::1]:46674) INSERT INTO default.netflow (FlowDateTime,SourceIPV4,DestinationIPV4,SourcePort,DestinationPort,tcpControlBits,tcpOptions,protocolIdentifier,octetDeltaCount,packetDeltaCount) VALUES

Guess i can just gracefully handle such errors on the client side.