Clickhouse Memory Issue

[rs_india]

Dear All,

Clickhouse is running good and it is really good database. I have observed that there are so many process which is taking huge memory. I did not changed anything in configuration. It is default one except time out parameter.

I have attached one screenshot and looking for suggestions, How to get rid of this. I am facing this issue after executing data migration from postgresql to click house. it happened when  2-3 scripts executed at once.

Your suggestions are welcome and really appreciated.

Thanks

[Denis Zhuravlev]

RES=2700M = 2.7GB

VIRT=228GB

2.7GB -- rather small value for CH.

CH "select" queries able to any amount of memory if they are not restricted by limits.

CH "insert" also able to use a lot of memory because they allocate 2MB buffers for every column for faster/better compression.

CH has several caches, i.e. by default mark cache is 5GB, uncompressed cache 8GB, external dictionaries stored in memory so they able to eat infinite amount of memory.

228GB -- it's a real memory, it's a virtual memory -- address space.

This value does not matter. 64 bit computer has 18,446,744,073,709,551,616 bytes, or about 18.4 exabytes.
CH uses MemoryMapped files in some cases, so for big files can book big space.

Also CH uses jemalloc http://jemalloc.net/ . This allocator caches address space by performance reasons.

You can observe that CH VIRT= ~2TB it absolutely normal and expected behavior and that's why overcommit should be enabled  https://clickhouse.yandex/docs/en/operations/tips/#ram

[Denis Zhuravlev]

*228GB -- it's NOT a real memory