IO utilization stuck at 99%
155 views
Skip to first unread message
Fatlum Bajrami
Dec 10, 2021, 7:01:05 AM12/10/21
to ClickHouse
Hi people,
I have two clickhouse servers running 1 shard each, they have been running okay for a long time but suddenly one of the servers is showing 98-99% disk I/O, i have tryed to restart the faulty server but with no success, io is constantly showing 99%, i have searched google and found nothing related to this topic other than this suppose being normal.
Below is a iotop log
99.99 % clickhouse-server --config=~server.pid [BackgrProcPool]
15509 be/4 clickhou 0.00 B/s 859.71 K/s 0.00 % 99.99 % clickhouse-server --config=~use-server.pid [TCPHandler]
15223 be/4 clickhou 0.00 B/s 429.85 K/s 0.00 % 99.94 % clickhouse-server --config=~server.pid [BackgrProcPool]
15218 be/4 clickhou 0.00 B/s 683.86 K/s 0.00 % 99.43 % clickhouse-server --config=~server.pid [BackgrProcPool]
15213 be/4 clickhou 0.00 B/s 371.24 K/s 0.00 % 99.31 % clickhouse-server --config=~server.pid [BackgrProcPool]
15221 be/4 clickhou 0.00 B/s 550.99 K/s 0.00 % 99.28 % clickhouse-server --config=~server.pid [BackgrProcPool]
15509 be/4 clickhou 0.00 B/s 859.71 K/s 0.00 % 99.99 % clickhouse-server --config=~use-server.pid [TCPHandler]
15223 be/4 clickhou 0.00 B/s 429.85 K/s 0.00 % 99.94 % clickhouse-server --config=~server.pid [BackgrProcPool]
15218 be/4 clickhou 0.00 B/s 683.86 K/s 0.00 % 99.43 % clickhouse-server --config=~server.pid [BackgrProcPool]
15213 be/4 clickhou 0.00 B/s 371.24 K/s 0.00 % 99.31 % clickhouse-server --config=~server.pid [BackgrProcPool]
15221 be/4 clickhou 0.00 B/s 550.99 K/s 0.00 % 99.28 % clickhouse-server --config=~server.pid [BackgrProcPool]
Alexey Milovidov
Dec 12, 2021, 2:52:56 PM12/12/21
to Fatlum Bajrami, ClickHouse
Hi. Disk IO is related to background merge operations and this is normal.
What's not normal is that it is processing with a speed of just a few
megabytes per second.
It may indicate one of the following:
- faulty disks that need replacement (you can test them with "fio" or
similar); if the issue is happening on only one server out of many,
this is the case.
- very slow storage subsystem; if you are using small network disks in
the cloud, this may be the case and you need to provision more IO.
> --
> You received this message because you are subscribed to the Google Groups "ClickHouse" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clickhouse+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/clickhouse/2a033e9f-5cfb-460a-a6a6-953d2e79dbf7n%40googlegroups.com.
What's not normal is that it is processing with a speed of just a few
megabytes per second.
It may indicate one of the following:
- faulty disks that need replacement (you can test them with "fio" or
similar); if the issue is happening on only one server out of many,
this is the case.
- very slow storage subsystem; if you are using small network disks in
the cloud, this may be the case and you need to provision more IO.
> You received this message because you are subscribed to the Google Groups "ClickHouse" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clickhouse+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/clickhouse/2a033e9f-5cfb-460a-a6a6-953d2e79dbf7n%40googlegroups.com.
Fatlum Bajrami
Dec 24, 2021, 4:41:00 AM12/24/21
to ClickHouse
Thank you Milovidov for the replay,
This was a gcp instance ubuntu 16.04 after some reading i had the idea that it probably is a kernel problem and updated to 20.04 and the problem was resolved.
This was a gcp instance ubuntu 16.04 after some reading i had the idea that it probably is a kernel problem and updated to 20.04 and the problem was resolved.
Reply all
Reply to author
Forward
0 new messages