Materialized View problem

[xu.big...@gmail.com]

Hi guys,  

I am using clickhouse 19.1.6.   I create a kafka engine table to read streaming data , and create a materialized view to store the data, just as the official documents shows.  but it always throw an exception after several minutes.  yes, it looks good at the first several minutes when be created.   And then the clickhouse service crashed, I can't restart the service except deleting the view sql file under /var/lib/clickhouse/metadata/

here is the error log

2019.02.22 11:10:05.037223 [ 2 ] {} <Error> BaseDaemon: (from thread 36) Received signal Segmentation fault (11).

2019.02.22 11:10:05.037246 [ 2 ] {} <Error> BaseDaemon: Address: NULL pointer.

2019.02.22 11:10:05.037266 [ 2 ] {} <Error> BaseDaemon: Access: read.

2019.02.22 11:10:05.037286 [ 2 ] {} <Error> BaseDaemon: Address not mapped to object.

2019.02.22 11:10:05.064897 [ 2 ] {} <Error> BaseDaemon: 0. clickhouse-server(DB::JSONEachRowRowInputStream::read(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, DB::RowReadExtension&)+0x30) [0x4cf3bf0]

2019.02.22 11:10:05.064963 [ 2 ] {} <Error> BaseDaemon: 1. clickhouse-server(DB::BlockInputStreamFromRowInputStream::readImpl()+0xa4) [0x4f577f4]

2019.02.22 11:10:05.064991 [ 2 ] {} <Error> BaseDaemon: 2. clickhouse-server(DB::IProfilingBlockInputStream::read()+0x205) [0x48a71d5]

2019.02.22 11:10:05.065015 [ 2 ] {} <Error> BaseDaemon: 3. clickhouse-server(DB::KafkaBlockInputStream::readImpl()+0x3e) [0x4be95ae]

2019.02.22 11:10:05.065038 [ 2 ] {} <Error> BaseDaemon: 4. clickhouse-server(DB::IProfilingBlockInputStream::read()+0x205) [0x48a71d5]

2019.02.22 11:10:05.065064 [ 2 ] {} <Error> BaseDaemon: 5. clickhouse-server(DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::atomic<bool>*)+0x5e) [0x48c04be]

2019.02.22 11:10:05.065102 [ 2 ] {} <Error> BaseDaemon: 6. clickhouse-server(DB::StorageKafka::streamToViews()+0x63c) [0x4be429c]

2019.02.22 11:10:05.065126 [ 2 ] {} <Error> BaseDaemon: 7. clickhouse-server(DB::StorageKafka::streamThread()+0x8b) [0x4be471b]

2019.02.22 11:10:05.065150 [ 2 ] {} <Error> BaseDaemon: 8. clickhouse-server(DB::BackgroundSchedulePool::TaskInfo::execute()+0xd8) [0x4d50b08]

2019.02.22 11:10:05.065173 [ 2 ] {} <Error> BaseDaemon: 9. clickhouse-server(DB::BackgroundSchedulePool::threadFunction()+0x62) [0x4d513b2]

2019.02.22 11:10:05.065194 [ 2 ] {} <Error> BaseDaemon: 10. clickhouse-server() [0x6387c2f]

2019.02.22 11:10:05.065222 [ 2 ] {} <Error> BaseDaemon: 11. /lib64/libpthread.so.0(+0x7dd5) [0x7fdfbf07edd5]

[xu.big...@gmail.com]

I tried the new version 19.3.5, but still meet the same issue.  BTW, the clickhouse server takes a lot of disk space , not data files, but files in /var/lib/clickhouse/cores.  files like core.636 core.4403 ... ...average 1~2 GB per file.

these files keep increasing even the clickhouse server is not running.

[xu.big...@gmail.com]

test case detail:

1. create kafka engine table

 CREATE TABLE BI.S_V3 (

transactionnumber UInt64, 

warehousenumber Nullable(String)

) 

ENGINE = Kafka SETTINGS kafka_broker_list = 'XXXXXX:9092',

kafka_topic_list = 'bi_realtime_shippingcarrierhistory',

kafka_group_name = 'test_v3',

kafka_format = 'JSONEachRow',

kafka_row_delimiter = '\n';

2.create mergetree table to store stream data

CREATE TABLE BI.F_V3 (

transactionnumber UInt64, 

warehousenumber Nullable(String),

date_key Date) ENGINE = MergeTree(date_key, (transactionnumber), 8192);

3. create materialized view.

CREATE MATERIALIZED VIEW BI.V_V3 TO BI.F_V3 AS SELECT *, toDate(now())as date_key from BI.S_V3;

it works fine if I didn't create Materialized View, and once the view created , clickhouse server crashed soon.

2019.03.04 15:46:39.356000 [ 46 ] {} <Trace> StorageKafka (S_V3): Committing message with offset 2450202

2019.03.04 15:46:39.356021 [ 46 ] {} <Trace> StorageKafka (S_V3): Polled batch of 1 messages

2019.03.04 15:46:39.356097 [ 42 ] {de558461-b3f1-4a23-941f-fb9f91088ff1} <Trace> StorageKafka (S_V3): Committing message with offset 2450203

2019.03.04 15:46:39.356184 [ 42 ] {de558461-b3f1-4a23-941f-fb9f91088ff1} <Debug> MemoryTracker: Peak memory usage (for query): 1.02 MiB.

2019.03.04 15:46:39.356226 [ 42 ] {de558461-b3f1-4a23-941f-fb9f91088ff1} <Debug> MemoryTracker: Peak memory usage (total): 1.02 MiB.

2019.03.04 15:46:39.356240 [ 42 ] {de558461-b3f1-4a23-941f-fb9f91088ff1} <Information> TCPHandler: Processed in 0.014 sec.

2019.03.04 15:47:18.364107 [ 42 ] {0cb1e0a2-0c7b-4257-b613-fc856308fa7c} <Debug> executeQuery: (from [::1]:52820) CREATE MATERIALIZED VIEW BI.V_V3 TO BI.F_V3 AS SELECT *, toDate(now())as date_key from BI.S_V3;

2019.03.04 15:47:18.365229 [ 42 ] {0cb1e0a2-0c7b-4257-b613-fc856308fa7c} <Debug> MemoryTracker: Peak memory usage (for query): 4.06 KiB.

2019.03.04 15:47:18.365266 [ 42 ] {0cb1e0a2-0c7b-4257-b613-fc856308fa7c} <Debug> MemoryTracker: Peak memory usage (total): 4.06 KiB.

2019.03.04 15:47:18.365299 [ 42 ] {0cb1e0a2-0c7b-4257-b613-fc856308fa7c} <Information> TCPHandler: Processed in 0.001 sec.

2019.03.04 15:47:18.865476 [ 29 ] {} <Debug> StorageKafka (S_V3): Started streaming to 1 attached views

2019.03.04 15:47:18.891144 [ 29 ] {} <Trace> StorageKafka (S_V3): Polled batch of 65536 messages

2019.03.04 15:47:18.919831 [ 47 ] {} <Error> BaseDaemon: ########################################

2019.03.04 15:47:18.919901 [ 47 ] {} <Error> BaseDaemon: (from thread 29) Received signal Segmentation fault (11).

2019.03.04 15:47:18.919912 [ 47 ] {} <Error> BaseDaemon: Address: NULL pointer.

2019.03.04 15:47:18.919919 [ 47 ] {} <Error> BaseDaemon: Access: read.

2019.03.04 15:47:18.919926 [ 47 ] {} <Error> BaseDaemon: Address not mapped to object.

2019.03.04 15:47:18.947618 [ 47 ] {} <Error> BaseDaemon: 0. clickhouse-server(DB::JSONEachRowRowInputStream::read(std::vector<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COWPtr<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, DB::RowReadExtension&)+0x30) [0x57e3600]

2019.03.04 15:47:18.947662 [ 47 ] {} <Error> BaseDaemon: 1. clickhouse-server(DB::BlockInputStreamFromRowInputStream::readImpl()+0xa4) [0x5a6b144]

2019.03.04 15:47:18.947674 [ 47 ] {} <Error> BaseDaemon: 2. clickhouse-server(DB::IBlockInputStream::read()+0x1f5) [0x5343945]

2019.03.04 15:47:18.947683 [ 47 ] {} <Error> BaseDaemon: 3. clickhouse-server(DB::KafkaBlockInputStream::readImpl()+0x19) [0x5d7a3c9]

2019.03.04 15:47:18.947689 [ 47 ] {} <Error> BaseDaemon: 4. clickhouse-server(DB::IBlockInputStream::read()+0x1f5) [0x5343945]

2019.03.04 15:47:18.947704 [ 47 ] {} <Error> BaseDaemon: 5. clickhouse-server(DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::atomic<bool>*)+0x60) [0x535c8f0]

2019.03.04 15:47:18.947710 [ 47 ] {} <Error> BaseDaemon: 6. clickhouse-server(DB::StorageKafka::streamToViews()+0x62c) [0x5d6f56c]

2019.03.04 15:47:18.947716 [ 47 ] {} <Error> BaseDaemon: 7. clickhouse-server(DB::StorageKafka::streamThread()+0x8b) [0x5d6f9bb]

2019.03.04 15:47:18.947721 [ 47 ] {} <Error> BaseDaemon: 8. clickhouse-server(DB::BackgroundSchedulePool::TaskInfo::execute()+0xd8) [0x58610d8]

2019.03.04 15:47:18.947727 [ 47 ] {} <Error> BaseDaemon: 9. clickhouse-server(DB::BackgroundSchedulePool::threadFunction()+0x62) [0x5861982]

2019.03.04 15:47:18.947732 [ 47 ] {} <Error> BaseDaemon: 10. clickhouse-server() [0x58619f4]

2019.03.04 15:47:18.947738 [ 47 ] {} <Error> BaseDaemon: 11. clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x199) [0x5da11e9]

2019.03.04 15:47:18.947743 [ 47 ] {} <Error> BaseDaemon: 12. clickhouse-server() [0x65c572f]

2019.03.04 15:47:18.947747 [ 47 ] {} <Error> BaseDaemon: 13. /lib64/libpthread.so.0(+0x7dd5) [0x7f30d6724dd5]

2019.03.04 15:47:21.807462 [ 42 ] {a89390f3-7d4e-472d-aa07-a827be851954} <Debug> executeQuery: (from [::1]:52820) SHOW TABLES;

2019.03.04 15:47:21.808194 [ 42 ] {a89390f3-7d4e-472d-aa07-a827be851954} <Trace> InterpreterSelectQuery: FetchColumns -> Complete

2019.03.04 15:47:21.808303 [ 42 ] {a89390f3-7d4e-472d-aa07-a827be851954} <Debug> executeQuery: Query pipeline:

[Mikhail Filimonov]

It looks like you've faced some bug in ClickHouse. It can be connected with build settings (which build do you use?) or with some specific circumstances of your workflow.  

Please, open the issue on github, and describe (in details) how to reproduce that problem. If developers will able to reproduce the problem they will be also able to create a fix for that.