Urgent: corrupt database

[Itai Shirav]

Hi,

We have a clickhouse server which was killed (kill --9) and now prints repeatedly errors such as the ones below. New data cannot be inserted. I guess something is corrupted, but is there a way to recover? Is it safe to delete the parts listed in the error?

Thanks

2017.02.01 14:10:19.750795 [ 12 ] <Error> metrics-1068.nastotals (Merger): Part 20170201_20170201_558862_558934_7 intersects previous part 20170201_20170201_548704_558916_953

2017.02.01 14:10:19.750995 [ 14 ] <Error> metrics-1068.nastotals (Merger): Part 20170201_20170201_558862_558934_7 intersects previous part 20170201_20170201_548704_558916_953

2017.02.01 14:10:19.751143 [ 17 ] <Error> metrics-1068.nastotals (Merger): Part 20170201_20170201_558862_558934_7 intersects previous part 20170201_20170201_548704_558916_953

2017.02.01 14:10:19.751239 [ 16 ] <Error> metrics-1068.nastotals (Merger): Part 20170201_20170201_558862_558934_7 intersects previous part 20170201_20170201_548704_558916_953

2017.02.01 14:10:19.896634 [ 14 ] <Error> metrics-1068.systemtotals (Merger): Part 20170201_20170201_654646_665150_946 intersects previous part 20170201_20170201_654646_665102_957

2017.02.01 14:10:19.896794 [ 4 ] <Error> metrics-1068.systemtotals (Merger): Part 20170201_20170201_654646_665150_946 intersects previous part 20170201_20170201_654646_665102_957

2017.02.01 14:10:19.896827 [ 16 ] <Error> metrics-1068.systemtotals (Merger): Part 20170201_20170201_654646_665150_946 intersects previous part 20170201_20170201_654646_665102_957

2017.02.01 14:10:19.896867 [ 7 ] <Error> metrics-1068.systemtotals (Merger): Part 20170201_20170201_654646_665150_946 intersects previous part 20170201_20170201_654646_665102_957

2017.02.01 14:10:19.896891 [ 19 ] <Error> metrics-1068.systemtotals (Merger): Part 20170201_20170201_654646_665150_946 intersects previous part 20170201_20170201_654646_665102_957

2017.02.01 14:10:19.896924 [ 9 ] <Error> metrics-1068.systemtotals (Merger): Part 20170201_20170201_654646_665150_946 intersects previous part 20170201_20170201_654646_665102_957

2017.02.01 14:10:19.896997 [ 15 ] <Error> metrics-1068.systemtotals (Merger): Part 20170201_20170201_654646_665150_946 intersects previous part 20170201_20170201_654646_665102_957

2017.02.01 14:10:19.938776 [ 14 ] <Error> metrics-20067.cachestats (Merger): Part 20170201_20170201_1444544_1453154_832 intersects previous part 20170201_20170201_1444544_1453142_936

2017.02.01 14:10:19.938977 [ 20 ] <Error> metrics-20067.cachestats (Merger): Part 20170201_20170201_1444544_1453154_832 intersects previous part 20170201_20170201_1444544_1453142_936

2017.02.01 14:10:19.939066 [ 11 ] <Error> metrics-20067.cachestats (Merger): Part 20170201_20170201_1444544_1453154_832 intersects previous part 20170201_20170201_1444544_1453142_936

2017.02.01 14:10:20.038083 [ 14 ] <Error> metrics-20067.iobysize (Merger): Part 20170201_20170201_1452354_1452550_20 intersects previous part 20170201_20170201_1444540_1452510_730

2017.02.01 14:10:20.038298 [ 15 ] <Error> metrics-20067.iobysize (Merger): Part 20170201_20170201_1452354_1452550_20 intersects previous part 20170201_20170201_1444540_1452510_730

2017.02.01 14:10:20.038360 [ 20 ] <Error> metrics-20067.iobysize (Merger): Part 20170201_20170201_1452354_1452550_20 intersects previous part 20170201_20170201_1444540_1452510_730

2017.02.01 14:10:20.038453 [ 12 ] <Error> metrics-20067.iobysize (Merger): Part 20170201_20170201_1452354_1452550_20 intersects previous part 20170201_20170201_1444540_1452510_730

2017.02.01 14:10:20.038510 [ 9 ] <Error> metrics-20067.iobysize (Merger): Part 20170201_20170201_1452354_1452550_20 intersects previous part 20170201_20170201_1444540_1452510_730

2017.02.01 14:10:20.038563 [ 21 ] <Error> metrics-20067.iobysize (Merger): Part 20170201_20170201_1452354_1452550_20 intersects previous part 20170201_20170201_1444540_1452510_730

2017.02.01 14:10:20.038616 [ 11 ] <Error> metrics-20067.iobysize (Merger): Part 20170201_20170201_1452354_1452550_20 intersects previous part 20170201_20170201_1444540_1452510_730

2017.02.01 14:10:20.038669 [ 16 ] <Error> metrics-20067.iobysize (Merger): Part 20170201_20170201_1452354_1452550_20 intersects previous part 20170201_20170201_1444540_1452510_730

2017.02.01 14:10:20.038720 [ 19 ] <Error> metrics-20067.iobysize (Merger): Part 20170201_20170201_1452354_1452550_20 intersects previous part 20170201_20170201_1444540_1452510_730

2017.02.01 14:10:20.099635 [ 14 ] <Error> void DB::BackgroundProcessingPool::threadFunction(): Code: 84, e.displayText() = DB::Exception: Directory /opt/clickhouse//data/metrics%2D1170/poolinfo/20170201_20170201_3616_3636_10 already exists, e.what() = DB::Exception, Stack trace:

0. clickhouse-server(StackTrace::StackTrace()+0x16) [0x10d1466]

1. clickhouse-server(DB::Exception::Exception(std::string const&, int)+0x1f) [0x100996f]

2. clickhouse-server(DB::MergeTreeDataMerger::mergePartsToTemporaryPart(std::vector<std::shared_ptr<DB::MergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::MergeTreeDataPart const> > >&, std::string const&, DB::MergeListEntry&, unsigned long, long, DB::DiskSpaceMonitor::Reservation*)+0x22ac) [0x1242dac]

3. clickhouse-server(DB::StorageMergeTree::merge(unsigned long, bool, std::string const&, bool)+0x671) [0x11a1171]

4. clickhouse-server(DB::StorageMergeTree::mergeTask()+0x148) [0x11a1998]

5. clickhouse-server(DB::BackgroundProcessingPool::threadFunction()+0x3d3) [0x11f53b3]

6. clickhouse-server(execute_native_thread_routine+0x20) [0x345bd00]

7. /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7fb12e9cb184]

8. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fb12dfe637d]

2017.02.01 14:10:20.109506 [ 14 ] <Error> void DB::BackgroundProcessingPool::threadFunction(): Code: 49, e.displayText() = DB::Exception: Could not find a column of minimum size in MergeTree, e.what() = DB::Exception, Stack trace:

0. clickhouse-server(StackTrace::StackTrace()+0x16) [0x10d1466]

1. clickhouse-server(DB::Exception::Exception(std::string const&, int)+0x1f) [0x100996f]

2. clickhouse-server(DB::MergeTreeDataPart::getMinimumSizeColumnName() const+0x1bc) [0x121a78c]

3. clickhouse-server(DB::MergeTreeBlockInputStream::injectRequiredColumns(std::vector<std::string, std::allocator<std::string> >&) const+0x6c6) [0x12ab306]

4. clickhouse-server(DB::MergeTreeBlockInputStream::MergeTreeBlockInputStream(std::string const&, unsigned long, std::vector<std::string, std::allocator<std::string> >, DB::MergeTreeData&, std::shared_ptr<DB::MergeTreeDataPart const> const&, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> > const&, bool, std::shared_ptr<DB::ExpressionActions>, std::string, bool, unsigned long, unsigned long, bool)+0x718) [0x12abbc8]

5. clickhouse-server(DB::MergeTreeDataMerger::mergePartsToTemporaryPart(std::vector<std::shared_ptr<DB::MergeTreeDataPart const>, std::allocator<std::shared_ptr<DB::MergeTreeDataPart const> > >&, std::string const&, DB::MergeListEntry&, unsigned long, long, DB::DiskSpaceMonitor::Reservation*)+0x6e9) [0x12411e9]

6. clickhouse-server(DB::StorageMergeTree::merge(unsigned long, bool, std::string const&, bool)+0x671) [0x11a1171]

7. clickhouse-server(DB::StorageMergeTree::mergeTask()+0x148) [0x11a1998]

8. clickhouse-server(DB::BackgroundProcessingPool::threadFunction()+0x3d3) [0x11f53b3]

9. clickhouse-server(execute_native_thread_routine+0x20) [0x345bd00]

10. /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7fb12e9cb184]

11. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fb12dfe637d]

[Itai Shirav]

Seems like the problem was 2 instances of clickhouse-server running at the same time. Not sure how I got to this scenario...

[Alex Zatelepin]

Greetings!

Clickhouse-server locks /opt/clickhouse/status to avoid duplicate server processes for the same database, so unless there is a bug or you've somehow tampered with the data directory, there shouldn't be any duplicate processes.

As for 'Part intersects previous part' messages, a bug was fixed recently (in version 1.1.54074) that could lead to such errors. To recover the data:

1) Update the server to a recent version

2) Create another table with the same schema and copy the data into it using INSERT SELECT.

[pooja poo]

Hi,

Im using Clickhouse in my ruby script to insert the data in remote server.

I successfully establish the connection to the clickhouse and trying to create and insert data to my remote server clickhouse database getting few different error from the clickhouse sever

* Received exception from server:

Code: 49. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Could not find a column of minimum size in MergeTree, part /var/lib/clickhouse//data/test/day_channel_summaries_test/20161215_20161230_12_12_0/.

* Received exception from server:

Code: 81. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Database test doesn't exist.

* Received exception from server:

Code: 60. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Table test.day_channel_summaries_test doesn't exist..

I have successfully created the database 'test'  and table 'day_channel_summaries_test'

When i get this error i restarted the clickhouse-server and running my insert query, every time im getting anyone of the error above, after running 10 to 15 times same query at any 1 point its successfully getting inserted.

May i know why im getting this error's!?

Thanks in advance..