Replication not working

[sparsh sidana]

Hello,

Clickhouse version: 20.5.3

We've replicated tables created in Clickhouse using the replicated merge tree engine, however, the replication isn't working. I dropped and re-created a replica, but getting this error in the logs:

2021.01.14 23:16:45.245428 [ 5584 ] {} <Error> buybox_shard_1.local_job_metrics: DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 226, e.displayText() = DB::Exception: Marks file '/data/lib/clickhouse/data/buybox_shard_1/local_job_metrics/tmp_fetch_f6ede5c0cdb525e3c6e2ba4bf92a4a14_49_49_0/job_id.mrk' doesn't exist, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x10f1e130 in /usr/bin/clickhouse

1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x95f82cd in /usr/bin/clickhouse

2. ? @ 0xe4c0323 in /usr/bin/clickhouse

3. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0xe40a4f5 in /usr/bin/clickhouse

4. DB::DataPartsExchange::Fetcher::downloadPart(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<DB::IReservation, std::__1::default_delete<DB::IReservation> >, DB::PooledReadWriteBufferFromHTTP&) @ 0xe3efb16 in /usr/bin/clickhouse

5. DB::DataPartsExchange::Fetcher::fetchPart(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, DB::ConnectionTimeouts const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xe3f3db6 in /usr/bin/clickhouse

6. ? @ 0xe2babb3 in /usr/bin/clickhouse

7. DB::StorageReplicatedMergeTree::fetchPart(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, unsigned long) @ 0xe304663 in /usr/bin/clickhouse

8. DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&) @ 0xe3078a3 in /usr/bin/clickhouse

9. DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&) @ 0xe308ed1 in /usr/bin/clickhouse

10. ? @ 0xe3092bd in /usr/bin/clickhouse

11. DB::ReplicatedMergeTreeQueue::processEntry(std::__1::function<std::__1::shared_ptr<zkutil::ZooKeeper> ()>, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&, std::__1::function<bool (std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&)>) @ 0xe5f7e12 in /usr/bin/clickhouse

12. DB::StorageReplicatedMergeTree::queueTask() @ 0xe2bc52e in /usr/bin/clickhouse

13. DB::BackgroundProcessingPool::workLoopFunc() @ 0xe3edba3 in /usr/bin/clickhouse

14. ? @ 0xe3ee4d2 in /usr/bin/clickhouse

15. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x9625f47 in /usr/bin/clickhouse

16. ? @ 0x9624433 in /usr/bin/clickhouse

17. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so

18. __clone @ 0xfe8dd in /usr/lib64/libc-2.17.so

 (version 20.5.3.27 (official build))

Structure of the table:

 CREATE TABLE buybox_shard_1.local_item_seller_gmv_aggregated

(

    `legacySellerId` String,

    `itemId` String,

    `gmv_offer` Float64,

    `gmv_item` Float64,

    `gmv_opportunity` Float64,

    `time` UInt64,

    `partner_id` String

)

ENGINE = ReplicatedMergeTree('/clickhouse/item_seller_gmv_aggregated/buybox_shard_1/events', 'replica_5')

PARTITION BY time

ORDER BY legacySellerId

TTL toDateTime(time / 1000) + toIntervalDay(15)

SETTINGS index_granularity = 8192 

The re-created replica has 0 records. We keep getting the missing mrk files error.
Replica status: buybox_shard_1.local_job_metrics: Absolute delay: 5821133. Relative delay: 5821133.

Having similar issue with another table. 

Thanks,

Sparsh