DB::NetException: Timeout exceeded while reading from socket - Insert into Merge Tree

[sparsh sidana]

Hello,

I'm inserting data to a merge tree table from  a distributed table. The insert statement takes around 3 min to run at times, however i've been getting this error recently:
The insert statement is being run prod-1 node but getting a timeout from node-10 as the table from which the data is being read is distributed.

2020.10.01 00:38:43.513971 [ 16114 ] {} <Error> DynamicQueryHandler: Code: 209, e.displayText() = DB::NetException: Timeout exceeded while reading from socket (10.22.192.82:9000): while receiving packet from prod-10:9000: While executing Remote: While executing CreatingSetsTransform, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x10f1e130 in /usr/bin/clickhouse

1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x95f82cd in /usr/bin/clickhouse

2. ? @ 0xe6521c2 in /usr/bin/clickhouse

3. DB::readVarUInt(unsigned long&, DB::ReadBuffer&) @ 0x97baea1 in /usr/bin/clickhouse

4. DB::Connection::receivePacket() @ 0xe64deb7 in /usr/bin/clickhouse

5. DB::MultiplexedConnections::receivePacket() @ 0xe66e07a in /usr/bin/clickhouse

6. DB::RemoteQueryExecutor::read() @ 0xd9f3754 in /usr/bin/clickhouse

7. DB::RemoteSource::generate() @ 0xe966b5f in /usr/bin/clickhouse

8. DB::ISource::work() @ 0xe72b16b in /usr/bin/clickhouse

9. DB::SourceWithProgress::work() @ 0xe96d6d7 in /usr/bin/clickhouse

10. ? @ 0xe758df1 in /usr/bin/clickhouse

11. ? @ 0xe75d576 in /usr/bin/clickhouse

12. ? @ 0xe75dbc2 in /usr/bin/clickhouse

13. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x9625f47 in /usr/bin/clickhouse

14. ? @ 0x9624433 in /usr/bin/clickhouse

15. start_thread @ 0x7ea5 in /usr/lib64/libpthread-2.17.so

16. __clone @ 0xfe8dd in /usr/lib64/libc-2.17.so

Any suggestions to fix this?