Tutorial => DB::NetException: I/O error: Broken pipe

[MB]

Hi all,

I am trying the tutorial, but after less then a minute loading the data consistently fails with

Code: 210. DB::NetException: I/O error: Broken pipe:  while reading from socket (127.0.0.1:9000)

  --- %      42.0 MiB / 1,141.7 MiB = 0.037    25 MiB/s       0:45

Any ideas what can be the cause ?

Ubuntu 16.04.2 LTS

ClickHouse client version 1.1.54236.

The logs seem to indicate an out-of-memory problem:  

2017.06.03 23:09:18.765561 [ 1 ] <Information> : Starting daemon with revision 54236

2017.06.03 23:09:18.767412 [ 1 ] <Information> Application: starting up

2017.06.03 23:09:18.767869 [ 1 ] <Debug> Application: rlimit on number of file descriptors is 262144

2017.06.03 23:09:18.767878 [ 1 ] <Debug> Application: Initializing DateLUT.

2017.06.03 23:09:18.767882 [ 1 ] <Trace> Application: Initialized DateLUT with time zone `Zulu'.

2017.06.03 23:09:18.785906 [ 1 ] <Debug> Application: Configuration parameter 'interserver_http_host' doesn't exist or exists and empty. Will use '== redacted ==' as replica host.

2017.06.03 23:09:18.785979 [ 1 ] <Debug> ConfigReloader: Loading config `/etc/clickhouse-server/users.xml'

2017.06.03 23:09:18.788555 [ 1 ] <Warning> ConfigProcessor: Include not found: networks

2017.06.03 23:09:18.788571 [ 1 ] <Warning> ConfigProcessor: Include not found: networks

2017.06.03 23:09:18.788961 [ 1 ] <Information> Application: Loading metadata.

2017.06.03 23:09:18.789090 [ 1 ] <Information> DatabaseOrdinary (system): Total 0 tables.

2017.06.03 23:09:18.789136 [ 1 ] <Information> DatabaseOrdinary (default): Total 1 tables.

2017.06.03 23:09:18.789690 [ 2 ] <Information> BackgroundProcessingPool: Create BackgroundProcessingPool with 16 threads

2017.06.03 23:09:18.790222 [ 2 ] <Debug> default.ontime (Data): Loading data parts

2017.06.03 23:09:18.793399 [ 2 ] <Debug> default.ontime (Data): Loaded data parts (3 items)

2017.06.03 23:09:18.793547 [ 1 ] <Debug> Application: Loaded metadata.

2017.06.03 23:09:18.793801 [ 1 ] <Information> Application: Listening http://[::1]:8123

2017.06.03 23:09:18.793823 [ 1 ] <Information> Application: Listening tcp: [::1]:9000

2017.06.03 23:09:18.793847 [ 1 ] <Information> Application: Listening interserver: [::1]:9009

2017.06.03 23:09:18.793864 [ 1 ] <Information> Application: Listening http://127.0.0.1:8123

2017.06.03 23:09:18.793878 [ 1 ] <Information> Application: Listening tcp: 127.0.0.1:9000

2017.06.03 23:09:18.793890 [ 1 ] <Information> Application: Listening interserver: 127.0.0.1:9009

2017.06.03 23:09:18.793982 [ 1 ] <Information> Application: Ready for connections.

2017.06.03 23:09:20.786129 [ 3 ] <Debug> ConfigReloader: Loading config `/etc/clickhouse-server/config.xml'

2017.06.03 23:09:20.788727 [ 3 ] <Warning> ConfigProcessor: Include not found: clickhouse_remote_servers

2017.06.03 23:09:20.788771 [ 3 ] <Warning> ConfigProcessor: Include not found: clickhouse_compression

2017.06.03 23:09:28.619820 [ 4 ] <Trace> TCPConnectionFactory: TCP Request. Address: 127.0.0.1:37866

2017.06.03 23:09:28.620295 [ 4 ] <Debug> TCPHandler: Connected ClickHouse client version 1.1.54236, user: default.

2017.06.03 23:09:28.622258 [ 4 ] <Debug> executeQuery: (from 127.0.0.1:37866) INSERT INTO ontime FORMAT CSV

2017.06.03 23:09:39.735689 [ 4 ] <Trace> default.ontime (Data): Renaming tmp_19871001_19871031_7_7_0.

2017.06.03 23:09:40.452852 [ 4 ] <Trace> default.ontime (Data): Renaming tmp_19871101_19871130_9_9_0.

2017.06.03 23:09:40.787460 [ 4 ] <Trace> default.ontime (Data): Renaming tmp_19871201_19871231_11_11_0.

2017.06.03 23:09:41.353668 [ 4 ] <Error> executeQuery: Code: 173, e.displayText() = DB::Exception: Allocator: Cannot mmap., errno: 12, strerror: Cannot allocate memory, e.what() = DB::Exception (from 127.0.0.1:37866) (in query: INSERT INTO ontime FORMAT CSV), Stack trace:

0. clickhouse-server(StackTrace::StackTrace()+0x16) [0x2969826]

1. clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x1f) [0x108f49f]

2. clickhouse-server(DB::throwFromErrno(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)+0x17c) [0x297138c]

3. clickhouse-server(Allocator<false>::alloc(unsigned long, unsigned long)+0x29f) [0x2974f9f]

4. clickhouse-server() [0x232936a]

5. clickhouse-server(DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const+0x1fb) [0x232d96b]

6. clickhouse-server(DB::NativeBlockInputStream::readData(DB::IDataType const&, DB::IColumn&, DB::ReadBuffer&, unsigned long)+0x134) [0x2317bb4]

7. clickhouse-server(DB::NativeBlockInputStream::readImpl()+0x7d5) [0x2318885]

8. clickhouse-server(DB::IProfilingBlockInputStream::read()+0x31a) [0x2307b6a]

9. clickhouse-server(DB::TCPHandler::receiveData()+0x7e) [0x10a61ee]

10. clickhouse-server(DB::TCPHandler::receivePacket()+0x1e1) [0x10a6811]

11. clickhouse-server(DB::TCPHandler::readData(DB::Settings const&)+0x1a9) [0x10a6b99]

12. clickhouse-server(DB::TCPHandler::processInsertQuery(DB::Settings const&)+0x274) [0x10a6ec4]

13. clickhouse-server(DB::TCPHandler::runImpl()+0x6b9) [0x10a7709]

14. clickhouse-server(DB::TCPHandler::run()+0x2b) [0x10a827b]

15. clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x3274d5f]

16. clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x13b) [0x327b19b]

17. clickhouse-server(Poco::PooledThread::run()+0xb7) [0x34e6e07]

18. clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0xa5) [0x34b2e45]

19. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f484797f6ba]

20. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f4846fa082d]

I am trying to run it on AWS using a t2.medium => 3 vCPU & 4 GB

How much memory should my VM have to complete the tutorial ?

Thanks

[Vitaliy Lyudvichenko]

On my laptop the insert from tutorial 

xz -v -c -d < ontime.csv.xz | clickhouse-client --query="INSERT INTO ontime FORMAT CSV"

consumes ~2GB RAM for clickhouse-server and ~4.5GB for all processes (Ubuntu 14.04).

воскресенье, 4 июня 2017 г., 2:37:10 UTC+3 пользователь MB написал: