Hello world of Apache Druid doesn't work for docker-compose

[aleksanderacai]

Hello world of Apache Druid doesn't work for docker-compose

Hi,

I wanted to launch instance of Apache Druid 0.22.0 on my Big Sur Mac OS with use od Docker Desktop 4.1.0 with Engine 20.10.8

What I've made was:

1. Ensure that all other containers in my Docker (especially these with name postgres) won't colide with desired configuration

2. Open page druid.apache.org/docs/latest/tutorials/docker.html and follow some advices there:

a) Create Druid subfolder in my local (being host for Docker) user directory

b) Download raw.githubusercontent.com/apache/druid/0.22.0/distribution/docker/docker-compose.yml to Druid subfolder

c) Download raw.githubusercontent.com/apache/druid/0.22.0/distribution/docker/environment to Druid subfolder

d) Enter to Druid subfolder in Terminal and run "docker-compose up"

e) Verify results by entering localhost:8888 in your Safari browser

3. What I found was and what I've done:

  a) that 3 of 7 services were already down, this was: historical, coordinator and broker

  b) so I changed password from FoolishPassword to druid in envirment file what caused that only 2 of 7 services are now down, this are: historical and broker

  c) that volumes on host side declared in docker-compose.yml were not created

  d) so I have prefixed all the names with ./ what resulted in: ./metadata_data:/var/lib/postgresql/data, ./druid_shared:/opt/shared and same for remaining 5 folders there

  e) the folders has been created but still 2 of 7 services are down

  f) Entering to localhost:8888 has welcomed me with error that:

  "It appears that the service serving this console is not responding. The console will not function at the moment."

        and 5 widgets got "Error: Requested failed with status code 500", this was for Datasource, Segments, Supervisors, Tasks, Services

        the Status were ok (10 extensions loaded) and Lookups reported that it is uninitilized

  g) In the docker logs for coordinator I see spaming message: "Uh... I have no servers. Not assigning anything..."

  h) Postgres has reported: "  could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted "

  i) Router has thrown Java exception starting with message: " org.apache.druid.java.util.common.ISE: No default server found! "

  j) middlemanager looks ok

  k) broker looks ok

  l) Postgres reported ealier something strange, first was  " CREATE DATABASE " then "  /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* "

  then "received fast shutdown request", "database system is shut down", "server stopped", "PostgreSQL init process complete; ready for start up.",

  "database system was shut down at 2021-10-20 15:43:37 UTC"  and finally " database system is ready to accept connections "

4. I was trying to delete whole druid group in Docker together with created folders on host (in my Druid subfolder which I have created in 2 a) ) and run everything again,

   but I still get same results

5. It might be that postgres should be bound to some particular version and some passwords for historical, coodrinator and broker should be set by environment file (see points 3 a) and b) )

Let me phrase with use of youtube.com/watch?v=P5Dhkz7471E - Please help

[Sergio Ferragut]

I was able to reproduce your results. The short story is that you need to add more memory to the Resources in Docker, I bumped it up to 8GB and that did it.

Longer story:

Before I could get it to "work", I ran into an issue in the environment file which is seemingly different on Ubuntu, but on Mac OS, environment variables cannot have spaces unless they are surrounded by single or double quotes so I had to change the variables in the environment file as follows:

this:

   druid_extensions_loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage"]

became this (notice the surrounding single quotes in the variable value):

   druid_extensions_loadList='["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage"]'

and also for these two variables:

   druid_indexer_runner_javaOptsArray='["-server", "-Xmx1g", "-Xms1g", "-XX:MaxDirectMemorySize=3g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]'

   DRUID_LOG4J='<?xml version="1.0" encoding="UTF-8" ?><Configuration status="WARN"><Appenders><Console name="Console" target="SYSTEM_OUT"><PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/></Console></Appenders><Loggers><Root level="info"><AppenderRef ref="Console"/></Root><Logger name="org.apache.druid.jetty.RequestLog" additivity="false" level="DEBUG"><AppenderRef ref="Console"/></Logger></Loggers></Configuration>'

After that I ran docker-compose up which resulted in the same situation you describe, except the failed containers were broker and historical in my case.

In the docker compose output noticed that the broker and historical failed with:

broker exited with code 137

historical exited with code 137

Googled it and found that containers fail with that error when there isn't enough memory available to docker. Increased memory and reran docker-compose up successfully.

Hope this helps,

Sergio 

[aleksanderacai]

Thank you! It worked!

So to summarise:

- increase docker memory from default 2GB to 8GB (well I had same issue long time ago about that and even I thought about that for a moment yesterday, but forgot during triage. I will suggest guys responsible for Docker Desktop to show some kind of popup since OOM exceptions are very exceptional, yet very important and should be thrown into face)

- changing variables in "environment" file to not contain any space, it can be achieved by surrounding values with quote character

and here are some additional advices:

- change content of volumes declaration, that left side instead of being name, becomes a local path for host, just prefix them with ./ and storage is going to be exposed to host (and not hidden in Docker volumes manager)

- add volumes for zookeeper in docker-compose with following entries: 

      - ./zookeeper_data:/data

      - ./zookeeper_datalog:/datalog

      - ./zookeeper_logs:/logs

- modify name of services in docker-compose for zookeeper and postgres (or maybe for all) to something like druid_zookeeper and druid_postgres as this sometimes collide with other containers (developers and architects are working on different things/with different images).

Thanks again for looking at this, reproducing and giving fast response!

Cheers,

Aleksander