Error while deploying elasticsearch with persistent storage volume (vmware docker volume plugin)

1,963 views
Skip to first unread message

Ashutosh Joshi

unread,
Nov 9, 2018, 12:59:33 AM11/9/18
to Wazuh mailing list
I am deploying wazuh from docker images. I am facing problem while deploying elasticsearch using persistent volume. I am using vmware docker volume plugin. After seeing the logs it seems like there i some permission problem.
I also found a note which talks about the permission problem in linux (https://github.com/deviantony/docker-elk#how-can-i-persist-elasticsearch-data), but I don't understand how can I solve this for external volumes something like I am using (vmware docker volumes). It works perfectly fine when I use local volume. Here is an example from docker-compose.yml file:

volumes:
    wazuh-m-data:
       driver: vsphere
       driver_opts:
         size: 10Gb

    wazuh-es-data:
       driver: local


Here is the docker compose file I am using:

# Wazuh App Copyright (C) 2018 Wazuh Inc. (License GPLv2)
version: '2'

services:
  wazuh:
    image: wazuh/wazuh:3.6.1_6.4.2
    hostname: wazuh-manager
    restart: always
    ports:
      - "1514:1514/udp"
      - "1515:1515"
      - "514:514/udp"
      - "55000:55000"
#      - "1516:1516"
    networks:
        - docker_elk
    volumes:
      - wazuh-m-data:/var/ossec/data
#      - my-path:/etc/postfix:Z
#      - my-path:/etc/filebeat
#      - my-custom-config-path/ossec.conf:/wazuh-config-mount/etc/ossec.conf
#   command: ["echo 'hello world'"]
    depends_on:
      - logstash
  logstash:
    image: wazuh/wazuh-logstash:3.6.1_6.4.2
    hostname: logstash
    restart: always
#    volumes:
#      - my-path:/etc/logstash/conf.d:Z
    links:
      - elasticsearch:elasticsearch
    ports:
      - "5000:5000"
    networks:
      - docker_elk
    depends_on:
      - elasticsearch
    environment:
      - LS_HEAP_SIZE=2048m
  elasticsearch:
    hostname: elasticsearch
    restart: always
    ports:
      - "9200:9200"
#      - "9300:9300"
    environment:
      - node.name=node-1
      - cluster.name=wazuh
      - network.host=0.0.0.0
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 2g
    volumes:
      - wazuh-es-data:/usr/share/elasticsearch/data
    networks:
        - docker_elk
  kibana:
    image: wazuh/wazuh-kibana:3.6.1_6.4.2
    hostname: kibana
    restart: always
#    ports:
#      - "5601:5601"
#    environment:
#      - ELASTICSEARCH_URL=http://elasticsearch:9200
    networks:
      - docker_elk
    depends_on:
      - elasticsearch
    links:
      - elasticsearch:elasticsearch
      - wazuh:wazuh
  nginx:
    image: wazuh/wazuh-nginx:3.6.1_6.4.2
    hostname: nginx
    restart: always
    environment:
      - NGINX_PORT=443
    ports:
      - "80:80"
      - "443:443"
#    volumes:
#      - my-path:/etc/nginx/conf.d:Z
    networks:
      - docker_elk
    depends_on:
      - kibana
    links:
      - kibana:kibana

networks:
  docker_elk:
    driver: bridge
    ipam:
      config:
      - subnet: 172.25.0.0/24

volumes:
    wazuh-m-data:
       driver: vsphere
       driver_opts:
         size: 10Gb

    wazuh-es-data:
       driver: vsphere
       driver_opts:
         size: 500Gb

Here are error logs from elastic search container:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2018-11-09T05:33:09,002][INFO ][o.e.n.Node               ] [node-1] initializing ...
[2018-11-09T05:33:09,013][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.2.jar:6.4.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.2.jar:6.4.2]
Caused by: java.lang.IllegalStateException: Failed to create node environment
        at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.2.jar:6.4.2]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:385) ~[?:?]
        at java.nio.file.Files.createDirectory(Files.java:682) ~[?:?]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:789) ~[?:?]
        at java.nio.file.Files.createDirectories(Files.java:775) ~[?:?]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:203) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.node.Node.<init>(Node.java:274) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.2.jar:6.4.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.2.jar:6.4.2]
        ... 6 more



Please advice.

Thank You,
Ashutosh Joshi

alfonso.r...@wazuh.com

unread,
Nov 14, 2018, 12:10:07 PM11/14/18
to Wazuh mailing list
Hello Ashutosh,

We are going to try to replicate your situation to see what we can do. 

As soon as we get it we will inform you. 

Thank you for your attention and for your patience, we are working on your problem.

Best regards,

Alfonso Ruiz-Bravo

Javier Castro

unread,
Nov 14, 2018, 6:45:11 PM11/14/18
to Wazuh mailing list
Hello,

the error you mention occurs when the mounting point is not accessible for the user id 1000.

You can take a look at elasticsearch's official documentation about it: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_b_bind_mounted_configuration

Just change the ownership of the folder you are trying to mount with chown 1000:1000 my_mount_folder.

Hope that helps.

Regards.
Reply all
Reply to author
Forward
0 new messages