unable to configure container volume for storing prometheus metric data

[sundeep kumar]

Hi All, 

I am getting the below error while trying to launch the prometheus container 

level=info ts=2017-11-20T09:43:22.499991852Z caller=main.go:314 msg="Starting TSDB"

level=info ts=2017-11-20T09:43:22.500031687Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090

level=error ts=2017-11-20T09:43:22.50012617Z caller=main.go:323 msg="Opening storage failed" err="open DB in data/: open /prometheus/data/843800589: permission denied"

my docker-compose file is as shown below 

services:

  prometheus:

    image: prom/prometheus:v2.0.0

    container_name: Prometheus-Monitoring

    volumes:

      - ./PROMETHEUS:/prometheus/

      - ./PROMETHEUS/PROMETHEUS_DATA/:/prometheus/data/

    command:

      - '--config.file=/etc/prometheus/prometheus.yml'

      - '--storage.tsdb.path=data/'

      - '--web.enable-lifecycle'

      - '--web.console.templates=consoles'

      - '--web.console.libraries=/etc/prometheus/console_libraries'

      - '--storage.tsdb.retention=15d'

      - '--log.level=debug'

      - '--web.enable-admin-api'

    restart: unless-stopped

    expose:

      - 9090

    ports:

      - 9090:9090

    networks:

      - monitoring

    labels:

      app: monitoring

My understanding is that the volume is not getting created and so the contianer is failing again and again .

[sundeep kumar]

I tried giving chmod 755 access to the volume on host ,, still i am gettting the error. 

Prometheus-Monitoring | level=info ts=2017-11-21T04:57:02.843606441Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)"

Prometheus-Monitoring | level=info ts=2017-11-21T04:57:02.843637069Z caller=main.go:217 host_details="(Linux 4.10.0-37-generic #41~16.04.1-Ubuntu SMP Fri Oct 6 22:42:59 UTC 2017 x86_64 b38c8e29f604 (none))"

Prometheus-Monitoring | level=info ts=2017-11-21T04:57:02.847622405Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090

Prometheus-Monitoring | level=info ts=2017-11-21T04:57:02.85151532Z caller=main.go:314 msg="Starting TSDB"

Prometheus-Monitoring | level=error ts=2017-11-21T04:57:02.8517323Z caller=main.go:323 msg="Opening storage failed" err="open DB in /prometheus/data: open /prometheus/data/288895144: permission denied"

Prometheus-Monitoring exited with code 1

Can someone please guide me as I am unable to understand what is that I am doing wrong .

Moreover this issue is happening only with the latest 2.0 version ..

[Matt Palmer]

On Mon, Nov 20, 2017 at 09:01:31PM -0800, sundeep kumar wrote:
> I tried giving chmod 755 access to the volume on host ,, still i am
> gettting the error.

That's not going to help unless the directory is owned by the user that
Prometheus is running as.

> Moreover this issue is happening only with the latest 2.0 version ..

Indeed, Prometheus 2.0 now runs as an unprivileged user, as described in the
migration guide (https://prometheus.io/docs/prometheus/2.0/migration/).

- Matt

[sundeep kumar]

Could you suggest on how I should modify my compose file to get it working

-v /local:/prometheus/data

Will the above command line work

Envoyé de mon iPhone

> --
> You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/khALXM9NhYQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20171121064116.ksygiaas6o6557sg%40hezmatt.org.
> For more options, visit https://groups.google.com/d/optout.

[sundeep kumar]

I tried running from command line even there its throwing an error .. 

sudo docker run -d -p 9090:9090 --name Prometheus-Monitoring -v ${PWD}/PROMETHEUS:/etc/prometheus/ -v ${PWD}/PROMETHEUS/PROMETHEUS_DATA:/data prom/prometheus:v2.0.0 --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/data --web.enable-lifecycle --web.console.templates=consoles --web.console.libraries=/etc/prometheus/console_libraries --log.level=debug

2b1e1a313ee67decb4252baa351c120070e4f2ddcdbaa0faa97bdc72a6b3ab6f

ubuntu@ip-10-0-0-32:~/CM/Docker-StandAlone$ docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

ubuntu@ip-10-0-0-32:~/CM/Docker-StandAlone$ docker ps -a

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                     PORTS               NAMES

2b1e1a313ee6        prom/prometheus:v2.0.0   "/bin/prometheus -..."   5 seconds ago       Exited (1) 4 seconds ago                       Prometheus-Monitoring

ubuntu@ip-10-0-0-32:~/CM/Docker-StandAlone$ docker logs Prometheus-Monitoring

level=info ts=2017-11-22T09:41:37.973165405Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)"

level=info ts=2017-11-22T09:41:37.973218201Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)"

level=info ts=2017-11-22T09:41:37.973236133Z caller=main.go:217 host_details="(Linux 4.4.0-1038-aws #47-Ubuntu SMP Thu Sep 28 20:05:35 UTC 2017 x86_64 2b1e1a313ee6 (none))"

level=info ts=2017-11-22T09:41:37.975595301Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090

level=info ts=2017-11-22T09:41:37.975564318Z caller=main.go:314 msg="Starting TSDB"

level=error ts=2017-11-22T09:41:37.975793217Z caller=main.go:323 msg="Opening storage failed" err="open DB in /data: open /data/093999223: permission denied"

ubuntu@ip-10-0-0-32:~/CM/Docker-StandAlone$

Error is highlighted in red color can someone please let me know where i am going wrong with this .. 

On Monday, 20 November 2017 15:19:20 UTC+5:30, sundeep kumar wrote:

[joelc...@gmail.com]

Did you ever figure this out Sundeep? ....I'm having the same problem and am tired of beating my head against it....

[joelc...@gmail.com]

Actually, I think I figured this our Sundeep.

I was having exactly the same problem, but I added the REXRAY_PREEMPT=true option when installing the rexray docker plugin:

docker plugin disable rexray/ebs -f

docker plugin rm rexray/ebs -f

docker plugin install rexray/ebs REXRAY_PREEMPT=true --grant-all-permissions

...afterward, I had to stop and then start the docker daemon ( a restart didn't work)

systemctl stop docker

systemctl start docker

hope that helps!

[Gagandeep Dhillon]

@Sundeep - Were you able to fix this problem..I am getting same error while installing prometheus.