problem with Prometheus Docker container exporter

[LMR]

Hi,

I was trying to follow the instructions here
http://rancher.com/docker-monitoring-continued-prometheus-and-sysdig/
to evaluate the product

One of the surprises came when launching the Prometheus container. The config file has changed from a pseudo-JSON format to YAML ;) mmm...

Note the date of the review above: it wasn't a decade or a year ago. I guess I'm just unlucky.
I couldn't figure out the YAML syntax and after 4 attempts I downloaded the converter. A MASSIVE THANK YOU for that Julius (I had to download the compiled artifact because the 'go get git') did not create the files and dir as you specify in the commands you suggested. Glad there was the exe :)
--

Now, when I connect my browser to the Prometheus container it sees itself but the container-exporter is apparently unhealthy. The comment gives me the URL, which is correct and says: connection refused:
==> Get http://localhost:9104/metrics: dial tcp 127.0.0.1:9104: connection refused.

The funny thing is that if I try that URL on another web page I can see the metrics....

Can anybody suggest what I might be doing wrong?

I'm running on Ubuntu 10.04; Docker 1.7.1; go 1.4.2 and you images just downloaded (docker run) about an hour ago

thank you very much

[Johannes Ziemke]

Hi,

On Thu, Aug 6, 2015 at 4:25 PM, LMR <luca.ra...@gmail.com> wrote:

Now, when I connect my browser to the Prometheus container it sees itself but the container-exporter is apparently unhealthy. The comment gives me the URL, which is correct and says: connection refused:
==> Get http://localhost:9104/metrics: dial tcp 127.0.0.1:9104: connection refused.

You are running prometheus and the exporter each in it's own container, right? Then you can't use 127.0.0.1 because your container has a separated network namespace, that means 127.0.0.1 in the container isn't 127.0.0.1 on the host.

You should either use the IP of your host or start the container with --network=host.

[tee...@gmail.com]

Hi guys,

I had the same issue, also when trying to update the global configuration with Chef cookbooks. I was getting the error:
http://localhost:9100/metrics DOWN instance="localhost:9100" 4.415s ago Get http://localhost:9100/metrics: dial tcp 127.0.0.1:9100: getsockopt: connection refused"

In de logs: "Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused"

Here's some adjustments I made in the prometheus.yml and the node-exporter docker run:
prometheus.yml:

for node-exporter create a seperate endpoint:
# Node-exporter endpoint
- job_name: 'node-exporter'
scrape_interval: 5s
static_configs:
- targets: ['<full hostname>:9101']

In the node-exporter command create:

-collector.supervisord.url=http://<full hostname>:9101/metrics

sudo docker run -d -p 9100:9100 \
-v "/proc:/host/proc:ro" \
-v "/sys:/host/sys:ro" \
-v "/:/rootfs:ro" \
--net="host" prom/node-exporter \
-collector.supervisord.url=http://<hostname>:9101/metrics \
-collector.procfs /host/proc \
-collector.sysfs /host/sys \
-collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)" \
-log.level=debug \
-collector.textfile.directory=/var/lib/node_exporter/textfile_collector

cookbook:
docker_container 'node-exporter' do
only_if { node.chef_environment == '<Chef environment' }
only_if { node.fqdn == '<full hostname or IP>' }
repo 'prom/node-exporter'
tag 'latest'
port '9100:9100' # can be also: <other free port>:9100.
volumes ['/:/rootfs:ro','/var/run:/var/run:rw','/sys:/sys:ro','/var/lib/docker/:/var/lib/docker:ro','/proc:/host/proc:ro','/sys:/host/sys:ro']
command ['-collector.supervisord.url=http://<full hostname or IP>:9100/metrics','-log.level=debug','-collector.textfile.directory=/var/lib/node_exporter/textfile_collector']
detach true
restart_policy 'always'
action :run
end

I think IP address should also work and my config is running on 9101:9100, but I think you could choose any other free port(s).

Then docker stop/ kill and rm the running containers and execute the docker run again or in case you use Chef, start the cookbook.

Op donderdag 6 augustus 2015 17:24:27 UTC+2 schreef Johannes Ziemke: