Sawarm service discovery not finding correct endpoint port

[Carlos Colaço]

Prometheus is using port 80 instead of 8080 when using service discovery to get Cadvisor containers.

Config:

```

  # Create a job for Docker Swarm containers.
  - job_name: 'dockerswarm'
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: tasks
    relabel_configs:
      - source_labels: [__meta_dockerswarm_service_mode]
        regex: global
        action: keep
      - source_labels: [__meta_dockerswarm_task_desired_state]
        regex: running
        action: keep
      - source_labels: [__meta_dockerswarm_node_hostname]
        target_label: job
```

Result:

cadvisor running on port 8080:

```
CONTAINER ID        IMAGE                                      COMMAND                  CREATED             STATUS                   PORTS                     NAMES
fbac945d3422        gcr.io/google-containers/cadvisor:latest   "/usr/bin/cadvisor -…"   2 hours ago         Up 2 hours (healthy)    8080/tcp                  monitor_cadvisor.arwk
gx4tfipgvebitdx47wx4u.ur61fpo4824zn0znwt5s6ubmj
```

Even removing all relabel prometheus finds every task with correct port except those ones using 8080 ...

[Carlos Colaço]

sorry forgot this details:

```
# docker --version
Docker version 19.03.13, build 4484c46d9d
#uname -r
4.18.0-193.28.1.el8_2.x86_64
```

[Julien Pivotto]

On 16 Nov 13:12, Carlos Colaço wrote:
>
> Prometheus is using port 80 instead of 8080 when using service discovery to
> get Cadvisor containers.
>
> Config:
>
> ```
> # Create a job for Docker Swarm containers.
> - job_name: 'dockerswarm'
> dockerswarm_sd_configs:
> - host: unix:///var/run/docker.sock
> role: tasks
> relabel_configs:
> - source_labels: [__meta_dockerswarm_service_mode]
> regex: global
> action: keep
> - source_labels: [__meta_dockerswarm_task_desired_state]
> regex: running
> action: keep
> - source_labels: [__meta_dockerswarm_node_hostname]
> target_label: job
> ```

It is probably because cadvisor is probably not *explicitly* exposing a port. You
will need more advance relabeling or setting port: to deal with this.

>
> Result:
>
> [image: swarm.png]
>
> cadvisor running on port 8080:
>
> ```
> CONTAINER ID IMAGE
> COMMAND CREATED STATUS
> PORTS NAMES
> fbac945d3422 gcr.io/google-containers/cadvisor:latest
> "/usr/bin/cadvisor -…" 2 hours ago Up 2 hours (healthy)
> 8080/tcp monitor_cadvisor.arwk
> gx4tfipgvebitdx47wx4u.ur61fpo4824zn0znwt5s6ubmj
> ```
>
> Even removing all relabel prometheus finds every task with correct port
> except those ones using 8080 ...
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/289d83d2-0205-45b9-aa26-f0cafceb70e5n%40googlegroups.com.



--
Julien Pivotto
@roidelapluie

[Carlos Colaço]

can you please clarify?

[Stuart Clark]

On 16/11/2020 21:12, Carlos Colaço wrote:
>
> Prometheus is using port 80 instead of 8080 when using service
> discovery to get Cadvisor containers.
>
> Config:
>
> ```
>   # Create a job for Docker Swarm containers.
>   - job_name: 'dockerswarm'
>     dockerswarm_sd_configs:
>       - host: unix:///var/run/docker.sock
>         role: tasks
>     relabel_configs:
>       - source_labels: [__meta_dockerswarm_service_mode]
>         regex: global
>         action: keep
>       - source_labels: [__meta_dockerswarm_task_desired_state]
>         regex: running
>         action: keep
>       - source_labels: [__meta_dockerswarm_node_hostname]
>         target_label: job
> ```

As per the documentation at
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dockerswarm_sd_config
when you use the role: tasks it will return targets using the published
ports. If there are none it will use the job's "port" value, which
defaults to 80.

Therefore it suggests that you don't have a published port for that
container.

To make it work you could publish port 8080, or you could adjust your
relabling to change the port being used (by adjusting __address__)
possibly by having a label on the task.

[Julien Pivotto]

On 16 Nov 13:17, Carlos Colaço wrote:
> can you please clarify?

Is `--mode=global --publish target=8080,mode=host` in use within the cadvisor
definition? Somehow you need to expose the port explicitly for
prometheus.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d0f20ae0-dd2c-4597-a878-258f787adf1dn%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Carlos Colaço]

Cool!

Missed that detail in the documentation, Thanks a lot ...

```
  cadvisor:
    image:  gcr.io/google-containers/cadvisor
    ports:
      - target: 8080
        published: 8080
        protocol: tcp
        mode: host
```

Working now!

[Carlos Colaço]

cool thanks all for the guidance ...

as mentioned and advised here... i didn't like the fact of having that port exposed and or having to filter it in the firewall for every swarm node .... ended up fixing it like this:


```

    relabel_configs:
      - source_labels: [__address__]
        target_label: __address__
        replacement: '${1}80'
```

shouldn't this be added to documentation as well instead of guiding possible new users to exposing this ports?

Ill be more than glad than submitting a MR on the documentation for both this and the other thread i've been at if you guys agree.

[Julien Pivotto]

Yes, I am open to improvements from domain experts in the documentation. Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/61295200-0685-410b-a67c-a3f63bd21b8an%40googlegroups.com.