Trouble with Marathon SD (Marathon server not a bare host)

Marcin Ciura

unread,

Oct 2, 2017, 3:50:30 AM10/2/17

to Prometheus Users

Hello prometheus-users,

I am trying to monitor my Marathon jobs from my localhost via VPN. The relevant part of my monitoring config is

scrape_configs:

- job_name: 'recommender'

marathon_sd_configs:

- servers:

- 'https://<my host>/marathon'

refresh_interval: 120s

In the browser, I can see the JSON with the list of my apps at https://<my host>/marathon/v2/apps; however, when I launch Prometheus (version=1.7.2) locally, it keeps logging errors:

ERRO[224672] Error while updating services: invalid character '<' looking for beginning of value source="marathon.go:137"

and does not discover my jobs at all. My wild guess is that it hits some HTML page instead of a JSON, possibly appending a path to the host rather than to the entire URL.

Another, transient error shows a correct URL:

ERRO[223466] Error while updating services: Get https://<my host>/marathon/v2/apps/?embed=apps.tasks: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) source="marathon.go:137"

Am I doing something wrong?

Cheers,

Marcin

Marcin Ciura

unread,

Oct 2, 2017, 5:17:28 AM10/2/17

to Prometheus Users

OK, figured it out to some extent. The page that chokes the JSON parser is
<html>
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.13.0</center>
</body>
</html>

Cheers,

Marcin

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/TIq6YYb2OXw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/25daad00-d548-4e4f-b636-ec895aa219d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marcin Ciura

unread,

Oct 4, 2017, 7:00:57 AM10/4/17

to Prometheus Users

My problem led me to https://github.com/prometheus/prometheus/issues/3090 (basic_auth support in marathon_sd_config).
Someone please review this PR that fixes this issue for me: https://github.com/prometheus/prometheus/pull/3238