Trying to work behind reverse proxy, web.external-url and web.route-prefix

3,693 views

Skip to first unread message

Brian Candler

unread,

Aug 16, 2017, 10:35:59 AM8/16/17

to Prometheus Users

I am trying to set up prometheus (2.0.0-beta1) behind a simple Apache reverse proxy, which is not changing the path:

ProxyPass /prometheus http://10.85.179.50:9090/prometheus

ProxyPassReverse /prometheus http://10.85.179.50:9090/prometheus

My problem is setting up prometheus to work with this prefix, and I'm unclear on the purpose of the two separate settings.

(1) If I set both

--web.external-url=https://example.com/prometheus

--web.route-prefix=/prometheus

then I end up with a double redirect to /prometheus/prometheus/graph which then gives a 404 (details below).

(1a) If I set only

--web.external-url=https://example.com/prometheus

then I get the same behaviour. (Aside: this seems to be expected, given that ./prometheus --help says that web.route_prefix "Defaults to path of --web.external-url.")

(2) If I set

--web.external-url=https://example.com/prometheus

--web.route-prefix=/

then the server does not respond on /prometheus, but does respond on /graph

(3) If I set

--web.external-url=https://example.com/

--web.route-prefix=/prometheus

then /prometheus/graph responds as expected; however all the embedded CSS and action links are absolute links against /, not /prometheus.

(3a) If I set only

--web.route-prefix=/prometheus

then I get identical result to case (3). Aside: the string "https://example.com/" doesn't seem to appear anywhere in the output, so I don't see what difference --web.external-url makes anyway.

Any clues please? I might be able to rewrite the HTML dynamically with mod_proxy_html, but is this really necessary?

Thanks,

Brian.

---- case (1) ----

(Testing performed from the reverse proxy itself)

root@wrn-mon2:~# curl -Lv 10.85.179.50:9090/prometheus

* Trying 10.85.179.50...

* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)

> GET /prometheus HTTP/1.1

> Host: 10.85.179.50:9090

> User-Agent: curl/7.47.0

> Accept: */*

< HTTP/1.1 301 Moved Permanently

< Location: /prometheus/

< Date: Wed, 16 Aug 2017 14:13:45 GMT

< Content-Length: 47

< Content-Type: text/html; charset=utf-8

* Ignoring the response-body

* Connection #0 to host 10.85.179.50 left intact

* Issue another request to this URL: 'http://10.85.179.50:9090/prometheus/'

* Found bundle for host 10.85.179.50: 0x5592b2630f20 [can pipeline]

* Re-using existing connection! (#0) with host 10.85.179.50

* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)

> GET /prometheus/ HTTP/1.1

> Host: 10.85.179.50:9090

> User-Agent: curl/7.47.0

> Accept: */*

< HTTP/1.1 302 Found

< Location: /prometheus/prometheus/graph

< Date: Wed, 16 Aug 2017 14:13:45 GMT

< Content-Length: 51

< Content-Type: text/html; charset=utf-8

* Ignoring the response-body

* Connection #0 to host 10.85.179.50 left intact

* Issue another request to this URL: 'http://10.85.179.50:9090/prometheus/prometheus/graph'

* Found bundle for host 10.85.179.50: 0x5592b2630f20 [can pipeline]

* Re-using existing connection! (#0) with host 10.85.179.50

* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)

> GET /prometheus/prometheus/graph HTTP/1.1

> Host: 10.85.179.50:9090

> User-Agent: curl/7.47.0

> Accept: */*

< HTTP/1.1 404 Not Found

< Content-Type: text/plain; charset=utf-8

< X-Content-Type-Options: nosniff

< Date: Wed, 16 Aug 2017 14:13:45 GMT

< Content-Length: 19

404 page not found

* Connection #0 to host 10.85.179.50 left intact

---- case (2) ----

root@wrn-mon2:~# curl -Lv 10.85.179.50:9090/prometheus

* Trying 10.85.179.50...

* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)

> GET /prometheus HTTP/1.1

> Host: 10.85.179.50:9090

> User-Agent: curl/7.47.0

> Accept: */*

< HTTP/1.1 404 Not Found

< Content-Type: text/plain; charset=utf-8

< X-Content-Type-Options: nosniff

< Date: Wed, 16 Aug 2017 14:19:08 GMT

< Content-Length: 19

404 page not found

* Connection #0 to host 10.85.179.50 left intact

root@wrn-mon2:~# curl -Lv 10.85.179.50:9090/prometheus/

* Trying 10.85.179.50...

* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)

> GET /prometheus/ HTTP/1.1

> Host: 10.85.179.50:9090

> User-Agent: curl/7.47.0

> Accept: */*

< HTTP/1.1 404 Not Found

< Content-Type: text/plain; charset=utf-8

< X-Content-Type-Options: nosniff

< Date: Wed, 16 Aug 2017 14:19:13 GMT

< Content-Length: 19

404 page not found

* Connection #0 to host 10.85.179.50 left intact

root@wrn-mon2:~# curl -Lv 10.85.179.50:9090/graph

* Trying 10.85.179.50...

* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)

> GET /graph HTTP/1.1

> Host: 10.85.179.50:9090

> User-Agent: curl/7.47.0

> Accept: */*

< HTTP/1.1 200 OK

< Date: Wed, 16 Aug 2017 14:19:25 GMT

< Content-Type: text/html; charset=utf-8

< Transfer-Encoding: chunked

<!DOCTYPE html>

<head>

<title>Prometheus Time Series Collection and Processing Server</title>

... etc

---- case (3) ----

# curl -Lv 10.85.179.50:9090/prometheus/

* Trying 10.85.179.50...

* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)

> GET /prometheus/ HTTP/1.1

> Host: 10.85.179.50:9090

> User-Agent: curl/7.47.0

> Accept: */*

< HTTP/1.1 302 Found

< Location: /prometheus/graph

< Date: Wed, 16 Aug 2017 14:21:51 GMT

< Content-Length: 40

< Content-Type: text/html; charset=utf-8

* Ignoring the response-body

* Connection #0 to host 10.85.179.50 left intact

* Issue another request to this URL: 'http://10.85.179.50:9090/prometheus/graph'

* Found bundle for host 10.85.179.50: 0x561979564f20 [can pipeline]

* Re-using existing connection! (#0) with host 10.85.179.50

* Connected to 10.85.179.50 (10.85.179.50) port 9090 (#0)

> GET /prometheus/graph HTTP/1.1

> Host: 10.85.179.50:9090

> User-Agent: curl/7.47.0

> Accept: */*

< HTTP/1.1 200 OK

< Date: Wed, 16 Aug 2017 14:21:51 GMT

< Content-Type: text/html; charset=utf-8

< Transfer-Encoding: chunked

<!DOCTYPE html>

<head>

<title>Prometheus Time Series Collection and Processing Server</title>

...

<li><a href="/alerts">Alerts</a></li>

<li><a href="/graph">Graph</a></li>

<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Status <span class="caret"></span></a>

<li><a href="/status">Runtime & Build Information</a></li>

<li><a href="/flags">Command-Line Flags</a></li>

<li><a href="/config">Configuration</a></li>

<li><a href="/rules">Rules</a></li>

<li><a href="/targets">Targets</a></li>

Brian Candler

unread,

Aug 16, 2017, 10:40:38 AM8/16/17

to Prometheus Users

OK, I think it works properly with my config example (1) *as long as* the URL goes directly to /prometheus/graph

If they go to /prometheus then they are redirected to /prometheus/prometheus/graph and I think that's a bug.

Reply all

Reply to author

Forward

0 new messages