proxy_url not working in azure service discoverer

[Brett Jacobson]

I am trying to set the proxy_url param on a scrape config job that uses azure_sd_configs for azure service discovery.  It appears that the azure service discovery module does not respect this setting from the scrape_config block as documented here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config.

It is my belief after perusing the code, that the cause of the problem is this service discoverer uses https://github.com/Azure/go-autorest instead of net/http directly, thus I think the proxy_url config setting is "lost".

I have proven my config works when it is in a prometheus that is *NOT* behind our corporate proxy (with no proxy_url value set atall of course), but moving the config to a prometheus instance behind the corp proxy and setting proxy_url does not work.  The only symptom is after 7 minutes, an error message appears in the logs:

level=error ts=2020-11-06T21:00:31.200Z caller=refresh.go:98 component="discovery manager scrape" discovery=azure msg="Unable to refresh target groups" err="could not get virtual machines: could not list virtual machines: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/xxxxxx/providers/Microsoft.Compute/virtualMachines?api-version=2018-10-01: StatusCode=0 -- Original Error: adal: Failed to execute the refresh request. Error = 'Post \"https://login.microsoftonline.com/xxxxx/oauth2/token?api-version=1.0\": read tcp <ipaddresses>: read: connection reset by peer'"

This is a sign that the net connection to authenticate to microsoft failed to navigate the proxy.

Does anyone have any experience with this?

Can anyone confirm if this is a bug?

[Brian Candler]

On Friday, 6 November 2020 21:16:18 UTC, Brett Jacobson wrote:

I am trying to set the proxy_url param on a scrape config job that uses azure_sd_configs for azure service discovery.  It appears that the azure service discovery module does not respect this setting from the scrape_config block as documented here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config.

https://groups.google.com/d/msg/prometheus-users/HdK6pr4ugwo/9f8KUFENCAAJ

As I understand it, the proxy_url in the scrape job is only used when scraping the target, not when talking to the service discovery.

Looking at the documentation, some of the service discovery mechanisms do support their own "proxy_url" setting, but azure_sd isn't one of them.  If the underlying azure client library doesn't provide a proxy_url, that's probably why prometheus can't provide it either.  So you'd first need to make a PR on the underlying library, and get that merged, before doing the same for prometheus itself.

[Brian Brazil]

The preferred way to do it would be to be able to pass in a Transport, as then our library for all the standard http options can be used rather than having to re-implement them individually.

--

Brian Brazil

www.robustperception.io

[Brett Jacobson]

Okay I'll file a bug report in github.

[Durga Prasad Kommareddy]

Hi All,

Is there any progress/solution or work around for this issue ? fro Service discovery with proxy

[Julien Pivotto]

Yes

Prometheus 2.43 has added a few options that you could look into:

[FEATURE] HTTP client: Add no_proxy to exclude URLs from proxied
requests.
[FEATURE] HTTP client: Add proxy_from_enviroment to read proxies from
env variables.

On 11 Apr 06:01, Durga Prasad Kommareddy wrote:
> Hi All,
>
> Is there any progress/solution or work around for this issue ? fro Service
> discovery with proxy
>
> On Saturday, November 7, 2020 at 11:37:48 AM UTC-5 Brett Jacobson wrote:
>
> > Okay I'll file a bug report in github.
> >
> > On Saturday, November 7, 2020 at 3:49:32 AM UTC-6 Brian Brazil wrote:
> >
> >> On Sat, 7 Nov 2020 at 08:21, Brian Candler <b.ca...@pobox.com> wrote:
> >>
> >>> On Friday, 6 November 2020 21:16:18 UTC, Brett Jacobson wrote:
> >>>>
> >>>> I am trying to set the proxy_url param on a scrape config job that uses
> >>>> azure_sd_configs for azure service discovery. It appears that the azure
> >>>> service discovery module does not respect this setting from the
> >>>> scrape_config block as documented here:
> >>>> https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
> >>>> .
> >>>>
> >>>>
> >>> https://groups.google.com/d/msg/prometheus-users/HdK6pr4ugwo/9f8KUFENCAAJ
> >>>
> >>> As I understand it, the proxy_url in the scrape job is only used when
> >>> scraping the target, not when talking to the service discovery.
> >>>
> >>> Looking at the documentation

> >>> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/>,

> >>> some of the service discovery mechanisms do support their own "proxy_url"
> >>> setting, but azure_sd

> >>> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#azure_sd_config>

> >>> isn't one of them. If the underlying azure client library doesn't provide
> >>> a proxy_url, that's probably why prometheus can't provide it either. So
> >>> you'd first need to make a PR on the underlying library, and get that
> >>> merged, before doing the same for prometheus itself.
> >>>
> >>
> >> The preferred way to do it would be to be able to pass in a Transport, as
> >> then our library for all the standard http options can be used rather than
> >> having to re-implement them individually.
> >>
> >> --
> >> Brian Brazil
> >> www.robustperception.io
> >>
> >
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/19ab672f-a81c-498b-9e15-eed2ba8ab715n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie