Re: [prometheus-users] How is "scraping" of dual-stack target works? (IPv4+IPv6)

[Brian Brazil]

On 9 December 2017 at 13:13, 'Evgeniy Kostenko' via Prometheus Users <promethe...@googlegroups.com> wrote:

Hello.

Could someone explain (or point where to dig) how Prometheus chooses IPv6 over IPv4 (and vice versa) for dual-stack target? We have some temporary IPv6-related issues between one of our prometheus and scraping target. And this issue is causing bunch of false positive alerts "target is down for XX minutes". To be clear, only IPv6 interconnection is down, but IPv4 works just fine. And it's not a "packet loss", it's 100% unavailability via IPv6 between Europe (prometheus) and Asia (scrape target), but only via IPv6. I know about "failure domain" and common recommendation "place Prometheus near your targets", but it's our "usage scenario" for now.

Does Prometheus chooses random transport on each attempt? I.e. one round it's a connection via IPv4 and next one could be over IPv6? Or it's IPv4 and only then IPv6 as fallback? Or there is no such logic in Prometheus and it's totally in underlying "standart" Go classes (and platform specifics like libc)?

Prometheus doesn't have any special logic here, it does whatever Go does - which is use v6 if it's available otherwise use v4. If you want to use a specific address, you should use relabelling to supply that as the __address__ label for the target. This will avoid the DNS lookup.

Brian

 

Thanks.

PS: BTW, /etc/gai.conf is "default" without any changes.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/49cd8f6f-9515-4951-8380-13e217c9a9ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Ben Kochie]

One additional comment to Brian's,  Prometheus is not really designed for long distance scrape tarets.  The intended use is to have the Prometheus server inside the same local LAN as the targets.  Like you noticed, having long-distance scrape targets causes you to end up having the network be a factor in monitoring the metrics.

On Sat, Dec 9, 2017 at 2:13 PM, 'Evgeniy Kostenko' via Prometheus Users <promethe...@googlegroups.com> wrote:

Hello.

Could someone explain (or point where to dig) how Prometheus chooses IPv6 over IPv4 (and vice versa) for dual-stack target? We have some temporary IPv6-related issues between one of our prometheus and scraping target. And this issue is causing bunch of false positive alerts "target is down for XX minutes". To be clear, only IPv6 interconnection is down, but IPv4 works just fine. And it's not a "packet loss", it's 100% unavailability via IPv6 between Europe (prometheus) and Asia (scrape target), but only via IPv6. I know about "failure domain" and common recommendation "place Prometheus near your targets", but it's our "usage scenario" for now.

Does Prometheus chooses random transport on each attempt? I.e. one round it's a connection via IPv4 and next one could be over IPv6? Or it's IPv4 and only then IPv6 as fallback? Or there is no such logic in Prometheus and it's totally in underlying "standart" Go classes (and platform specifics like libc)?

Thanks.

PS: BTW, /etc/gai.conf is "default" without any changes.