High scaping interval & timeout
76 views
Skip to first unread message
Ebenolt
Apr 8, 2020, 4:54:12 AM4/8/20
to Prometheus Users
Hello,
I'm new to Prometheus, i'd like to monitor wireless AP (Cisco Meraki ones) through SNMP, and Cisco made a centralized SNMP system, you only need to scrape one target and it will drop you metrics for all your APs.
I currently got 20 APs to monitor, so scraping takes a while (request times in seconds: https://share.ebenolt.com/08_04_20_0ee981.png )
Scrape interval can't go lower than scrape timeout (502 Bad Gateway) , but scrape interval can't go higher than 65s (context deadline exceeded).
so my current job config times:
scrape_timeout: 60s
scrape_interval: 65s
Is there any way to allow Prometheus to scrape with a timeout ~= 120s ?
Thanks
Stuart Clark
Apr 8, 2020, 4:56:11 AM4/8/20
to Ebenolt, Prometheus Users
You can set your scrape interval to 2 minutes if you want...
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
--
Stuart Clark
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
--
Stuart Clark
Ebenolt
Apr 8, 2020, 4:57:16 AM4/8/20
to Prometheus Users
Hey
Scrape interval can't go higher than 65s or i'll get a "context deadline exceeded"
Scrape interval can't go higher than 65s or i'll get a "context deadline exceeded"
Le mercredi 8 avril 2020 10:56:11 UTC+2, Stuart Clark a écrit :
You can set your scrape interval to 2 minutes if you want...
On 2020-04-08 09:54, Ebenolt wrote:
> Hello,
> I'm new to Prometheus, i'd like to monitor wireless AP (Cisco Meraki
> ones) through SNMP, and Cisco made a centralized SNMP system, you only
> need to scrape one target and it will drop you metrics for all your
> APs.
> I currently got 20 APs to monitor, so scraping takes a while (request
> times in seconds: https://share.ebenolt.com/08_04_20_0ee981.png )
> Scrape interval can't go lower than scrape timeout (502 Bad Gateway) ,
> but scrape interval can't go higher than 65s (context deadline
> exceeded).
> so my current job config times:
>
> scrape_timeout: 60s
> scrape_interval: 65s
>
> Is there any way to allow Prometheus to scrape with a timeout ~= 120s
> ?
> Thanks
>
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to promethe...@googlegroups.com.
Stuart Clark
Apr 8, 2020, 4:59:48 AM4/8/20
to Ebenolt, Prometheus Users
On 2020-04-08 09:57, Ebenolt wrote:
> Hey
> Scrape interval can't go higher than 65s or i'll get a "context
> deadline exceeded"
>
Did you also increase the timeout?
> Hey
> Scrape interval can't go higher than 65s or i'll get a "context
> deadline exceeded"
>
> Le mercredi 8 avril 2020 10:56:11 UTC+2, Stuart Clark a écrit :
>
>> You can set your scrape interval to 2 minutes if you want...
>>
>> On 2020-04-08 09:54, Ebenolt wrote:
>>> Hello,
>>> I'm new to Prometheus, i'd like to monitor wireless AP (Cisco
>> Meraki
>>> ones) through SNMP, and Cisco made a centralized SNMP system, you
>> only
>>> need to scrape one target and it will drop you metrics for all
>> your
>>> APs.
>>> I currently got 20 APs to monitor, so scraping takes a while
>> (request
>>> times in seconds: https://share.ebenolt.com/08_04_20_0ee981.png
>>> Scrape interval can't go lower than scrape timeout (502 Bad
>> Gateway) ,
>>> but scrape interval can't go higher than 65s (context deadline
>>> exceeded).
>>> so my current job config times:
>>>
>>> scrape_timeout: 60s
>>> scrape_interval: 65s
>>>
>>> Is there any way to allow Prometheus to scrape with a timeout ~=
>> 120s
>>> ?
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>> send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>>
>>
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
>> [2]
>> Gateway) ,
>>> but scrape interval can't go higher than 65s (context deadline
>>> exceeded).
>>> so my current job config times:
>>>
>>> scrape_timeout: 60s
>>> scrape_interval: 65s
>>>
>>> Is there any way to allow Prometheus to scrape with a timeout ~=
>> 120s
>>> ?
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>> send
>>> an email to promethe...@googlegroups.com.
>>> To view this discussion on the web visit
>>>
>>
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
>>> [1].
>>>
>>>
>>> Links:
>>> ------
>>> [1]
>>>
>>
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
>> [3]
>>>
>>>
>>> Links:
>>> ------
>>> [1]
>>>
>>
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
>>
>> --
>> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-use...@googlegroups.com.
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/1c88d8d7-e0a6-48a5-9fa3-fc24b90c716b%40googlegroups.com
> [4].
>
>
> Links:
> ------
> [1] https://share.ebenolt.com/08_04_20_0ee981.png
> [2]
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
> [3]
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
> [4]
> https://groups.google.com/d/msgid/prometheus-users/1c88d8d7-e0a6-48a5-9fa3-fc24b90c716b%40googlegroups.com?utm_medium=email&utm_source=footer
--
Stuart Clark
Julien Pivotto
Apr 8, 2020, 5:00:15 AM4/8/20
to Ebenolt, Prometheus Users
On 08 Apr 01:57, Ebenolt wrote:
> Hey
> Scrape interval can't go higher than 65s or i'll get a "context deadline
> exceeded"
can you provide more details? that should not happen.
> Hey
> Scrape interval can't go higher than 65s or i'll get a "context deadline
> exceeded"
Are you reloading configuration often? or using service descovery?
which version of prometheus are you using?
>
> Le mercredi 8 avril 2020 10:56:11 UTC+2, Stuart Clark a écrit :
> >
> > You can set your scrape interval to 2 minutes if you want...
> >
> >
> > On 2020-04-08 09:54, Ebenolt wrote:
> > > Hello,
> > > I'm new to Prometheus, i'd like to monitor wireless AP (Cisco Meraki
> > > ones) through SNMP, and Cisco made a centralized SNMP system, you only
> > > need to scrape one target and it will drop you metrics for all your
> > > APs.
> > > I currently got 20 APs to monitor, so scraping takes a while (request
> > > times in seconds: https://share.ebenolt.com/08_04_20_0ee981.png )
> > > Scrape interval can't go lower than scrape timeout (502 Bad Gateway) ,
> > > but scrape interval can't go higher than 65s (context deadline
> > > exceeded).
> > > so my current job config times:
> > >
> > > scrape_timeout: 60s
> > > scrape_interval: 65s
> > >
> > > Is there any way to allow Prometheus to scrape with a timeout ~= 120s
> > > ?
> > > Thanks
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "Prometheus Users" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > To view this discussion on the web visit
> > >
> > https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
> > > [1].
> > >
> > >
> > > Links:
> > > ------
> > > [1]
> > >
> > https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> > --
> > Stuart Clark
> >
>
> > >
> > https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
> > > [1].
> > >
> > >
> > > Links:
> > > ------
> > > [1]
> > >
> > https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> > --
> > Stuart Clark
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1c88d8d7-e0a6-48a5-9fa3-fc24b90c716b%40googlegroups.com.
--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu
Ebenolt
Apr 8, 2020, 5:01:47 AM4/8/20
to Prometheus Users
Yes,
I tried to increase timeout and interval independently, and both at the same time, nothing works higher than 65s
I tried to increase timeout and interval independently, and both at the same time, nothing works higher than 65s
Stuart Clark
Apr 8, 2020, 5:03:08 AM4/8/20
to Ebenolt, Prometheus Users
On 2020-04-08 10:01, Ebenolt wrote:
> Yes,
> I tried to increase timeout and interval independently, and both at
> the same time, nothing works higher than 65s
What happens if you manually hit the exporter endpoint (e.g. curl or a
> Yes,
> I tried to increase timeout and interval independently, and both at
> the same time, nothing works higher than 65s
browser)?
>>> [2]
>>>
>>
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
>> [2]
>>> [3]
>>>
>>
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
>> [5]
>>> [4]
>>>
>>
> https://groups.google.com/d/msgid/prometheus-users/1c88d8d7-e0a6-48a5-9fa3-fc24b90c716b%40googlegroups.com?utm_medium=email&utm_source=footer
>> [6]
>>
>> --
>> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-use...@googlegroups.com.
>> --
>> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/37a9c933-e3b1-490f-9426-2040dd52f4d9%40googlegroups.com
> [7].
>
>
> Links:
> ------
> [1] https://share.ebenolt.com/08_04_20_0ee981.png
> [2]
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
> [3]
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
> [4]
> https://groups.google.com/d/msgid/prometheus-users/1c88d8d7-e0a6-48a5-9fa3-fc24b90c716b%40googlegroups.com
> [5]
>
> Links:
> ------
> [1] https://share.ebenolt.com/08_04_20_0ee981.png
> [2]
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com
> [3]
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
> [4]
> https://groups.google.com/d/msgid/prometheus-users/1c88d8d7-e0a6-48a5-9fa3-fc24b90c716b%40googlegroups.com
> https://groups.google.com/d/msgid/prometheus-users/9e83f657-3b6b-4c28-a629-67641d8aa7fb%40googlegroups.com?utm_medium=email&utm_source=footer
> [6]
> https://groups.google.com/d/msgid/prometheus-users/1c88d8d7-e0a6-48a5-9fa3-fc24b90c716b%40googlegroups.com?utm_medium=email&utm_source=footer
> [7]
> https://groups.google.com/d/msgid/prometheus-users/37a9c933-e3b1-490f-9426-2040dd52f4d9%40googlegroups.com?utm_medium=email&utm_source=footer
--
Stuart Clark
Ebenolt
Apr 8, 2020, 5:03:58 AM4/8/20
to Prometheus Users
Hello,
I reload config (prometheus as a service), i restart it everytime i change config.
I reload config (prometheus as a service), i restart it everytime i change config.
I'm using version 2.16.0
and i don't know how to check service discovery.
Thanks
> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
Ebenolt
Apr 8, 2020, 5:05:26 AM4/8/20
to Prometheus Users
I can get metrics, it can takes more than 90s but it works, and through snmpwalk i put a graph on the issue 1st message, it works too with high times
Reply all
Reply to author
Forward
0 new messages