Prometheus setting off Checkpoint firewalls

[Andy Kruta]

My apologies if this has been answered already, but I've looked through the configs for a setting that would allow me to define how many targets can be scraped at once and came up empty.  Essentially, what I've got going on here is my prometheus is being blocked by my checkpoint firewalls (for between 10-20 minutes) due to the number of targets that it's scraping at once ( because of the Suspicious Activity Monitoring module.)  

My configuration:

• Central Prometheus server
• Multiple Data Centers 
• SNMP monitored by local SNMP Exporters local to each datacenter
• Windows / Linux boxes monitored via Telegraf scraping
• Various other exporters (generally on the Prometheus server itself unless large number of targets in remote datacenter)

Unfortunately, I've already talked to Checkpoint and made all of the changes they recommend without any improvement.  I've also already increased the scrape interval (currently sitting at 4m) but the scrapes appear to all be happening within say a minute of each other.  This results in the checkpoints blocking the activity and the targets appearing to be down.  

My only other idea to resolve this is to increase the time in the alert configuration to give additional time so that while the firewall is still blocking the traffic, we don't get the alerts.  This feels moronic though, and I'm holding it back as a "just keep my mailbox empty" route. 

Has anyone come up with a clever way to work around this?

Thanks,

Andy

[Brian Brazil]

Prometheus already spreads the scrapes across time, this is fundamentally an issue with your firewall blocking scrapes.  The generally recommended architecture would be to have a Prometheus inside each datacenter, rather than trying to scrape everything across datacenters.

--

Brian Brazil

www.robustperception.io

[Andy Kruta]

Although I wish it was, unfortunately, it's not an option.  The good news is that I don't have to deal with the checkpoints much longer.  The bad news is that until I get rid of them, I have to silence the noise.