hitcount by ipaddress

[deln...@gmail.com]

Working on converting custom script that monitors number of hits per specific ip and alerts accordingly. Can use script exporter/text collector but seems that Prometheus is not a right tool to detect and notify on the ip address itself. What approach you're taking in these situations?

[Brian Candler]

Prometheus isn't usable here because of the cardinality explosion you'll get if you treat every source IP address as a different label.

You need a log processing system - e.g loki, or one of the many log processing systems with an elasticsearch backend.  Or write logs into a SQL database.  Or use Kafka and KSQL.  You can even write logs into a CSV file and do direct SQL queries on that file: http://harelba.github.io/q/

If you're trying to get an alert if more than a certain number of hits come from the same IP address then you could also consider various "Host Intrusion Detection Systems" (HIDS) like ossec, wazuh, sagan.  I looked at all those and found them to be incredibly crufty and broken.  Even if you have to write your own bit of code which hangs off the back of your log processing pipeline, it will likely be more dependable and maintainable.