How to debug prometheus When we lose data?

[lx]

Hi all:

       I'm new to use the Prometheus, I send data by Pushgateway and display data by Grafana.

I find I lose data for 3 minutes when I see grafana in two days. I think I send the data and the network is ok.

How to debug this case? 

these versions:

#########################

prometheus-2.42.0-rc.0.linux-amd64

pushgateway-1.5.1.linux-amd64

grafana-9.3.6
#########################

Prometheus config:

#########################

  - job_name: "pushgatewayoverflow"
    #honor_labels: true
    scrape_interval: 30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
    static_configs:
      - targets: ["127.0.0.1:9091"]

  - job_name: "pushgatewayportflow"
    #honor_labels: true
    scrape_interval: 30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
    static_configs:
      - targets: ["127.0.0.1:9092"]

  - job_name: "pushgatewaybuflow"
    #honor_labels: true
    scrape_interval: 30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
    static_configs:
      - targets: ["127.0.0.1:9093"]

#########################

If I use only one PushgateWay, Data loss will become more frequent. This is Why?

Thank you.

[Brian Candler]

Whatever the question is, pushgateway is usually the wrong answer.

Pushgateway has only one legitimate use case, which is to have a "sticky" metric which persists after the process which generated it has terminated. This could be the status of the most recent execution of a cronjob, for instance.

It's *not* intended as a mechanism for pushing data in general into Prometheus.  To do that, you need to use the Remote Write protocol.

The question is, why doesn't regular scraping of the data source work for you, which makes you think you need the pushgateway?  If the reason is because of firewalls or NAT, then the easiest way to use the Remote Write protocol is to run a second instance of Prometheus in Agent mode, behind the firewall, scraping a normal exporter.

local exporter <--- Prometheus Agent --------------[remote write protocol] ------------> Central Prometheus