Slow ingest on my exporter

[Dongyi Zhang]

I built a new exporter named redfish_exporter. It takes about 10 seconds to ingest samples each time.

I have the following config:

- file_sd_configs:
  - files:
    - redfish.yml
  job_name: Redfish
  metrics_path: /metrics
  scrape_interval: 3m
  scrape_timeout: 180s
  relabel_configs:
    - source_labels: [__address__]
      regex: (.*)
      target_label: __param_target
      replacement: ${1}
    - source_labels: [__param_target]
      regex: (.*)
      target_label: instance
      replacement: ${1}
    - source_labels: []
      regex: .*
      target_label: __address__
      replacement: 127.0.0.1:10002  # redfish exporter.

redfish.yml:

- labels:
    __param_password: '12345678'
    __param_username: ADMIN
    exporter: redfish
    tid: 1
  targets:
  - 10.10.20.147
- labels:
    __param_password: adminadmin
    __param_username: ADMIN
    exporter: redfish
    tid: 2
  targets:
  - 10.10.20.177
- labels:
    __param_password: '12345678'
    __param_username: ADMIN
    exporter: redfish
    tid: 3
  targets:
  - 10.10.20.146
- labels:
    __param_password: '12345678'
    __param_username: ADMIN
    exporter: redfish
    tid: 4
  targets:
  - 10.10.20.167

The initial collection interval of the two monitoring processes is very long, about ten minutes. 

When I restart Prometheus, it takes at least 40 minutes to perform all monitoring processes for the first time.

[Brian Brazil]

This is not secure, anyone who has access to the Prometheus will be able to see the password.

Is your Prometheus generally overloaded? How many samples are being returned by these targets?

--

Brian Brazil

www.robustperception.io

[Dongyi Zhang]

在 2017年4月19日星期三 UTC+8下午2:47:50，Brian Brazil写道：

This is not secure, anyone who has access to the Prometheus will be able to see the password.

Eventually I will encrypt it.

 

 

Is your Prometheus generally overloaded? How many samples are being returned by these targets?

--

Brian Brazil

www.robustperception.io

 At the same time I also monitored four node_exporter, and the node_exporter collection is normal.

After 30 minutes of restarting Prometheus, the result is this:

Endpoint	State	Labels	Last Scrape
http://127.0.0.1:10002/metrics	UP	exporter="redfish" instance="10.10.20.146" tid="3"	6m43.498s
http://127.0.0.1:10002/metrics	UNKNOWN	exporter="redfish" instance="10.10.20.177" tid="2"	Never
http://127.0.0.1:10002/metrics	UP	exporter="redfish" instance="10.10.20.167" tid="4"	32m7.288s
http://127.0.0.1:10002/metrics	UP	exporter="redfish" instance="10.10.20.147" tid="1"	14m58.591s

After 10 minutes, all the programs are running normally, and then run at 'scrape_interval'.

[ge jin]

@Brian Brazil

We also have the same condition on my 30m per interval scraper, slow ingest on my exporter.

在 2017年4月19日星期三 UTC+8上午11:40:46，Dongyi Zhang写道：

[万福余]

HI @Brian Brazil
I wonder how long it takes to start scraping targets which is  added through file discovery, is there any accurate relationship between the start time and the scrape Interval of that job ??? 

This is very important to me, to my team.

Please answer me if you see this comment  🙏🏻

在 2017年4月19日星期三 UTC+8下午2:47:50，Brian Brazil写道：

On 19 April 2017 at 04:40, Dongyi Zhang <txd...@163.com> wrote:

--

Brian Brazil

www.robustperception.io

[Julius Volz]

Target changes to file SD targets should be picked up immediately. Then the next scrape should happen within the configured scrape interval.

If your Prometheus takes 30 minutes to scrape new targets, it is either overloaded or something else is very wrong with it.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/77848389-f8d9-4e54-a91d-8cf8c47a7d24%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ge jin]

Hi, Julius Volz

Thanks for your reply.  And I suggest there should be any configurable parameter to control If it can immediately scrapes.

You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/AAHYP7XxMY4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BT6YoyTYb3MGj%2BB%2B6osx5vS0ZM8%3DZtP%2Bcdtt_B746jQGgxSyQ%40mail.gmail.com.

[Dongyi Zhang]

The configured scrape interval is 30 minutes. It was too long to scrape new targets, and I could not control the time of the first scrape.

After the configuration file is valid, can I manually trigger the next scrape immediately?

在 2017年4月26日星期三 UTC+8上午10:05:35，Julius Volz写道：

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Julius Volz]

On Wed, Apr 26, 2017 at 8:42 AM, Dongyi Zhang <txd...@163.com> wrote:

The configured scrape interval is 30 minutes. It was too long to scrape new targets, and I could not control the time of the first scrape.

30 minutes is too long of a scrape time in general. With Prometheus's default staleness interval of 5 minutes (meaning a sample will only be "visible" for a query if it is at most 5 minutes older than the query timestamp), you will have to scrape at *least* every 5 minutes to do useful things with the data.

After the configuration file is valid, can I manually trigger the next scrape immediately?

There is no way to trigger an immediate scrape of all targets of a scrape configuration / job. In fact, Prometheus intentionally spreads out the scrapes for the targets of a given job across the entire scrape interval to get to an even scrape load instead of period spikey loads.

 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3d39a07e-d7f6-4be6-83f9-2ddaa7b8bd39%40googlegroups.com.