On improving scrape performance in multi-process mode

[Karan Chaudhary]

Hi, i'm looking into improving scrape latency that reaches 1.8s when number of samples is high. One of the hot spots py-spy showed was json.decoding keys when reading from mmap files. 

I've noticed that _key_cache's scope is limited to per scrape, not across scrapes to keep cost of json decoding in check per scrape, in multiprocess mode. However i noticed that samples are only appended to prom db files and apart from values, the key itself is never mutated, which means the json decoded key can itself be cached throughout the lifetime of the process. 

On our data set, picked up from services that shows worst scrape latency (cp /prom/* local and run local benchmarks on it), it reduces latency by around 9-10%.

I believe we can get even higher gains by using similar strategy at other places, like mb caching utf-8 decoded keys when reading samples from files. 

Tradeoff is increased cache memory that can be capped to a reasonable value. 

But before that, it'd be great it can be confirmed that caching is safe given samples keys are immutable once written to mmap files. 

Thanks,

Karan

[Ben Kochie]

I would recommend taking a look at what GitLab did to improve the performance of multi-process metrics for Ruby.

https://gitlab.com/gitlab-org/ruby/gems/prometheus-client-mmap

They used a Rust binary to handle the raw file handling. Perhaps we could do a similar wheel binary for client_python.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/prometheus-developers/25fbedc0-4aaf-44c2-a3ea-e4144831a757n%40googlegroups.com.