Remote Write Server Side Traffic Mirroring + Obfuscator for Prometheus Ecosystem
106 views
Skip to first unread message
Bartłomiej Płotka
Apr 20, 2020, 2:24:34 PM4/20/20
to Prometheus Users
Hi!
This question is not strictly related to Prometheus, but rather to server-side Remote Write APIs.
We are looking at how to have more realistic staging environments for servers like that. In order to achieve so, we want to "mirror" / "fork" portion of production remote write traffic to other clusters APIs (e.g staging or dev environment).
As part of this mirroring, data has to be potentially obfuscated to avoid leaking of sensitive data, but also without totally changing the characteristic of data (e.g same number labels, labels values/names with the same amount of characters, etc).
In the future, we could add some more advanced features if needed (e.g load balancing).
Wonder if anyone in the community had been working on something like that already and has something to share/is already shared?
ProxySQL is something like that but in the SQL world. Would be awesome to have the same for remote write (and Query API as well I guess, but let's think about it in a separate thread) (:
Some discussion on Thanos project: https://github.com/thanos-io/thanos/issues/2480
Please help if you know or have worked on something like this (: Would be a nice community Project if nothing exists!
This question is not strictly related to Prometheus, but rather to server-side Remote Write APIs.
We are looking at how to have more realistic staging environments for servers like that. In order to achieve so, we want to "mirror" / "fork" portion of production remote write traffic to other clusters APIs (e.g staging or dev environment).
As part of this mirroring, data has to be potentially obfuscated to avoid leaking of sensitive data, but also without totally changing the characteristic of data (e.g same number labels, labels values/names with the same amount of characters, etc).
In the future, we could add some more advanced features if needed (e.g load balancing).
Wonder if anyone in the community had been working on something like that already and has something to share/is already shared?
ProxySQL is something like that but in the SQL world. Would be awesome to have the same for remote write (and Query API as well I guess, but let's think about it in a separate thread) (:
Some discussion on Thanos project: https://github.com/thanos-io/thanos/issues/2480
Please help if you know or have worked on something like this (: Would be a nice community Project if nothing exists!
Kind Regards,
Bartek
Aliaksandr Valialkin
Apr 20, 2020, 2:38:42 PM4/20/20
to Bartłomiej Płotka, Prometheus Users
Such a mirroring can be done with vmagent - just configure multiple `-remoteWrite.url` targets with distinct `-remoteWrite.urlRelabelConfig` configs for obfuscation and filtering. The final system will look like the following:
-> remote target1 (prod)
Prometheus -> vmagent -> filtering -> remote target2 (staging)
-> obfuscation -> remote target3 (dev)
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMssQwYa3kW8UMPtJ2PuW8%3Dd8kWB-sz1E99D20ODn28KZTb%2BNQ%40mail.gmail.com.
Best Regards,
Aliaksandr
Aliaksandr
Bartłomiej Płotka
Apr 20, 2020, 2:44:39 PM4/20/20
to Aliaksandr Valialkin, Prometheus Users
Thanks, Aliaksandr!
So vmagent on top of scraping ALSO receive remote write API? 😱 What it CANNOT do? =D
It looks indeed that feature-wise it is what we meant, just I am a bit concerned about one binary that enables everything. It might be quite hard work to maintain it... You must be some kind of superhuman Aliaksandr! (: Definitely will take a look, thanks. (:
Kind Regards,
It looks indeed that feature-wise it is what we meant, just I am a bit concerned about one binary that enables everything. It might be quite hard work to maintain it... You must be some kind of superhuman Aliaksandr! (: Definitely will take a look, thanks. (:
Kind Regards,
Bartek
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPbKnmBEvDbThrr1cx9FgPhZRVrJqeS2%3DsgEOZguoMWH6FS7BA%40mail.gmail.com.
Aliaksandr Valialkin
Apr 20, 2020, 3:50:31 PM4/20/20
to Bartłomiej Płotka, Prometheus Users
> just I am a bit concerned about one binary that enables everything. It might be quite hard work to maintain it.
vmagent is just easy to use metrics proxy, which performs the following tasks:
- Accepts data via various popular ingestion protocols (Prometheus remote_write, Influx line protocol, Graphite, OpenTSDB, CSV). Additionally, it can scrape Prometheus targets.
- Augments and filters the accepted data with Prometheus-compatible relabeling.
- Pushes the filtered data to the configured remote storage targets via Prometheus remote_write protocol. Additional per-target relabeling can be applied to data before sending it to each remote storage target.
vmagent uses independent file-base buffers for each configured remote storage target, so it may buffer the data locally until temporarily unavailable remote storage target becomes available again.
Best Regards,
Aliaksandr
Aliaksandr
Goutham Veeramachaneni
Apr 21, 2020, 3:07:49 AM4/21/20
to Prometheus Users
Hi Bartek,
Thanks for bringing this up! This is great news, infact we were going to look into the same in Cortex as well! For testing the new blocks based storage we were going to mirror writes from a subset of our users to the new storage engine and run both in parallel. We already have a query-tee tool which mirrors queries, next is mirroring the write path for us. While we don't need the obfuscation let us all collaborate to create a proxy/mirror. And thanks for sharing the vmagent Aliaksandr, will definitely look into it.
Thanks,
Goutham.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMssQwYa3kW8UMPtJ2PuW8%3Dd8kWB-sz1E99D20ODn28KZTb%2BNQ%40mail.gmail.com.
--Best Regards,--
Aliaksandr
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPbKnmBEvDbThrr1cx9FgPhZRVrJqeS2%3DsgEOZguoMWH6FS7BA%40mail.gmail.com.
--Best Regards,
Aliaksandr
Reply all
Reply to author
Forward
0 new messages