Prometheus Remote Write "compliance suite"

[Tom Wilkie]

Hello Prometheans!

In preparation for my PromCon talk, I build a little test suite the checks various Prometheus Remote Write compatible "agents":

https://github.com/tomwilkie/remote-write-compliance

There are ~13 pretty simple tests, and each one exports some metrics, forks off the "agent" configured to scrape those metrics and send remote write requests back to the test, and then validates the results it receives.  Its still a WIP but there are a few interesting results (including a bug in the Grafana Agent! that has already been fixed...)

I guess I'm sending this email both as an FYI, to see if anyone has any questions, and to see what people think of transferring this to the prometheus-community github org?

Cheers

Tom

    --- FAIL: TestRemoteWrite/vmagent (0.00s)

        --- FAIL: TestRemoteWrite/vmagent/Invalid (10.21s)

        --- PASS: TestRemoteWrite/vmagent/Histogram (10.21s)

        --- FAIL: TestRemoteWrite/vmagent/NameLabel (10.21s)

        --- PASS: TestRemoteWrite/vmagent/SortedLabels (10.25s)

        --- PASS: TestRemoteWrite/vmagent/Counter (10.25s)

        --- PASS: TestRemoteWrite/vmagent/Gauge (10.25s)

        --- PASS: TestRemoteWrite/vmagent/InstanceLabel (10.27s)

        --- PASS: TestRemoteWrite/vmagent/Up (10.32s)

        --- PASS: TestRemoteWrite/vmagent/EmptyLabels (10.18s)

        --- FAIL: TestRemoteWrite/vmagent/Staleness (10.27s)

        --- PASS: TestRemoteWrite/vmagent/Timestamp (10.27s)

        --- FAIL: TestRemoteWrite/vmagent/RepeatedLabels (10.30s)

        --- PASS: TestRemoteWrite/vmagent/JobLabel (10.32s)

    --- PASS: TestRemoteWrite/prometheus (0.00s)

        --- PASS: TestRemoteWrite/prometheus/Timestamp (10.14s)

        --- PASS: TestRemoteWrite/prometheus/NameLabel (10.14s)

        --- PASS: TestRemoteWrite/prometheus/Counter (10.14s)

        --- PASS: TestRemoteWrite/prometheus/EmptyLabels (10.14s)

        --- PASS: TestRemoteWrite/prometheus/Staleness (10.14s)

        --- PASS: TestRemoteWrite/prometheus/InstanceLabel (10.14s)

        --- PASS: TestRemoteWrite/prometheus/RepeatedLabels (10.14s)

        --- PASS: TestRemoteWrite/prometheus/SortedLabels (10.17s)

        --- PASS: TestRemoteWrite/prometheus/Gauge (10.07s)

        --- PASS: TestRemoteWrite/prometheus/JobLabel (10.07s)

        --- PASS: TestRemoteWrite/prometheus/Up (10.07s)

        --- PASS: TestRemoteWrite/prometheus/Invalid (10.08s)

        --- PASS: TestRemoteWrite/prometheus/Histogram (10.08s)

    --- FAIL: TestRemoteWrite/otelcollector (0.00s)

        --- PASS: TestRemoteWrite/otelcollector/Timestamp (10.02s)

        --- FAIL: TestRemoteWrite/otelcollector/NameLabel (10.02s)

        --- PASS: TestRemoteWrite/otelcollector/EmptyLabels (10.02s)

        --- FAIL: TestRemoteWrite/otelcollector/Staleness (10.02s)

        --- FAIL: TestRemoteWrite/otelcollector/Up (10.02s)

        --- FAIL: TestRemoteWrite/otelcollector/InstanceLabel (10.03s)

        --- FAIL: TestRemoteWrite/otelcollector/Counter (10.03s)

        --- FAIL: TestRemoteWrite/otelcollector/RepeatedLabels (10.04s)

        --- PASS: TestRemoteWrite/otelcollector/Gauge (10.03s)

        --- PASS: TestRemoteWrite/otelcollector/SortedLabels (10.03s)

        --- FAIL: TestRemoteWrite/otelcollector/Invalid (10.03s)

        --- FAIL: TestRemoteWrite/otelcollector/JobLabel (10.03s)

        --- FAIL: TestRemoteWrite/otelcollector/Histogram (10.03s)

    --- FAIL: TestRemoteWrite/telegraf (0.00s)

        --- PASS: TestRemoteWrite/telegraf/Counter (10.03s)

        --- FAIL: TestRemoteWrite/telegraf/RepeatedLabels (10.03s)

        --- PASS: TestRemoteWrite/telegraf/SortedLabels (10.03s)

        --- FAIL: TestRemoteWrite/telegraf/EmptyLabels (10.03s)

        --- PASS: TestRemoteWrite/telegraf/Timestamp (10.03s)

        --- FAIL: TestRemoteWrite/telegraf/Staleness (10.03s)

        --- PASS: TestRemoteWrite/telegraf/Histogram (10.03s)

        --- FAIL: TestRemoteWrite/telegraf/NameLabel (10.03s)

        --- PASS: TestRemoteWrite/telegraf/JobLabel (10.03s)

        --- FAIL: TestRemoteWrite/telegraf/InstanceLabel (10.03s)

        --- FAIL: TestRemoteWrite/telegraf/Up (10.03s)

        --- PASS: TestRemoteWrite/telegraf/Gauge (10.03s)

        --- FAIL: TestRemoteWrite/telegraf/Invalid (10.03s)

    --- PASS: TestRemoteWrite/grafana (0.00s)

        --- PASS: TestRemoteWrite/grafana/Histogram (10.17s)

        --- PASS: TestRemoteWrite/grafana/InstanceLabel (10.17s)

        --- PASS: TestRemoteWrite/grafana/Invalid (10.18s)

        --- PASS: TestRemoteWrite/grafana/JobLabel (10.19s)

        --- PASS: TestRemoteWrite/grafana/Up (10.19s)

        --- PASS: TestRemoteWrite/grafana/Timestamp (10.19s)

        --- PASS: TestRemoteWrite/grafana/Gauge (10.19s)

        --- PASS: TestRemoteWrite/grafana/Counter (10.20s)

        --- PASS: TestRemoteWrite/grafana/Staleness (10.12s)

        --- PASS: TestRemoteWrite/grafana/EmptyLabels (10.13s)

        --- PASS: TestRemoteWrite/grafana/NameLabel (10.11s)

        --- PASS: TestRemoteWrite/grafana/RepeatedLabels (10.11s)

        --- PASS: TestRemoteWrite/grafana/SortedLabels (10.12s)

[Julius Volz]

Great! I think as we start taking compatibility more seriously, we could even create a home for remote_write / PromQL / ... compatibility tests in either the main Prometheus org, or in a dedicated "Prometheus Compatibility Project" org?

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAB58Z120jLW%3D17H9RZwcMmGMfUAyVKH3O0MTvBqW45gMwo0oYA%40mail.gmail.com.

[Richard Hartmann]

On Tue, Apr 13, 2021 at 2:16 PM Julius Volz <juliu...@gmail.com> wrote:

> Great! I think as we start taking compatibility more seriously, we could even create a home for remote_write / PromQL / ... compatibility tests in either the main Prometheus org, or in a dedicated "Prometheus Compatibility Project" org?

I strongly believe that the best place for tests we consider official
is the main Prometheus org, as long as the main author is fine with
moving them. I do not have strong opinions about the specific repo
they live in.

[Tom Wilkie]

I only suggested the -community org so its not in my personal org, to encourage contributions from the wider community.  I would be happy to move it to the main prometheus org.  

As a monorepo guy, I'm less keen on a proliferation of many orgs so less keen on a dedicated prometheus-compatibility org...  but could be convinced.

Cheers

Tom

[Julius Volz]

I'm fine with the main org too. Then there's just the question of whether we have one big compatibility repo for remote_write / PromQL / etc., or one for each tested interface. There's definitely pros + cons for each approach, but I'd ultimately be fine with either I think.

[Tom Wilkie]

Yeah I’m fine with either - but would probably prefer a single big repo. Are your PromQl tests golang unit tests by any chance?

[Julius Volz]

Nope, they are run as a binary that takes flags, a config, and outputs JSON / TEXT / TSV / CSV for consumption by humans / website code / etc.

https://github.com/promlabs/promql-compliance-tester/blob/master/cmd/promql-compliance-tester/main.go

[Julien Pivotto]

I am in favour of a repo in the prometheus org.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoztaOmqH_wBsF%2BJue9BywkfdO3otQEOxbWVW-P-VeMb6w%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

[Callum Styan]

Repo in prometheus org makes the most sense to me as well.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20210413220449.GA1265218%40oxygen.

[Tom Wilkie]

SGTM, I've moved it to https://github.com/prometheus/compliance (and moved all the remote_write code to a subdirectory).  

Cheers

Tom

[Bartłomiej Płotka]

Sounds good, thanks Tom for this work!

Kind Regards,

Bartek Płotka (@bwplotka)

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAB58Z13WA9ayf7hK6FQmY%2Bmq%3DfqNMNm%2BA%3DkbNkSmRQPccBRmtA%40mail.gmail.com.

[Julius Volz]

Great, PromLabs is donating its PromQL compliance tester to this effort as well: https://github.com/prometheus/compliance/pull/7

[Mat Arye]

Thanks to Tom and Julius for the work!

Would there be interest in a remote_read compliance test as well? If so, my team would be happy to write it and contribute, but only if people think it's useful (and it's not already being worked on by others).

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxY10RLegTv-va-su1RKbc9TEh%2By0VCSntF_%2BjKSp2AtA%40mail.gmail.com.

--

Mat Arye, Promscale Team Lead

See what we're working on (feedback welcome!): tsdb.co/prom-design-doc

[Julius Volz]

Yeah! While it's not as high priority because fewer people implement and use it, it would still be great to have :)

[Richard Hartmann]

On Thu, Apr 15, 2021 at 4:16 PM Mat Arye <m...@timescale.com> wrote:

> Would there be interest in a remote_read compliance test as well? If so, my team would be happy to write it and contribute, but only if people think it's useful (and it's not already being worked on by others).

Yes, absolutely!