Extending tsdb tests

[Julien Pivotto]

Hi there,

Would you think that it would be valuable to test the pull requests with
the third parties that use our codebase (corthanos)?

What I imagine would be that those tests would run tests against
corthanos master, update the prometheus dependencies to the commit and
the PR, and report back test results.

I don't think failures over interface changes should prevent us to
merge, but would it bring extra coverage and information regarding to
our code? Would that be valuable?

--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu

[Matthias Rampke]

I am a little concerned about "tests that are okay to fail" … especially since merging one PR with that means we have no signal for subsequent PRs. Meanwhile corthanos might adjust their usage, breaking older branches that don't even touch this (solvable with rebasing but adds additional round trips).

How could we mark these tests as less important? How could we mark expected failures?

Are there other testing strategies that would maintain the "this breaks interfaces" signal without coupling things too closely?

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200321210234.GA18497%40oxygen.

[Bartłomiej Płotka]

Interesting idea. I think it's something to be considered by our explorations like modularize. For each change of exported package we could just trigger some jobs in Thanos and Cortex CI which will try to upgrade dep and run tests. But still, it will be pain to maintain in terms of interface change, etc.

But yes, tests which you can ignore are rather an anti pattern, just noise.

Kind Regards,

Bartek

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAFU3N5UTSTNXp7sadvZc8%3D8tuwd5utWdG4t19%3DRtce1x4OJ73w%40mail.gmail.com.