How to match series if label has certain value or label is missing?

[Sander Saares]

I have series of the following style:

asdf{instance=x}

asdf{instance=y, foo=bar}

asdf{instance=y, foo=wonk}

And I want to write a query that returns all series where foo is not bar, including those where foo is missing. How can I accomplish this?

If I just do asdf{foo != "bar"} then I do not get the series without the foo label. The only way I have found so far is to do asdf{foo != "bar"} or asdf{foo = ""}, as only the second one finds those without this label.

[Brian Brazil]

You can do asdf{foo!=""}, however you should generally try to ensure that labels for a metric consistently have values.

--

Brian Brazil

www.robustperception.io

[Sander Saares]

I think something got misaligned here between our posts. To clarify, from the following set: 

1. asdf{instance=x}

2. asdf{instance=y, foo=bar}

3. asdf{instance=y, foo=wonk}

I want to select 1 and 3 (anything that does not have the "bar" value). Right now I have to select them separately and "or" them together because just doing foo!=bar will not return series 1. I would prefer not to have to do this "or" operation.

I agree that my label structure is not ideal. Just trying to tackle the topic from both directions at once.

[Brian Brazil]

It sounds like you are running into a bug in Prometheus 2.0, try the 2.2.0-rc.1. 

--

Brian Brazil

www.robustperception.io

[Sander Saares]

You are correct. Upgrading to latest build makes the foo!=bar query return both series 1 and series 3!