Protobuf support

[Clément Jean]

Hey everyone,

I met with Richard Hartmann at KubeCon Shanghai and he mentioned that the team is interested in adding support for Protobuf. I'm here to see what I can do to help. If you have any recommendation on how to get started, I'd be happy to start contributing.

[Bjoern Rabenstein]

Thanks for your interest in contributing to Prometheus.

Could you provide more details about what kind of protobuf support you
would like to add? Prometheus already supports protobuf for scraping,
remote write, and remote read.

In general, you could check out issues on the many prometheus
repositories to get inspiration what's needed. Ideally, you pick an
issue that also scratches one of your own itches. If it is tagged as
"good first issue", even better. I would avoid the issues tagged as
"not as easy as it looks" until you have gathered more experience.

Cheers,
--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Clément Jean]

I'm not entirely sure yet because I'm new to contributing to prometheus. If there is any use cases that you guys already discussed around Protobuf, I'd be happy to help.

[Bjoern Rabenstein]

On 05.10.23 17:48, Clément Jean wrote:
> I'm not entirely sure yet because I'm new to contributing to prometheus. If
> there is any use cases that you guys already discussed around Protobuf, I'd
> be happy to help.

If you are really deep into protobuf, there is definitely one big and
fat issue to solve: We are still using gogo-protobuf in
prometheus/prometheus, which has good performance properties, but is
unmaintained. The plan has been for a while to migrate to another
protobuf implementation that performs similarly well. Here is the
discussion on this mailing list:
https://groups.google.com/g/prometheus-developers/c/uFWRyqZaQis/m/1OOGT7s5AwAJ

And here is a branch that contains a PoC of migrating to the vitess
protoc plugin:
https://github.com/austince/prometheus/tree/feat/drop-gogo
(Note that it is more than two years old, and I'm not sure if the
vitess plugin performs well enough. But it could be a starting point.)

[Clément Jean]

That actually sounds interesting to me! I have few questions though:

- why not moving to the official generated code? It seems it's the only stable one on the long term

- Is there a benchmark that I could run for prometheus using different plugins?

- How can I test if my code changes didn't break the project? go test ./... ?

- Are there some platform specific tests?

Hope we can work on that

[Bryan Boreham]

> - why not moving to the official generated code? It seems it's the only stable one on the long term

At least because nobody has put the work in to do the conversion and demonstrate it is no worse than today.

I don't like the extra fields in generated structs, but I didn't measure the impact.

> - Is there a benchmark that I could run for prometheus using different plugins?

Yes, here for instance: https://github.com/prometheus/prometheus/blob/122f9506e9c6/storage/remote/queue_manager_test.go#L872

For completeness one should check the 4 combinations: remote-read sender/receiver and remote-write sender/receiver.

There are some unit tests but I don't think they will help to check forwards/backwards compatibility.

You could set up another Prometheus and have them to talk to each other.

Some discussion about how to check at https://github.com/prometheus/prometheus/pull/9991

Bryan

[Clément Jean]

Hey,

I finally go around to do this and the unit tests seem to be passing. You can find what I did here: https://github.com/Clement-Jean/prometheus-move-to-proto.

I still have few questions:

- For the benchmarking, should I try running the benchmark on the current version of prometheus and then run it on my version, or is there a better way to do this?
- Is `make bench_tsdb` enough or should I run other benchmarks?

- For forward/backward compat, I'm not sure how I should go about it. Could you clarify a little bit?

Thank you

[Bjoern Rabenstein]

Disclaimer: I'm not the expert for remote-write/-read, and only do a
little with the protobuf stuff. But since you didn't get an answer
in a week, I'm trying now.

On 29.02.24 20:38, Clément Jean wrote:
>
> I finally go around to do this and the unit tests seem to be passing. You
> can find what I did here:
> https://github.com/Clement-Jean/prometheus-move-to-proto.

Thank you. This looks very unconventional with separate patch
files. Wouldn't it be more straight forward to just fork
prometheus/prometheus and work in the normal git style in that fork?

> I still have few questions:
> - For the benchmarking, should I try running the benchmark on the current
> version of prometheus and then run it on my version, or is there a better
> way to do this?

That sounds like the way to go.

> - Is `make bench_tsdb` enough or should I run other benchmarks?

`make bench_tsdb` is a rather special benchmark for certain TSDB
aspects. I doubt it touches protobuf at all.

We usually use the benchmarking framework as built into the Go
toolchain, i.e. you run `go test -bench NameOfBenchmarkToRun`, usually
also using the `-benchmem` flag to see memary allocation stats.

Bryan referred to
https://github.com/prometheus/prometheus/blob/122f9506e9c6/storage/remote/queue_manager_test.go#L872
previously. You would run that benchmark in the storage/remote
directory by typing `go test -bench BenchmarkSampleSend -benchmem`

> - For forward/backward compat, I'm not sure how I should go about it. Could
> you clarify a little bit?

It's important that the protobuf messages on the wire are still
encodable and decodable by older and newer versions of the code. So I
believe what Brian was referring to is that you set up a Prometheus
server with and without your changes and let them send remote write
and remote read to each other.

In different news: On this same mailing list, somebody else
(mircod...@gmail.com) is also working on the same topic, see thread
titled "Migrating away from github.com/gogo/protobuf". They have run a
bunch of benchmarks already, and their approach to replace
gogo-protobuf has run into some issues. I suggest you two join forces
and exchange your experiences. If you want a chat-like medium to
discuss things, you should know that most of the dev conversation
happens on the CNCF Slack these days (channel #prometheus-dev). (There
is also a #prometheus-dev on IRC, but I'm afraid Slack has
successfully sucked most devs into its black hole.)

Hope that helps a bit.