Integrate Kafka as remote storage backend.

[georg...@gmail.com]

Hi,

I integrated kafka into the existing remote storage structure with pull request #2315

I know there is vulcan and so on, but basically I only needed a fast way to send the data to our central kafka entrypoint. This was the fastest.

What do you think?

[Brian Brazil]

We want to keep all remote storage options external to Prometheus itself, particularly those not suitable to use as storage such as Kafka. See https://www.robustperception.io/using-the-remote-write-path/ for how to integrate. 

--

Brian Brazil

www.robustperception.io

[Georg Öttl]

Thanks  for the link, seems to be the more elegant solution. Moves away the problem from prometheus.

For kafka to work you'd need to implement a protobuf consumer and enable the REST api. Am I right?

Regards,

Georg

[georg...@gmail.com]

We tested the remote_write in its current implementation a bit. Getting everything to work with the default settings isn't easy. The default, namely Protocol Buffer over HTTP, with snappy compression, isn't the easiest way to start for a user. Makes remote_write to start with more complicated than it should be.

For my liking I'd prefer a start with everything in plaintext without protobufs and snappy. Also the documentation lacks the protobuf scheme, so it is hard to actually implement consumers for the data.

I hope this uninvited feedback doesn't do any harm. If, please ignore it :-)

Regards,
Georg

[Tom Wilkie]

Hi Georg

Thanks for the feedback - sorry you found it difficult.  May I ask, what language were you using?

There is an example remote write 'server' implementation (linked from Brian's post) [1] and the definition of the protobufs is checked into the repo [2].  

For a plain-text without-compression view of the data, you could have used the federation interface - pulling data out of prometheus.  Other than that what else could we have done to make this easier?  Perhaps providing examples in Java etc would make it easier.

Thanks

Tom

[1] https://github.com/prometheus/prometheus/blob/master/documentation/examples/remote_storage/server.go

[2] https://github.com/prometheus/prometheus/blob/master/storage/remote/remote.proto

>
> --
>
>
> Brian Brazil
> www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/146337c3-b1d1-4a3b-bb0e-e06f494fbc90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Georg Öttl]

Federation interface, writing a custom server to translate the message or write a new kafka exporter inside prometheus (which i did) - everything needs some kind of implementation to be done from the user side. Actually if the remote_write would allow to send the events in json over http, I think it would  be easier to integrate it with existing systems. At least for proof of concepts. No need to compile protobufs, write custom implementations ...

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

[Tom Wilkie]

Hi Georg

We're trying to keep the Prometheus server as "light" as possible, and avoiding supporting multiple different formats is the point of the "generic" remove write path - previously Prometheus embedded a bunch of different remote writes for things like influxdb, opentsdb and graphite.  

Having said that, it would be quite straight forward to write a small proxy which translates from the proto/snappy format to JSON - would that help?

Tom

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

To post to this group, send email to prometheus-developers@googlegroups.com.

[Georg Öttl]

Sounds good to me if this is the way to go.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

[guido.garc...@gmail.com]

We have recently written a remote write endpoint for Kafka. It receives metrics and writes them to a Kafka topic in JSON format.

You might find it useful: https://github.com/Telefonica/prometheus-kafka-adapter

I think that the protobuf-to-json conversion could still be improved. Contributions are welcome.

El lunes, 23 de enero de 2017, 15:48:53 (UTC+1), Georg Öttl escribió:
> Sounds good to me if this is the way to go.
>

> Tom Wilkie <tom@x> schrieb am Mo., 23. Jan. 2017, 15:47:
>
> We're trying to keep the Prometheus server as "light" as possible [...]

[Siddhesh Divekar]

Hi Guido,

Just curious, what do you do on the consumer side once the data goes to kafka.