Proposal: In-memory Exemplar Storage

[Callum Styan]

Hi all,

As many of you know we've been planning to add minimal in-memory storage of exemplars within Prometheus for some time now. Some work has already been done in this area; including the exposition of exemplars in the Python client, and parsing of the Open Metrics exemplar format by the M3/Chronosphere folks, who also proposed a possible storage interface here.

More recently some of us have been putting together a design doc for the end to end exemplar experience within Prometheus; discussing what the query flow for exemplars might look like from Grafana itself, what the storage interface (external and internal) API might look like, and how the internal storage could be implemented. 

Some of you have also seen the experimental implementation PR I opened a few days ago. From the comments there it sounds like there may be exemplar querying use cases that haven't been discussed yet, which I would ask you to make note of in the design doc. 

With all that said, here's the design doc. Note that an implementation has not been finalized yet.

Thanks,

Callum.

[Callum Styan]

Hi all,

For those who are interested, I've added notes on how the exemplar storage might handle tail based sampling of traces. These notes are based on discussions I've had with a few people involved in/working on tracing systems.

As usual, please let me know if you have any feedback.

The current exemplar storage PR is here: https://github.com/prometheus/prometheus/pull/6635 In the current state, building Prometheus from that branch will allow you to scrape, store, and query exemplars.

Thanks,

Callum.

[Bartłomiej Płotka]

Awesome! Exciting work, thanks for sharing. Will take a look in following days / weekend (: 

Kind Regards,

Bartek

On Thu, 19 Mar 2020 at 21:27, Callum Styan <callu...@gmail.com> wrote:

Hi all,

For those who are interested, I've added notes on how the exemplar storage might handle tail based sampling of traces. These notes are based on discussions I've had with a few people involved in/working on tracing systems.

As usual, please let me know if you have any feedback.

The current exemplar storage PR is here: https://github.com/prometheus/prometheus/pull/6635 In the current state, building Prometheus from that branch will allow you to scrape, store, and query exemplars.

Thanks,

Callum.

On Thu, Jan 9, 2020 at 3:39 PM Callum Styan <callu...@gmail.com> wrote:

Hi all,

As many of you know we've been planning to add minimal in-memory storage of exemplars within Prometheus for some time now. Some work has already been done in this area; including the exposition of exemplars in the Python client, and parsing of the Open Metrics exemplar format by the M3/Chronosphere folks, who also proposed a possible storage interface here.

More recently some of us have been putting together a design doc for the end to end exemplar experience within Prometheus; discussing what the query flow for exemplars might look like from Grafana itself, what the storage interface (external and internal) API might look like, and how the internal storage could be implemented. 

Some of you have also seen the experimental implementation PR I opened a few days ago. From the comments there it sounds like there may be exemplar querying use cases that haven't been discussed yet, which I would ask you to make note of in the design doc. 

With all that said, here's the design doc. Note that an implementation has not been finalized yet.

Thanks,

Callum.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAN2d5OT%2BJi4DB4fupTyjsgRa_T%2ByZiiVfG-FuEJDRV%3DWXDJDvg%40mail.gmail.com.

[Callum Styan]

Hi all,

I've taken up my work on exemplars again; the above PR and design doc are still relevant but I'll paste the links again below. 

I've changed the PR from a draft to open, it's still rough but I would appreciate feedback. Note that the implementation is in memory, using a single fixed size circular buffer (size configurable by users) so that overall memory usage by exemplar storage is bounded.

design doc

PR

Thanks,

Callum.