Callbacks around series lifecycle in TSDB for the use of external users
GANESH VERNEKAR
Hi All,
I would like to propose a callback for the TSDB for series creation and deletion. This would help external projects like Cortex and Thanos in implementing limits on series while having no performance impact on Prometheus with minimal code addition.
The callback would look something like this (names TBD) with no performance impact on TSDB.
type SeriesLifecycleCallback interface {
PreCreationCallback(labels.Labels) error
PostCreationCallback(labels.Labels)
PostDeletionCallback(labels.Labels)
}
1. SeriesLifecycleCallback
It would be a part of DB.Options
2. PreCreationCallback
It would be called just before holding the lock here. And if the method returns an error, it means we don’t want to create a new series.
Further the hashes would be checked as usual, and if we reach the step of adding a new hash, it would be either executed or skipped based on the result of `PreCreationCallback`.
If the series creation was skipped, the error returned by `PreCreationCallback` would be propagated back into the call stack and the `Add` or `AddFast` method would finally return back the error. (This means additional return values for the methods in that call stack).
3. PostCreationCallback
This would be called after releasing the lock here letting the caller know that the series was created.
4. PostDeletionCallback
It would be called after deleting a series during garbage collection here (after unlocking) so that external projects can do the bookkeeping.
In the context of Prometheus, the interface would be a noop, hence no performance impact.
Some reasoning behind the design decisions and what guarantees it provides:
The idea was to avoid having the callbacks inside the locked section which could lead to performance degradation. Hence the Series creation callback was split into 2 methods. PreCreationCallback method tells whether to create the series if we need to create and the next method `PostCreationCallback` to let the user know if it was created, so that appropriate bookkeeping can be done.
Since all the callbacks happen outside the locks (even if it was inside the lock, there could be concurrent calls for different shards of the stripe series), the user can expect some soft consistencies here. For example, the PreCreationCallback could be called concurrently by multiple series creation when you are at the edge of the series limit and all the series creation would be allowed, which might cause the limit to cross by a small number. Similarly during the deletion phase: some creation of series would fail as the deletion callback is called a little later. Hence the user is required to expect soft consistencies while using these callbacks.
What do you all think about this addition in TSDB?
Thanks,
Ganesh
Brian Brazil
Hi All,
I would like to propose a callback for the TSDB for series creation and deletion. This would help external projects like Cortex and Thanos in implementing limits on series while having no performance impact on Prometheus with minimal code addition.
The callback would look something like this (names TBD) with no performance impact on TSDB.
type SeriesLifecycleCallback interface {
PreCreationCallback(labels.Labels) error
PostCreationCallback(labels.Labels)
PostDeletionCallback(labels.Labels)
}
1. SeriesLifecycleCallback
It would be a part of DB.Options
2. PreCreationCallback
It would be called just before holding the lock here. And if the method returns an error, it means we don’t want to create a new series.
Further the hashes would be checked as usual, and if we reach the step of adding a new hash, it would be either executed or skipped based on the result of `PreCreationCallback`.
If the series creation was skipped, the error returned by `PreCreationCallback` would be propagated back into the call stack and the `Add` or `AddFast` method would finally return back the error. (This means additional return values for the methods in that call stack).
3. PostCreationCallback
This would be called after releasing the lock here letting the caller know that the series was created.
4. PostDeletionCallback
It would be called after deleting a series during garbage collection here (after unlocking) so that external projects can do the bookkeeping.
----In the context of Prometheus, the interface would be a noop, hence no performance impact.
Some reasoning behind the design decisions and what guarantees it provides:
The idea was to avoid having the callbacks inside the locked section which could lead to performance degradation. Hence the Series creation callback was split into 2 methods. PreCreationCallback method tells whether to create the series if we need to create and the next method `PostCreationCallback` to let the user know if it was created, so that appropriate bookkeeping can be done.
Since all the callbacks happen outside the locks (even if it was inside the lock, there could be concurrent calls for different shards of the stripe series), the user can expect some soft consistencies here. For example, the PreCreationCallback could be called concurrently by multiple series creation when you are at the edge of the series limit and all the series creation would be allowed, which might cause the limit to cross by a small number. Similarly during the deletion phase: some creation of series would fail as the deletion callback is called a little later. Hence the user is required to expect soft consistencies while using these callbacks.
What do you all think about this addition in TSDB?
GANESH VERNEKAR
On Tuesday, April 21, 2020 at 8:59:12 PM UTC+5:30, Brian Brazil wrote:
On Tue, 21 Apr 2020 at 15:17, 'GANESH VERNEKAR' via Prometheus Developers <prometheus...@googlegroups.com> wrote:Hi All,
I would like to propose a callback for the TSDB for series creation and deletion. This would help external projects like Cortex and Thanos in implementing limits on series while having no performance impact on Prometheus with minimal code addition.
The callback would look something like this (names TBD) with no performance impact on TSDB.
type SeriesLifecycleCallback interface {
PreCreationCallback(labels.Labels) error
PostCreationCallback(labels.Labels)
PostDeletionCallback(labels.Labels)
}
1. SeriesLifecycleCallback
It would be a part of DB.Options
2. PreCreationCallback
It would be called just before holding the lock here. And if the method returns an error, it means we don’t want to create a new series.
Further the hashes would be checked as usual, and if we reach the step of adding a new hash, it would be either executed or skipped based on the result of `PreCreationCallback`.
If the series creation was skipped, the error returned by `PreCreationCallback` would be propagated back into the call stack and the `Add` or `AddFast` method would finally return back the error. (This means additional return values for the methods in that call stack).
3. PostCreationCallback
This would be called after releasing the lock here letting the caller know that the series was created.
4. PostDeletionCallback
It would be called after deleting a series during garbage collection here (after unlocking) so that external projects can do the bookkeeping.
----In the context of Prometheus, the interface would be a noop, hence no performance impact.
It'd still cost a few cycles.
Some reasoning behind the design decisions and what guarantees it provides:
The idea was to avoid having the callbacks inside the locked section which could lead to performance degradation. Hence the Series creation callback was split into 2 methods. PreCreationCallback method tells whether to create the series if we need to create and the next method `PostCreationCallback` to let the user know if it was created, so that appropriate bookkeeping can be done.
Since all the callbacks happen outside the locks (even if it was inside the lock, there could be concurrent calls for different shards of the stripe series), the user can expect some soft consistencies here. For example, the PreCreationCallback could be called concurrently by multiple series creation when you are at the edge of the series limit and all the series creation would be allowed, which might cause the limit to cross by a small number. Similarly during the deletion phase: some creation of series would fail as the deletion callback is called a little later. Hence the user is required to expect soft consistencies while using these callbacks.
What do you all think about this addition in TSDB?
This seems to require adding error handling for cases where Prometheus cannot have errors, what happens to a scrape or rule evaluation when this fails?
I'm also not sure it's considering that the semantics of how this is all managed inside the tsdb is not what you might expect. For example a failed scrape due to hitting the sample limit can still cause series to be created, which usually doesn't come up but might now.
--Brian Brazil
Brian Brazil
GANESH VERNEKAR
GANESH VERNEKAR
Coming back to the concerns regarding error in no-error cases and how would it work in case of failed scrapes:
Thanks,