On 20/03/2021 12:52,
an...@signoz.io wrote:
> Got it.. and does each team manage their own data and don't share it
> with other teams? Or each team has a prom instance running with a
> central data storage?
It very much depends on what you want, team capabilities and
security/performance/maintenance concerns.
In general the decentralised nature of Prometheus allows each
team/application/environment to have its own set of servers, which
prevents a single central failure from stopping everyone. If the metrics
are only useful to a single team then it might not be more widely
available, but similarly it could be shared if multiple teams need
visibility - this could be using federation or a remote write system.
Ultimately it is down to who should control what (ideally you don't want
another team controlling a system you depend upon if you could do it
yourself) and who should be able to access what (there might be some
systems which are "secret" or some business level metrics you want to be
more careful about who can access).
You could send everything to a single central remote write system, but
you'd need to be careful about creating a single point of failure and
how maintenance outages of that system are handled. Also, that single
central system may need to have a single "lowest common denominator"
setup to allow all teams to do what they want, while also having to live
within and constraints (e.g. storage) that might apply. Personally I try
to keep things more separated, so using different storage for each team,
which makes maintenance easier and allows each team to control their own
configuration, at the cost of a bit more complexity/infrastructure.
--
Stuart Clark