Prometheus docs

61 views
Skip to first unread message

Richard Hartmann

unread,
Mar 12, 2020, 7:04:34 AM3/12/20
to promethe...@googlegroups.com
Dear all,

if we as Prometheus team were to try and improve our docs, what would
be the most pressing areas in your opinion?


Best,
Richard

Riddhi Prajapati

unread,
Mar 12, 2020, 8:51:24 AM3/12/20
to Richard Hartmann, promethe...@googlegroups.com
Please could you add more documentation on remote write for kubernetes with istio integrated  

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAD77%2BgSPXH2g_pEgfExxmK5CyZPRc6E251e73_JrCBtbrhWZNA%40mail.gmail.com.

Julien Pivotto

unread,
Mar 14, 2020, 8:05:30 AM3/14/20
to Richard Hartmann, promethe...@googlegroups.com
Richard,

That is a great question. The Prometheus documentation looks more like a man page than modern documentation.

Given how complex it is, it would be nice to see how we can improve the documentation about vector matching, range vectors, instant vectors, and or without group_right.

I am thinking, e g., about adding simple but coherent diagrams/images that would help newcomers to grasp those principles. Having updated relevant examples to go with(in) the figures would help, too.


Thanks

Brian Brazil

unread,
Mar 14, 2020, 8:46:25 AM3/14/20
to Julien Pivotto, Richard Hartmann, Prometheus Users
On Sat, 14 Mar 2020 at 12:05, Julien Pivotto <roidel...@inuits.eu> wrote:
Richard,

That is a great question. The Prometheus documentation looks more like a man page than modern documentation.

Given how complex it is, it would be nice to see how we can improve the documentation about vector matching, range vectors, instant vectors, and or without group_right.

I am thinking, e g., about adding simple but coherent diagrams/images that would help newcomers to grasp those principles. Having updated relevant examples to go with(in) the figures would help, too.

The current docs are primarily reference docs, trying to also make a given page serve an additional goal will only make it less usable for everyone.  https://www.divio.com/blog/documentation/ has a good rundown on it.

I've seen this in the past where attempting to make the reference docs more usable to newcomers didn't really help new users due to all the detail and precision they still have to contain, while also making it harder for experienced users to now do a quick reference lookup.  Any new type of doc should go on different pages, such as we have with the guides.

Brian
 


Thanks


----- Original Message -----
From: Richard Hartmann <richih.ma...@gmail.com>
To: promethe...@googlegroups.com
Sent: Thu, 12 Mar 2020 12:04:18 +0100 (CET)
Subject: [prometheus-users] Prometheus docs

Dear all,

if we as Prometheus team were to try and improve our docs, what would
be the most pressing areas in your opinion?


Best,
Richard

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAD77%2BgSPXH2g_pEgfExxmK5CyZPRc6E251e73_JrCBtbrhWZNA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

Bjoern Rabenstein

unread,
Mar 16, 2020, 2:07:40 PM3/16/20
to Brian Brazil, Julien Pivotto, Richard Hartmann, Prometheus Users
I guess we could have reference-style documentation and more
tutorial-style documentation on the same site, but clearly separated.

Arguably, we already have that with the Guides section.

However, I don't think that this section is well advertised. The flow
through the site is in general not very obvious or smooth. I could
imagine, for someone completely new to the site, it's very hard to
find the right thing by something else than using a search engine to
jump directly to parts of the site. This problam _also_ affects the
pure reference part.

Examples:

- As said above, what are "Guides" even? How do I see that that's a
good way to find some cookbook-style advice? There is "First steps",
but it doesn't link to the Guides.

- Then there isn "Best Practices". In what way are those different
from "Guides"? Again, there is no overall guide through the
site. The landing page has those eight section it points to, and a
Get Started pointer, but none is to Guides or Best Practices.

- The site is not very clear what is prometheus/prometheus AKA "the
Prometheus server" and what is general Prometheus ecosystem and what
are Prometheus components like the Alertmanager. For example, if I
click "Get Started" on the landing page, I get to the "Getting
Started" section of the prometheus/prometheus documentation. Perhaps
the "First steps" section in the Introduction would make more
sense. But the latter is anyway (perhaps?) an outdated version of
the former? The section about the Prometheus server is simply called
"Prometheus", but that's what the whole site is about, isn't it?

- Another example for the previous item: alerting. The "Precise
Alerting" link on the landing page goes to the Alerting Rules
section two levels deep in the prometheus/prometheus
documentation. However, there is a top-level section "Alerting",
which is also relevant. It is mostly concerned with Alertmanager
(which is not obvious from the section titel) but also contains a
sub section "Alerting overview" (which is probably where I'd naively
expect the landing page link about alerting pointing to.

- Another flow problem: If I am in the "Prometheus" section, there is
a subsection "Management API". However, if I'm looking for the TSDB
admin API, I won't find anything there, but instead I have to click
on "Querying" and then "HTTP API".

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

Philip Potter

unread,
Mar 16, 2020, 4:39:48 PM3/16/20
to Richard Hartmann, promethe...@googlegroups.com
Since this was a "dear all" on prometheus-users, I guess I can throw my opinion in.

In my view, the one thing I most desire that is currently missing from the Prometheus docs is a rationale / design principles document.  Something that explains why Prometheus is the way that it is.

The sorts of things this document could help with:
  • what problems is Prometheus designed to solve? what is it not designed to solve?
    • it does metrics for operational monitoring - not for audit, not 100% accuracy
    • a prometheus server should have no external dependencies, so that it can run even when other things are breaking
    • something something failure domains
    • pull model
    • the prometheus server is in charge of scheduling scrapes; the clients don't set cadence
  • some example implications of design choices - features / patterns of prometheus and how they tie back to the above
    • why do counters monotonically increase? why does the rate() function do what it does and how does it compare with other metrics tools?
    • why does prometheus use local disk for storage? why is there no replication option? if i'm used to setting up replication/backups on all my stateful services, what should I do with Prometheus? (I have seen someone decide that remote_write to Postgres/TimescaleDB was absolutely required because we know how to operate Postgres already)
    • why does the increase() function return fractional values?
    • why do we prefer small single-purpose exporters?
I kind of feel I know the answers to these questions, but it has been picked up slowly through a long process of osmosis, and it makes it hard to upskill others at my workplace. I have become the local Prometheus expert because I've been playing with it long enough to have picked up this oral tradition; but I'd like to see it written down somewhere official so that I can point others at it.  It's also very slow to pick this information up because you often learn it through a disjointed series of "why..?" questions (the second list above) and have to infer the underlying design principles (the first list above) piece by piece.

There are bits of this kind of information dotted around the docs: the bottom of the overview page <https://prometheus.io/docs/introduction/overview/> has some very high level principles, but it doesn't feel complete; the "Best Practices" section has some of this information, but it varies massively in scope and level of detail (from "how do I design a console or dashboard" which is super high-level and not even specific to Prometheus, to "how should I instrument inner loops in my program" which is pretty low-level and specific to a particular problem; neither is really at the level I'm talking about here).

The thing is, having used Prometheus for 2 years now it's clear that Prometheus and its community does have a very strong sense of shared design principles, and this is a good thing; but the lack of a place to point beginners at to explain these design principles has been a barrier for me teaching Prometheus to others at my workplace.  I would really like to see a single page or set of pages to capture these ideas.

Phil





Julien Pivotto

unread,
Mar 16, 2020, 4:44:59 PM3/16/20
to Philip Potter, Richard Hartmann, promethe...@googlegroups.com
On 16 Mar 20:39, Philip Potter wrote:
> Since this was a "dear all" on prometheus-users, I guess I can throw my
> opinion in.
>
> In my view, the one thing I most desire that is currently missing from the
> Prometheus docs is a rationale / design principles document. Something
> that explains *why* Prometheus is the way that it is.


Some questions are answered by https://github.com/prometheus/docs/pull/1054/files

>
> The sorts of things this document could help with:
>
> - what problems is Prometheus designed to solve? what is it *not*
> designed to solve?
> - it does metrics for operational monitoring - not for audit, not
> 100% accuracy
> - a prometheus server should have no external dependencies, so that
> it can run even when other things are breaking
> - something something failure domains
> - pull model
> - the prometheus server is in charge of scheduling scrapes; the
> clients don't set cadence
> - some example implications of design choices - features / patterns
> of prometheus and how they tie back to the above
> - why do counters monotonically increase? why does the rate() function
> do what it does and how does it compare with other metrics tools?
> - why does prometheus use local disk for storage? why is there no
> replication option? if i'm used to setting up
> replication/backups on all my
> stateful services, what should I do with Prometheus? (I have seen someone
> decide that remote_write to Postgres/TimescaleDB was absolutely required
> because we know how to operate Postgres already)
> - why does the increase() function return fractional values?
> - why do we prefer small single-purpose exporters?
--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu
signature.asc

Philip Potter

unread,
Mar 16, 2020, 4:49:03 PM3/16/20
to Julien Pivotto, Richard Hartmann, promethe...@googlegroups.com
On Mon, 16 Mar 2020 at 20:44, Julien Pivotto <roidel...@inuits.eu> wrote:
>
> On 16 Mar 20:39, Philip Potter wrote:
> > Since this was a "dear all" on prometheus-users, I guess I can throw my
> > opinion in.
> >
> > In my view, the one thing I most desire that is currently missing from the
> > Prometheus docs is a rationale / design principles document. Something
> > that explains *why* Prometheus is the way that it is.
>
>
> Some questions are answered by https://github.com/prometheus/docs/pull/1054/files

Oh, amazing! Yes, that looks like the sort of thing I'm looking for.

Phil

Bjoern Rabenstein

unread,
Mar 17, 2020, 9:50:23 AM3/17/20
to Brian Brazil, Julien Pivotto, Richard Hartmann, Prometheus Users
And another item that keep itching me:

The governance doc is deeply buried, given how important it is.

To find it without using a search engine, I have to click "Community"
and then find the very inconspicuous link to it in the middle of the
page, i.e. there is not a single menu item that links to the
governance directly.

Tristan Colgate

unread,
Mar 17, 2020, 11:31:59 AM3/17/20
to Bjoern Rabenstein, Brian Brazil, Julien Pivotto, Richard Hartmann, Prometheus Users
I always really liked the prometheus documentation. I've never really
understood people that have struggled with it. This may be just a
historical thing, from those of us who "grew up" with man pages and
reference books, and less short form tutorials.

The Best Practice section was absolutely invaluable. Any
updating/extending of that would always be welcome I think.

The only subject that I felt was poorly covered was label matching.
The documentation is rather terse in proportion to the depth and
subtlety of what is going on. More worked examples, especially WRT to
group_(left/right), and failure cases for matching, would help a lot.
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20200317135018.GW14683%40jahnn.



--
Tristan Colgate-McFarlane
----
"You can get all your daily vitamins from 52 pints of guiness, and a
glass of milk"

Bjoern Rabenstein

unread,
Apr 7, 2020, 6:16:33 PM4/7/20
to Brian Brazil, Julien Pivotto, Richard Hartmann, Prometheus Users
Another thing that just came up with a coworker:

It would be great if the bottom of each page had a link to the source
file of that page. Instead, it links to
https://github.com/prometheus/docs#contributing-changes , which
explains how to find the sources. However, it is still confusing,
especially for the prometheus/prometheus server. The contribution
guide mentions that, but it's like a treasure hunt...
Reply all
Reply to author
Forward
0 new messages