Deploying Thanos

[George]

Hi all

Question... in our current management  tool set we have a proxy server on each of the subnets that is responsible for collecting the performance metrics of all servers on that subnet,

those values are then send to a large management server where everything is submitted.

in the Prometheus architecture the server itself collects the data and stores it in the local TSB.

We also have 2 DC's, thinking is to deploy prometheus servers in each of the DC's, responsible for the local servers.

My thinking is to deploy a prometheus server on each of our subnets, and then a Thanos side car on the prometheus server.

In this configuration... does the various Thanos deployed get told about each other, so that they can display all the metrics of all the prometheus servers as one large deployment ?

For Grafana, at the moment it's pointed at my prometheus server ip address and a port, how does this change, or does it not, and Thanos inserts itself in the middle of that path.

please comment.

G

--

You have the obligation to inform one honestly of the risk, and as a person
you are committed to educate yourself to the total risk in any activity!

Once informed & totally aware of the risk,
every fool has the right to kill or injure themselves as they see fit!

[Ben Kochie]

This sounds very much like a standard multi-DC / multi-subnet Prometheus+Thanos setup.

Basically what you have is a Thanos Query front-end, that is configured to connect to all of the sidecars. It can query data from all Prometheus servers simultaneously.

Grafana is configured to talk to Thanos Query, so you can get all data in one view.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CALw5Ujv24MAYfXFojRXd%2BSDd8LS%3D68GbsPQb4d-NmE3pnWLv_A%40mail.gmail.com.

[George]

Hi Ben

So would you recommend we deploy one Thanos server/deployment per DC. to which the local Grafana server then connects. 

That Thanos then talks to all local and remote (2nd DC) prometheus servers.

G

[Ben Kochie]

There is no "Thanos Server". There are a number of components that are deployed in different places. I suggest you review the official Thanos docs.

You only need Thanos Query deployed where Grafana is. Thanos Query talks directly to the sidecars and optionally store servers.

[George]

Doing some more googling as we're speaking.

Looks like each Prometheus server gets the Thanos side car, and then Thanos Query is deployed onto my Grafana server/s,

Grafana talks to Thanos Query (locally), thats configured to know about all my Thanos side cars + Thanos Store for aging out older metrics to S3 buckets.

G

[George]

Stretching my question a bit here...

Most of the examples of deploying thanos assume a Prometheus on K8S (which would be great option down the road once we have a management K8s cluster), as I don't want to run my management environment on the K8s cluster that I also want to monitor.

Don't see allot for deploying Thanos on a Linux host for Prometheus, although there is the one example of simply building the Thanos executable from the Go code.

... or downloading the tar ball, looking inside this is a single executable.

... this example also does not mention if you are building the Thanos sidecar, Thanos Query or the Thanos Store component... assuming it builds the one executable that can do all 3... and is then configured. What am I missing here.

G

[Ben Kochie]

On Thu, Aug 27, 2020 at 4:59 PM George <geor...@gmail.com> wrote:

Stretching my question a bit here...

Most of the examples of deploying thanos assume a Prometheus on K8S (which would be great option down the road once we have a management K8s cluster), as I don't want to run my management environment on the K8s cluster that I also want to monitor.

There is nothing wrong with running monitoring inside the Kubernetes cluster. This is very common and normal.

 

Don't see allot for deploying Thanos on a Linux host for Prometheus, although there is the one example of simply building the Thanos executable from the Go code.

... or downloading the tar ball, looking inside this is a single executable.

... this example also does not mention if you are building the Thanos sidecar, Thanos Query or the Thanos Store component... assuming it builds the one executable that can do all 3... and is then configured. What am I missing here.

Lots of people have built Chef, Ansible, automation for doing non-K8s deploys. We built our own for Chef, but are planning to move away from that and move it to Kubernetes.

[George]

Hi Ben

We will have multiple K8s clusters (see future), various workloads on mostly production networks, happy that each of these run a local prometheus store and local Grafana dashboard for those users, but then additionally have a couple of hundred linux and windows VM's.

I want to get sidecar deployed on all prometheus deployments, be that the K8S hosted or a central hosted Prometheus that looks after my Linux and Windows hosts / DC.

currently we don't have those K8S clusters yet, but we do have all the linux and windows... so want to deploy a stack now, Linux installed Prometheus server with Thanos sidecar. Dedicated Grafana server + Thanus Query, environment can then be expanded, with all future prometheus servers also having a side car, all talking to a single Thanus Store with a shared S3 bucket.

Once we have a management K8S cluster, the linux and windows monitoring prometheus can be migrated.

G