Implemention advice: Prometheus for mature Windows client/server system

[Mister Boy]

I'm brand new to Prometheus and my head is spinning after spending time reading the docs so I hope someone can offer a little advice.

We have a mature client-server system involving dozens of (mostly) C++ modules; workstations and servers all run on Windows and modules mostly communcate via DCOM. We want to start recording metrics about real-time state which is new to me.

I understand Prometheus is pull-based and appears to pull only via HTTP(?) so I am struggling how to reconcile this with our system. Engineering HTTP endpoints into our modules is realistically unlikely and some of our modules use obscure tech which doesn't even allow this. 

Our metrics would generally come from:

- in-memory state in a few key modules running on the app-server

- state of the machines themselves e.g. CPU/RAM use 

- database data, e.g. many metrics would map nicely to SQL queries "how many widgets are active?"

My initial thought would be to create new module(s) which exist solely to expose metrics to Prometheus, essentially a bridge - does that seem reasonable? I'm open to the answer that perhaps a pull-based system is simply not optimal and we should look at a different tool? Sorry for the long post, I hope I've been able to give enough information where I'm coming from.

Thanks.

[Stuart Clark]

The concept of "exporters" will probably help you.

Part of the design of Prometheus is a decentralised focus, so rather
than having a single source for all metrics for a platform you might
source different selections of information from all the different
pieces.

An example of that would be the machine metrics. The standard solution
to obtaining such information is to install the Node Exporter (or
Windows Exporter for Windows machines) on all servers/VMs. This then
gives you basic CPU, network, memory, disk, etc. metrics without any
development access.

The same is true for databases - there are various exporters for the
different database types, some of which can additionally support custom
SQL queries to add more application specific metrics (such as your "how
many widgets are active?" query).

For other applications the approach depends on what is possible &
achievable. In the ideal world you would embed one of the many client
libraries (available for many languages) to directly expose native
Prometheus format metrics from every application. Sometimes that isn't
possible due to it being non-modifyable (COTS, developed elsewhere,
etc.), architectural issues (lack of ability to run long lived HTTP(s)
endpoints) or political (lack of buy-in from teams/management
responsible for development effort).

For the applications where full native metrics aren't possible, you
again have several options. If the application does support another form
of metrics (JMX, StatsD, Graphite, etc.) then there are exporters
available to bridge from those systems into Prometheus. The Node
Exporter also has the textfile collector which can be used to expose
metrics from an application, without requiring such major code changes.

Finally you can use more standalone exporters. There are some more
generic ones available for some form of API type calls (e.g. blackbox,
JSON exporter), but otherwise you can write your own exporter in a
variety of languages - as long as it is possible to get the data out of
the application using some mechanism.

--
Stuart Clark