A Configuration

[Latisha Gervase]

Theterm is very common in computer science and mathematics, and in scientific and technological fields in general. Thus, for example, two scientists won a 1962 Nobel Prize for their description of the configuration of the DNA molecule. Since then, researchers have studied what different configurations within the DNA strands mean and what they control, and genetic engineers have tried to configure or reconfigure DNA in new ways to prevent or treat diseases.

Prometheus is configured via command-line flags and a configuration file. Whilethe command-line flags configure immutable system parameters (such as storagelocations, amount of data to keep on disk and in memory, etc.), theconfiguration file defines everything related to scraping jobs and theirinstances, as well aswhich rule files to load.

Prometheus can reload its configuration at runtime. If the new configurationis not well-formed, the changes will not be applied.A configuration reload is triggered by sending a SIGHUP to the Prometheus process orsending a HTTP POST request to the /-/reload endpoint (when the --web.enable-lifecycle flag is enabled).This will also reload any configured rule files.

A scrape_config section specifies a set of targets and parameters describing howto scrape them. In the general case, one scrape configuration specifies a singlejob. In advanced configurations, this may change.

Note that the IP number and port used to scrape the targets is assembled as:. However, in someConsul setups, the relevant address is in __meta_consul_service_address.In those cases, you can use the relabelfeature to replace the special __address__ label.

The relabeling phase is the preferred and more powerfulway to filter services or nodes for a service based on arbitrary labels. Forusers with thousands of services it can be more efficient to use the Consul APIdirectly which has basic support for filtering nodes (currently by nodemetadata and a single tag).

DigitalOcean SD configurations allow retrieving scrape targets from DigitalOcean'sDroplets API.This service discovery uses the public IPv4 address by default, by that can bechanged with relabeling, as demonstrated in the Prometheus digitalocean-sdconfiguration file.

The relabeling phase is the preferred and more powerfulway to filter containers. For users with thousands of containers itcan be more efficient to use the Docker API directly which has basic support forfiltering containers (using filters).

The services role discovers all Swarm servicesand exposes their ports as targets. For each published port of a service, asingle target is generated. If a service has no published ports, a target perservice is created using the port parameter defined in the SD configuration.

The tasks role discovers all Swarm tasksand exposes their ports as targets. For each published port of a task, a singletarget is generated. If a task has no published ports, a target per task iscreated using the port parameter defined in the SD configuration.

The relabeling phase is the preferred and more powerfulway to filter tasks, services or nodes. For users with thousands of tasks itcan be more efficient to use the Swarm API directly which has basic support forfiltering nodes (using filters).

A DNS-based service discovery configuration allows specifying a set of DNSdomain names which are periodically queried to discover a list of targets. TheDNS servers to be contacted are read from /etc/resolv.conf.

The IAM credentials used must have the ec2:DescribeInstances permission todiscover scrape targets, and may optionally have theec2:DescribeAvailabilityZones permission if you want the availability zone IDavailable as a label (see below).

The relabeling phase is the preferred and more powerfulway to filter targets based on arbitrary labels. For users with thousands ofinstances it can be more efficient to use the EC2 API directly which hassupport for filtering instances.

OVHcloud SD configurations allow retrieving scrape targets from OVHcloud's dedicated servers and VPS usingtheir API.Prometheus will periodically check the REST endpoint and create a target for every discovered server.The role will try to use the public IPv4 address as default address, if there's none it will try to use the IPv6 one. This may be changed with relabeling.For OVHcloud's public cloud instances you can use the openstacksdconfig.

It reads a set of files containing a list of zero or mores. Changes to all defined files are detected via disk watchesand applied immediately. Files may be provided in YAML or JSON format. Onlychanges resulting in well-formed target groups are applied.

If Prometheus is running within GCE, the service account associated with theinstance it is running on should have at least read-only permissions to thecompute resources. If running outside of GCE make sure to create an appropriateservice account and place the credential file in one of the expected locations.

Hetzner SD configurations allow retrieving scrape targets fromHetzner Cloud API andRobot API.This service discovery uses the public IPv4 address by default, but that can bechanged with relabeling, as demonstrated in the Prometheus hetzner-sdconfiguration file.

It fetches targets from an HTTP endpoint containing a list of zero or mores. The target must reply with an HTTP 200 response.The HTTP header Content-Type must be application/json, and the body must bevalid JSON.

IONOS SD configurations allows retrieving scrape targets fromIONOS Cloud API. This service discovery uses thefirst NICs IP address by default, but that can be changed with relabeling. Thefollowing meta labels are available on all targets duringrelabeling:

The node role discovers one target per cluster node with the address defaultingto the Kubelet's HTTP port.The target address defaults to the first existing address of the Kubernetesnode object in the address type order of NodeInternalIP, NodeExternalIP,NodeLegacyHostIP, and NodeHostName.

The service role discovers a target for each service port for each service.This is generally useful for blackbox monitoring of a service.The address will be set to the Kubernetes DNS name of the service and respectiveservice port.

The pod role discovers all pods and exposes their containers as targets. For each declaredport of a container, a single target is generated. If a container has no specified ports,a port-free target per container is created for manually adding a port via relabeling.

The endpoints role discovers targets from listed endpoints of a service. For each endpointaddress one target is discovered per port. If the endpoint is backed by a pod, alladditional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.

The endpointslice role discovers targets from existing endpointslices. For each endpointaddress referenced in the endpointslice object one target is discovered. If the endpoint is backed by a pod, alladditional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.

This SD discovers "monitoring assignments" based on Kuma Dataplane Proxies,via the MADS v1 (Monitoring Assignment Discovery Service) xDS API, and will create a target for each proxyinside a Prometheus-enabled mesh.

Linode SD configurations allow retrieving scrape targets from Linode'sLinode APIv4.This service discovery uses the public IPv4 address by default, by that can bechanged with relabeling, as demonstrated in the Prometheus linode-sdconfiguration file.

Marathon SD configurations allow retrieving scrape targets using theMarathon REST API. Prometheuswill periodically check the REST endpoint for currently running tasks andcreate a target group for every app that has at least one healthy task.

By default every app listed in Marathon will be scraped by Prometheus. If not allof your services provide Prometheus metrics, you can use a Marathon label andPrometheus relabeling to control which instances will actually be scraped.See the Prometheus marathon-sd configuration filefor a practical example on how to set up your Marathon app and your Prometheusconfiguration.

The cn role discovers one target for per compute node (also known as "server" or "global zone") making up the Triton infrastructure.The account must be a Triton operator and is currently required to own at least one container.

This role uses the first address it finds in the following order: private IPv4, public IPv4, public IPv6. This can bechanged with relabeling, as demonstrated in the Prometheus scaleway-sdconfiguration file.Should an instance have no address before relabeling, it will not be added to the target list and you will not be able to relabel it.

Relabeling is a powerful tool to dynamically rewrite the label set of a target beforeit gets scraped. Multiple relabeling steps can be configured per scrape configuration.They are applied to the label set of each target in order of their appearancein the configuration file.

Initially, aside from the configured per-target labels, a target's joblabel is set to the job_name value of the respective scrape configuration.The __address__ label is set to the : address of the target.After relabeling, the instance label is set to the value of __address__ by default ifit was not set during relabeling. The __scheme__ and __metrics_path__ labelsare set to the scheme and metrics path of the target respectively. The __param_label is set to the value of the first passed URL parameter called .

If a relabeling step needs to store a label value only temporarily (as theinput to a subsequent relabeling step), use the __tmp label name prefix. Thisprefix is guaranteed to never be used by Prometheus itself.

Metric relabeling is applied to samples as the last step before ingestion. Ithas the same configuration format and actions as target relabeling. Metricrelabeling does not apply to automatically generated timeseries such as up.

This information relates to a pre-release product that may be substantially modified before it's commercially released. Microsoft makes no warranties, express or implied, with respect to the information provided here.

3a8082e126