Show DNS name on Grafana Dashboard

[nina guo]

Hi,

Is there a way to show DNS name for a server on Grafana Dashboard?

For example:

(function_name) usrname@hostname:~> virtualip list

Configured Virtual IPs:

-----------------------

function_name

Virtual IPs State:

------------------

function_name: (info) Virtual IP xxxx is up and active on hostname

I want function_name can be shown on Grafana.

But with node exporter, no related info shows. So what should I do to get function_name be shown on Grafana?

node_uname_info{domainname="(none)", instance="xxx", job="xxx", machine="x86_64", nodename="hostname", release="4default", sysname="Linux", version="#1 }

[nina guo]

A correction:

(function_name) usrname@hostname:~> virtualip list

Configured Virtual IPs:

-----------------------

function_name on eth0

Virtual IPs State:

------------------

function_name: (info) Virtual IP xxxx is up and active on hostname

[Brian Candler]

You haven't shown the actual metric which returns the virtual IP state.  It may already have the info you need in labels.  Which exporter are you using, and can you give an example of a metric?  If you're not already collecting it, then obviously that's your first step.  e.g. if your virtual IP is managed by keepalived then you can enable its SNMP function and talk to it using snmp_exporter; or you can run your virtualip list query from a cronjob which writes out data in prometheus metric format into a file which is picked up by node_exporter's textfile collector.

The normal way to add info from node_uname_info onto some other metric is to do a join on a common label, most commonly "instance".

To do this, you need to ensure that the labels are identical, and this means removing the port number from the label, using relabelling. That is, instead of

node_uname_info{job="node",instance="foo.bar:9100",nodename="hostname",...}

you want

node_uname_info{job="node",instance="foo.bar",nodename="hostname",...}

Ditto for whichever metric shows your VIP state, i.e. it also needs instance="foo.bar" not "foo.bar:<someport>"

For more info see:

https://www.robustperception.io/controlling-the-instance-label

https://groups.google.com/forum/#!topic/prometheus-users/I4dVtwI5YrM

https://groups.google.com/forum/#!msg/prometheus-users/PPwF-2CQiSc/FsSvfCm3AwAJ

Once you've done that, you can use a many-to-one join in PromQL to pick up additional labels from node_uname_info.  For example:

node_arp_entries * on(instance) group_left(hostname,domainname) node_uname_info

For more info see:

https://www.robustperception.io/how-to-have-labels-for-machine-roles

https://www.robustperception.io/exposing-the-software-version-to-prometheus

https://www.robustperception.io/left-joins-in-promql

https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches

[nina guo]

I'm using node exporter to get the metrics. I dont have the virtual IP infos, I want to get the value first and display it on Grafana.

You kindly provided 2 ways, we don't want to install too much exporter on target systems, so we prefer to use a cronjob to get the virtual IP data.

May I know if  there is an example for how to write this cronjob?

[nina guo]

With node exporter, whether we can obtain the info of virtual IP?

[nina guo]

One more question is with node_uname_info, we get the nodename, this nodename is the hostname of the server by default?

[Brian Candler]

On Wednesday, 8 September 2021 at 10:00:42 UTC+1 ninag...@gmail.com wrote:

May I know if  there is an example for how to write this cronjob?

There are lots of examples in this repo:

https://github.com/prometheus-community/node-exporter-textfile-collector-scripts

 

Basically you just write out metrics in the prometheus text-based exposition format to a text file, and enable the node_exporter textfile collector to read it.

> With node exporter, whether we can obtain the info of virtual IP?

I don't know, because you haven't said what sort of virtual IPs you're using, except for some CLI tool "virtualip list" which I've never seen before.

You can look at the list of node_exporter collectors which are enabled by default and disabled by default - maybe there's one which meets your needs.  For example, if you're using IPVS then I see some metrics for that.  If you're using keepalived, then it exports its status via SNMP.  (You would run snmpd on that host, and snmp_exporter on your prometheus server).

> One more question is with node_uname_info, we get the nodename, this nodename is the hostname of the server by default?

I believe it's whatever "uname -n" or "hostname" shows.

[nina guo]

Thank you very much for your detailed reply.

We want to get this output "cat /etc/sysconfig/network/virtualip". So which collectors may contain this value?

[nina guo]

I checked again that the info we require can be shown with ifconfig command. We want to get "eth0:AAA" . Is there any metric include this value?

username@hostname:~> ifconfig

eth0      Link encap:Ethernet  HWaddr FA:

          xxxxxxx

          ..........................

eth0:AAA Link encap:Ethernet  HWaddr FA:

          inet addr:  Bcast:  Mask:

          UP BROADCAST RUNNING MULTICAST   Metric:

[nina guo]

So we only want to get a specific static value to be shown on Grafana, it may not be a generanl metric.

[Brian Candler]

> We want to get this output "cat /etc/sysconfig/network/virtualip". So which collectors may contain this value?

I am unaware of any.  That looks like a RedHat-specific config file.

Remember that prometheus collects *metrics* which are floating point numbers only.  Anything which is not in that format has to appear in the label of a metric, and the unique set of labels defines a new timeseries.  If you don't need any floating-point value for a metric, then the convention is to return a static value of "1" which makes it easy to combine with other metrics, using the multiplication operator.  node_uname_info is a very good example of this, as are others like node_exporter_build_info and node_network_info.

If you show us what the content of this file looks like, maybe we can suggest what a suitable metric would look like.

> I checked again that the info we require can be shown with ifconfig command. We want to get "eth0:AAA" . Is there any metric include this value?

You can see all of the metrics returned by node_exporter using this command:

curl localhost:9100/metrics

I can see metrics giving layer 2 info (MAC addresses):

node_network_info{address="0a:4e:86:6c:ab:ed",broadcast="ff:ff:ff:ff:ff:ff",device="veth6698c6af",duplex="full",ifalias="",operstate="up"} 1

- there is more info here - but none giving layer 3 info (IP addresses).  Either this is available in an optional node_exporter collector which is disabled by default - I pointed you to the documentation on those before - or you'd need to create a new metric yourself (e.g. with textfile collector), or find another exporter that does what you want.

> So we only want to get a specific static value to be shown on Grafana, it may not be a generanl metric.

Everything in prometheus is a metric.  Static values have to be labels on metrics.  Again, see how node_uname_info does this.

Of course, since Grafana is a separate piece of software, it might be possible in Grafana to extract information from some other source and combine it with other info in your dashboard.  I don't know how you'd do that, and you'd have to ask elsewhere, because this is a mailing list for prometheus, not grafana.  (Grafana has its own discussion forum).

[nina guo]

Thank you. The output is as below.

(function_name) username@hostname:~> cat /etc/sysconfig/network/virtualip

eth0:function_name

[Brian Candler]

So you could simply return

virtualip{interface="eth0",function_name="foo"} 1

and then prometheus itself would add "instance" and "job" labels when scraping.

I have searched for /etc/sysconfig/network/virtualip and I see no reference to it on the Internet, so I am guessing this is a custom file you've created, and your own script is interpreting it.

[nina guo]

#!/bin/bash

#

# Description: Expose metrics from virtualip.

#

# 

OUTPUT=$(cat "/etc/sysconfig/network/virtualip")

ET=$（cut -d":" -f 1 OUTPUT）

SHORT_NAME=$(cut -d":" -f 2 OUTPUT)

echo '# HELP show virtualip.'

echo '# TYPE virtualip'

echo 'virtualip{interface=ET,short_name=SHORT_NAME} 1' > /var/lib/node_exporter/textfile_collector/virtualip.prom.$$

mv /var/lib/node_exporter/textfile_collector/virtualip.prom.$$ /var/lib/node_exporter/textfile_collector/virtualip.prom

------------------------------------------------------------------------------------------------------------------------------------------

I'm writing the script as above.

Our Prometheus solution is deployed in k8s cluster. I have add ----collector.textfile.directory to node exporter deployment file. Do we also need to copy this script to node exporter POD?

[nina guo]

For my understanding, the script is required to be copied to target systems.

[Brian Candler]

The script will go wherever node_exporter runs.  Normally this would be on the host, but some people run node_exporter inside a pod; if it's in a pod, then you'll need to bind-mount /etc/sysconfig/network/virtualip so that the pod can read it.

There are some bugs in that script.  In particular you'll need double-quotes around the echo which does expansion, and you'll need to put double-quotes around the label values too.

echo "virtualip{interface=\"$ET\",short_name=\"$SHORT_NAME\"} 1" > ...etc

[nina guo]

Thank you very much Brian 

[nina guo]

Hi Brian,

We are trying another approach regarding the case.

Our targets are discoverd by file discvoery. If we define some of the labels by ourselves, whether the value of the label can be obtained by node_uname_info?

eg.:

If adding a new label label_A="label_a_value", then whether the value of the label can be got with node_uname_info?

[Brian Candler]

Yes, I showed that before.  You write your PromQL query like this:

(mymetric) * on(instance) group_left(hostname,domainname) node_uname_info

where "mymetric" is whatever query you would have written before.

See also:

https://www.robustperception.io/how-to-have-labels-for-machine-roles

https://www.robustperception.io/exposing-the-software-version-to-prometheus

This works as long as mymetric and node_uname_info have *exactly* the same value for the "instance" label.  In particular, you need to make sure that the exporter port number (:9100) doesn't appear in the instance label of node_uname info.

If that's not the case, then you need to fix it:

https://www.robustperception.io/controlling-the-instance-label

https://groups.google.com/forum/#!topic/prometheus-users/I4dVtwI5YrM

https://groups.google.com/forum/#!msg/prometheus-users/PPwF-2CQiSc/FsSvfCm3AwAJ

[nina guo]

Got it. Thank you very much Brian.

[nina guo]

Sorry Brian disturb you again.

We still faced problems regarding this issue.

With file discovery we defined as below:

Target:[]

Labels: dnshost

Then the dnshost can be shown on Prometheus UI like below:

1. Even the labels can be shown on Prometheus UI, we still need to write a custom metric:

#!/bin/bash

#

# Description: Expose metrics from virtualip.

#

# 

OUTPUT=$(cat "/etc/sysconfig/network/virtualip")

ET=$（cut -d":" -f 1 OUTPUT）

SHORT_NAME=$(cut -d":" -f 2 OUTPUT)

echo '# HELP show virtualip.'  

echo '# TYPE virtualip'

echo "virtualip{interface=\"$ET\",short_name=\"$SHORT_NAME\"} 1"> /var/lib/node_exporter/textfile_collector/virtualip.prom.$$

mv /var/lib/node_exporter/textfile_collector/virtualip.prom.$$ /var/lib/node_exporter/textfile_collector/virtualip.prom

2. with the query (mymetric) * on(instance) group_left(hostname,domainname) node_uname_info to get the value on Grafana. 

[nina guo]

Could you please to review if the step 1 and step 2 are right?

[nina guo]

The label "dnshost" is totally different pn Prometheus UI with the label which we want to be shown on Grafana. Is this understanding correct?