Prometheus connection refused to target node-exporter when curl to target works well

[yobs...@redhat.com]

Hi, all

This is a duplicate to an issue created on
https://github.com/prometheus/prometheus/issues/4419

This is a weird situation when host and port of node-exporter are reachable by curl/wget/telnet
but prometheus refused that connection.

Probably anyone has stacked with similar situation.
Please, suggest the solution.

BTW.
The output of # nmap -v 10.0.0.107

Starting Nmap 6.40 ( http://nmap.org ) at 2018-07-26 17:51 IDT
Initiating ARP Ping Scan at 17:51
Scanning 10.0.0.107 [1 port]
Completed ARP Ping Scan at 17:51, 0.03s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 17:51
Completed Parallel DNS resolution of 1 host. at 17:51, 0.00s elapsed
Initiating SYN Stealth Scan at 17:51
Scanning 10.0.0.107 [1000 ports]
Discovered open port 22/tcp on 10.0.0.107
Discovered open port 9100/tcp on 10.0.0.107
Completed SYN Stealth Scan at 17:51, 4.56s elapsed (1000 total ports)
Nmap scan report for 10.0.0.107
Host is up (0.00085s latency).
Not shown: 998 filtered ports
PORT     STATE SERVICE
22/tcp   open  ssh
9100/tcp open  jetdirect
MAC Address: FA:16:3E:58:C3:1E (Unknown)

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 4.67 seconds
Raw packets sent: 2001 (88.028KB) | Rcvd: 5 (204B)

Node-exporter vm # netstat -anlpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      541/rpcbind        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      975/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      958/master         
tcp        0      0 172.0.1.14:22           10.0.0.42:42792         ESTABLISHED 2930/sshd: cloud-us
tcp6       0      0 :::9100                 :::*                    LISTEN      1902/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      541/rpcbind        
tcp6       0      0 :::22                   :::*                    LISTEN      975/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      958/master 

Thank you
Yuri

 

[Simon Pasquier]

I would run tcpdump/wireshark on both ends.

Anything unusual in /var/log/audit/audit.log?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/517f010d-9926-47a2-8191-b3ae3cb318fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[yobs...@redhat.com]

I've ran tcpdump
But nothing interesting
# tcpdump -v -i any |grep 10.0.0.107
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
    172.17.0.2.56142 > 10.0.0.107.sun-as-jpda: Flags [S], cksum 0xb6ac (incorrect -> 0x46dc), seq 808082479, win 29200, options [mss 1460,sackOK,TS val 3195833993 ecr 0,nop,wscale 7], length 0
    172.17.0.2.56142 > 10.0.0.107.sun-as-jpda: Flags [S], cksum 0xb6ac (incorrect -> 0x46dc), seq 808082479, win 29200, options [mss 1460,sackOK,TS val 3195833993 ecr 0,nop,wscale 7], length 0
    seal12.qa.lab.tlv.redhat.com > 172.17.0.2: ICMP 10.0.0.107 tcp port sun-as-jpda unreachable, length 68
    172.17.0.2.56142 > 10.0.0.107.sun-as-jpda: Flags [S], cksum 0xb6ac (incorrect -> 0x46dc), seq 808082479, win 29200, options [mss 1460,sackOK,TS val 3195833993 ecr 0,nop,wscale 7], length 0
    seal12.qa.lab.tlv.redhat.com > 172.17.0.2: ICMP 10.0.0.107 tcp port sun-as-jpda unreachable, length 68
    172.17.0.2.56142 > 10.0.0.107.sun-as-jpda: Flags [S], cksum 0xb6ac (incorrect -> 0x46dc), seq 808082479, win 29200, options [mss 1460,sackOK,TS val 3195833993 ecr 0,nop,wscale 7], length 0
    172.17.0.2.56142 > 10.0.0.107.sun-as-jpda: Flags [S], cksum 0xb6ac (incorrect -> 0x42f1), seq 808082479, win 29200, options [mss 1460,sackOK,TS val 3195834996 ecr 0,nop,wscale 7], length 0
    172.17.0.2.56142 > 10.0.0.107.sun-as-jpda: Flags [S], cksum 0xb6ac (incorrect -> 0x42f1), seq 808082479, win 29200, options [mss 1460,sackOK,TS val 3195834996 ecr 0,nop,wscale 7], length 0
    seal12.qa.lab.tlv.redhat.com > 172.17.0.2: ICMP 10.0.0.107 tcp port sun-as-jpda unreachable, length 68
    172.17.0.2.56142 > 10.0.0.107.sun-as-jpda: Flags [S], cksum 0xb6ac (incorrect -> 0x42f1), seq 808082479, win 29200, options [mss 1460,sackOK,TS val 3195834996 ecr 0,nop,wscale 7], length 0
    seal12.qa.lab.tlv.redhat.com > 172.17.0.2: ICMP 10.0.0.107 tcp port sun-as-jpda unreachable, length 68
    172.17.0.2.56142 > 10.0.0.107.sun-as-jpda: Flags [S], cksum 0xb6ac (incorrect -> 0x42f1), seq 808082479, win 29200, options [mss 1460,sackOK,TS val 3195834996 ecr 0,nop,wscale 7], length 0
    172.17.0.2.55528 > 10.0.0.107.jetdirect: Flags [S], cksum 0xb6ac (incorrect -> 0x00c2), seq 1466136578, win 29200, options [mss 1460,sackOK,TS val 3195835480 ecr 0,nop,wscale 7], length 0
    172.17.0.2.55528 > 10.0.0.107.jetdirect: Flags [S], cksum 0xb6ac (incorrect -> 0x00c2), seq 1466136578, win 29200, options [mss 1460,sackOK,TS val 3195835480 ecr 0,nop,wscale 7], length 0
    seal12.qa.lab.tlv.redhat.com > 172.17.0.2: ICMP 10.0.0.107 tcp port jetdirect unreachable, length 68
    172.17.0.2.55528 > 10.0.0.107.jetdirect: Flags [S], cksum 0xb6ac (incorrect -> 0x00c2), seq 1466136578, win 29200, options [mss 1460,sackOK,TS val 3195835480 ecr 0,nop,wscale 7], length 0
    seal12.qa.lab.tlv.redhat.com > 172.17.0.2: ICMP 10.0.0.107 tcp port jetdirect unreachable, length 68
    172.17.0.2.55528 > 10.0.0.107.jetdirect: Flags [S], cksum 0xb6ac (incorrect -> 0x00c2), seq 1466136578, win 29200, options [mss 1460,sackOK,TS val 3195835480 ecr 0,nop,wscale 7], length 0
    172.17.0.2.55528 > 10.0.0.107.jetdirect: Flags [S], cksum 0xb6ac (incorrect -> 0xfcd7), seq 1466136578, win 29200, options [mss 1460,sackOK,TS val 3195836482 ecr 0,nop,wscale 7], length 0
    172.17.0.2.55528 > 10.0.0.107.jetdirect: Flags [S], cksum 0xb6ac (incorrect -> 0xfcd7), seq 1466136578, win 29200, options [mss 1460,sackOK,TS val 3195836482 ecr 0,nop,wscale 7], length 0
    seal12.qa.lab.tlv.redhat.com > 172.17.0.2: ICMP 10.0.0.107 tcp port jetdirect unreachable, length 68
    172.17.0.2.55528 > 10.0.0.107.jetdirect: Flags [S], cksum 0xb6ac (incorrect -> 0xfcd7), seq 1466136578, win 29200, options [mss 1460,sackOK,TS val 3195836482 ecr 0,nop,wscale 7], length 0
    seal12.qa.lab.tlv.redhat.com > 172.17.0.2: ICMP 10.0.0.107 tcp port jetdirect unreachable, length 68
    172.17.0.2.55528 > 10.0.0.107.jetdirect: Flags [S], cksum 0xb6ac (incorrect -> 0xfcd7), seq 1466136578, win 29200, options [mss 1460,sackOK,TS val 3195836482 ecr 0,nop,wscale 7], length 0
^C121818 packets captured
126968 packets received by filter
4942 packets dropped by kernel

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[yobs...@redhat.com]

Regarding /var/log/audit/audit.log
Not sure it is relevant
type=VIRT_CONTROL msg=audit(1532617239.079:430185): pid=4244 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 msg='ctr_id_short=? reason=api op=_ping vm=? vm-pid=? hostname=? user=? auid=4294967295 exe=?  exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'

Thank you

[Simon Pasquier]

We have debugged it offline and the problem was the network (it's always the network...).

Running the Prometheus container with "--network=host" solved the issue.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e5491a50-f509-480d-9d23-e59423f09611%40googlegroups.com.

[yobs...@redhat.com]

I really appreciate for your help.
Thanks a lot for quick and concrete assistance.

Yuri