Node-exportor unable to get the stat of EBS volume attached to node

psreej...@gmail.com

unread,

Aug 12, 2017, 2:48:53 PM8/12/17

to Prometheus Users

@All

I am running node-exporter in a container using the docker file given below:

FROM quay.io/prometheus/node-exporter

MAINTAINER Sreejith.Pilakkat

ENTRYPOINT [ "/bin/node_exporter" ]

EXPOSE 9100

[ec2-user@host~]$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

f78cce718b7e xxxxx.dkr.ecr.us-east-1.amazonaws.com/bxbd/node-exporter "/bin/node_exporter" 23 minutes ago Up 23 minutes 0.0.0.0:9100->9100/tcp ecs-prometheus-node-expo

[ec2-user@host ~]$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 7.8G 911M 6.8G 12% /

devtmpfs 3.9G 112K 3.9G 1% /dev

tmpfs 3.9G 0 3.9G 0% /dev/shm

/dev/xvdf 9.8G 8.1G 1.2G 88% /mnt/test-disk

I am not able to get the metrics for the disk attached

[ec2-user@host ~]$ curl http://localhost:9100/metrics | egrep xvdf

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0node_disk_bytes_read{device="xvdf"} 2.01216e+06

node_disk_bytes_written{device="xvdf"} 9.803378688e+09

node_disk_io_now{device="xvdf"} 0

node_disk_io_time_ms{device="xvdf"} 242668

node_disk_io_time_weighted{device="xvdf"} 3.2829592e+07

node_disk_read_time_ms{device="xvdf"} 1740

node_disk_reads_completed{device="xvdf"} 270

node_disk_reads_merged{device="xvdf"} 0

node_disk_sectors_read{device="xvdf"} 3930

node_disk_sectors_written{device="xvdf"} 1.9147224e+07

node_disk_write_time_ms{device="xvdf"} 3.2827916e+07

node_disk_writes_completed{device="xvdf"} 75043

node_disk_writes_merged{device="xvdf"} 32793

Can anyone check on this and let me know if i am missing something from my side.

Thanks

Sreejith

psreej...@gmail.com

unread,

Aug 12, 2017, 3:13:34 PM8/12/17

to Prometheus Users, psreej...@gmail.com

Updated my dockerfile to

FROM quay.io/prometheus/node-exporter

MAINTAINER Sreejith.Pilakkat

ENTRYPOINT [ "/bin/node_exporter" ]

CMD [ "-collector.procfs=/home/proc", "-collector.sysfs=/sys", "-collector.filesystem.ignored-mount-points=\"^/(sys|proc|dev|host|etc)($|/)\"" ]

EXPOSE 9100

But still not able to get the filesystem size of the disk attached

[ec2-user@host ~]$ curl http://localhost:9100/metrics | grep xvdf

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0node_disk_bytes_read{device="xvdf"} 2.01216e+06

node_disk_bytes_written{device="xvdf"} 9.803378688e+09

node_disk_io_now{device="xvdf"} 0

node_disk_io_time_ms{device="xvdf"} 242668

node_disk_io_time_weighted{device="xvdf"} 3.2829592e+07

node_disk_read_time_ms{device="xvdf"} 1740

node_disk_reads_completed{device="xvdf"} 270

node_disk_reads_merged{device="xvdf"} 0

node_disk_sectors_read{device="xvdf"} 3930

node_disk_sectors_written{device="xvdf"} 1.9147224e+07

node_disk_write_time_ms{device="xvdf"} 3.2827916e+07

node_disk_writes_completed{device="xvdf"} 75043

node_disk_writes_merged{device="xvdf"} 32793

Stuart Clark

unread,

Aug 12, 2017, 3:16:49 PM8/12/17

to psreej...@gmail.com, Prometheus Users

What is the command you are using to run the container? Are you volume mounting the host file system?
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

psreej...@gmail.com

unread,

Aug 12, 2017, 3:25:10 PM8/12/17

to Prometheus Users, psreej...@gmail.com

I am trying to run the node-exporter in the AWS ECS infra.

Please find the docker inspect of the container and let me know if you need any more info

[ec2-user@host ~]$ docker inspect 8b3db10d0e34

[

{

"Id": "8b3db10d0e34fa353b870b125cc8020f86d3a0aea3b0a5742f5e47778ab2066f",

"Created": "2017-08-12T19:10:46.615919275Z",

"Path": "/bin/node_exporter",

"Args": [

"-collector.procfs=/home/proc",

"-collector.sysfs=/sys",

"-collector.filesystem.ignored-mount-points=\"^/(sys|proc|dev|host|etc)($|/)\""

],

"State": {

"Status": "running",

"Running": true,

"Paused": false,

"Restarting": false,

"OOMKilled": false,

"Dead": false,

"Pid": 1098,

"ExitCode": 0,

"Error": "",

"StartedAt": "2017-08-12T19:10:47.550407421Z",

"FinishedAt": "0001-01-01T00:00:00Z"

},

"Image": "sha256:84777c63fa61b03be3d6bc1f11f66db4f3e868ec1bb90195edb6ce54e889de7f",

"ResolvConfPath": "/var/lib/docker/containers/8b3db10d0e34fa353b870b125cc8020f86d3a0aea3b0a5742f5e47778ab2066f/resolv.conf",

"HostnamePath": "/var/lib/docker/containers/8b3db10d0e34fa353b870b125cc8020f86d3a0aea3b0a5742f5e47778ab2066f/hostname",

"HostsPath": "/var/lib/docker/containers/8b3db10d0e34fa353b870b125cc8020f86d3a0aea3b0a5742f5e47778ab2066f/hosts",

"LogPath": "/var/lib/docker/containers/8b3db10d0e34fa353b870b125cc8020f86d3a0aea3b0a5742f5e47778ab2066f/8b3db10d0e34fa353b870b125cc8020f86d3a0aea3b0a5742f5e47778ab2066f-json.log",

"Name": "/ecs-prometheus-node-expo-cadvisor-7-node-exporter-a28ab7a0bfb6da9f4a00",

"RestartCount": 0,

"Driver": "devicemapper",

"MountLabel": "",

"ProcessLabel": "",

"AppArmorProfile": "",

"ExecIDs": null,

"HostConfig": {

"Binds": [

"/:/rootfs:ro",

"/sys:/sys:ro",

"/proc:/home/proc:ro"

],

"ContainerIDFile": "",

"LogConfig": {

"Type": "json-file",

"Config": {}

},

"NetworkMode": "bridge",

"PortBindings": {

"9100/tcp": [

{

"HostIp": "0.0.0.0",

"HostPort": "9100"

}

]

},

"RestartPolicy": {

"Name": "",

"MaximumRetryCount": 0

},

"AutoRemove": false,

"VolumeDriver": "",

"VolumesFrom": null,

"CapAdd": null,

"CapDrop": null,

"Dns": null,

"DnsOptions": null,

"DnsSearch": null,

"ExtraHosts": null,

"GroupAdd": null,

"IpcMode": "",

"Cgroup": "",

"Links": null,

"OomScoreAdj": 0,

"PidMode": "",

"Privileged": false,

"PublishAllPorts": false,

"ReadonlyRootfs": false,

"SecurityOpt": null,

"UTSMode": "",

"UsernsMode": "",

"ShmSize": 67108864,

"Runtime": "runc",

"ConsoleSize": [

0,

0

],

"Isolation": "",

"CpuShares": 2,

"Memory": 134217728,

"CgroupParent": "",

"BlkioWeight": 0,

"BlkioWeightDevice": null,

"BlkioDeviceReadBps": null,

"BlkioDeviceWriteBps": null,

"BlkioDeviceReadIOps": null,

"BlkioDeviceWriteIOps": null,

"CpuPeriod": 0,

"CpuQuota": 0,

"CpusetCpus": "",

"CpusetMems": "",

"Devices": null,

"DiskQuota": 0,

"KernelMemory": 0,

"MemoryReservation": 0,

"MemorySwap": 268435456,

"MemorySwappiness": -1,

"OomKillDisable": false,

"PidsLimit": 0,

"Ulimits": null,

"CpuCount": 0,

"CpuPercent": 0,

"IOMaximumIOps": 0,

"IOMaximumBandwidth": 0

},

"GraphDriver": {

"Name": "devicemapper",

"Data": {

"DeviceId": "10239",

"DeviceName": "docker-202:1-263258-da84893bcb33831266ab4f45868ead7fd53b9b709beeb34813ffd850361a5f25",

"DeviceSize": "10737418240"

}

},

"Mounts": [

{

"Source": "/",

"Destination": "/rootfs",

"Mode": "ro",

"RW": false,

"Propagation": "rprivate"

},

{

"Source": "/sys",

"Destination": "/sys",

"Mode": "ro",

"RW": false,

"Propagation": "rprivate"

},

{

"Source": "/proc",

"Destination": "/home/proc",

"Mode": "ro",

"RW": false,

"Propagation": "rprivate"

}

],

"Config": {

"Hostname": "8b3db10d0e34",

"Domainname": "",

"User": "",

"AttachStdin": false,

"AttachStdout": false,

"AttachStderr": false,

"ExposedPorts": {

"9100/tcp": {}

},

"Tty": false,

"OpenStdin": false,

"StdinOnce": false,

"Env": [

"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

],

"Cmd": [

"-collector.procfs=/home/proc",

"-collector.sysfs=/sys",

"-collector.filesystem.ignored-mount-points=\"^/(sys|proc|dev|host|etc)($|/)\""

],

"Image": "366611831214.dkr.ecr.us-east-1.amazonaws.com/bxbd/node-exporter",

"Volumes": null,

"WorkingDir": "",

"Entrypoint": [

"/bin/node_exporter"

],

"OnBuild": null,

"Labels": {

"com.amazonaws.ecs.cluster": "DevOps-Cluster",

"com.amazonaws.ecs.container-name": "node-exporter",

"com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-1:366611831214:task/c908a851-3700-4171-ab33-71d646802e19",

"com.amazonaws.ecs.task-definition-family": "prometheus-node-expo-cadvisor",

"com.amazonaws.ecs.task-definition-version": "7"

}

},

"NetworkSettings": {

"Bridge": "",

"SandboxID": "ee46b0d5f1c81d26c0a91434f01b69bff726eb79df47c3b0d2ba5d634ce644a0",

"HairpinMode": false,

"LinkLocalIPv6Address": "",

"LinkLocalIPv6PrefixLen": 0,

"Ports": {

"9100/tcp": [

{

"HostIp": "0.0.0.0",

"HostPort": "9100"

}

]

},

"SandboxKey": "/var/run/docker/netns/ee46b0d5f1c8",

"SecondaryIPAddresses": null,

"SecondaryIPv6Addresses": null,

"EndpointID": "49eb199e5228e37d063833165ac1cbd7addad56bf6ac22ff321cfab864f2b069",

"Gateway": "172.17.0.1",

"GlobalIPv6Address": "",

"GlobalIPv6PrefixLen": 0,

"IPAddress": "172.17.0.2",

"IPPrefixLen": 16,

"IPv6Gateway": "",

"MacAddress": "02:42:ac:11:00:02",

"Networks": {

"bridge": {

"IPAMConfig": null,

"Links": null,

"Aliases": null,

"NetworkID": "6636b616d03eb3bbce8a8b8474ab3153644f187c04de801c226cf4abb1ed35bd",

"EndpointID": "49eb199e5228e37d063833165ac1cbd7addad56bf6ac22ff321cfab864f2b069",

"Gateway": "172.17.0.1",

"IPAddress": "172.17.0.2",

"IPPrefixLen": 16,

"IPv6Gateway": "",

"GlobalIPv6Address": "",

"GlobalIPv6PrefixLen": 0,

"MacAddress": "02:42:ac:11:00:02"

}

]

Matthias Rampke

unread,

Aug 14, 2017, 4:00:14 AM8/14/17

to psreej...@gmail.com, Prometheus Users

Can you get /proc/diskstats and /home/proc/diskstats from the running container somehow?

Is the container privileged?

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2673d954-c026-40a7-ba21-576b1f845278%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

psreej...@gmail.com

unread,

Aug 14, 2017, 4:57:27 AM8/14/17

to Prometheus Users, psreej...@gmail.com

Please find the diskstats you requested

/ # cat /proc/diskstats

202 0 xvda 6212 33 425673 5836 2771878 2216324 298620328 25439204 0 2564452 25430960

202 1 xvda1 6071 33 422689 5768 2771878 2216324 298620328 25439204 0 2564396 25429584

202 26368 xvdcz 105870080 212837 9940085120 2480486256 1192088 697379 66912152 5938028 0 580534080 2486424980

202 26369 xvdcz1 105869862 212837 9940082080 2480486196 1192087 697379 66912144 5938012 0 580534016 2486424972

253 0 dm-0 41425 0 331424 38904 1009059 0 8072472 4098328 0 2614444 4137304

253 1 dm-1 106029533 0 9939634096 2480661340 909963 0 58839192 22982732 0 579431268 2503647652

253 2 dm-2 106029531 0 9939634080 2480683400 909963 0 58839192 22983208 0 579435160 2503671112

253 3 dm-3 171 0 17482 76 35 0 280 36 0 76 112

202 80 xvdf 270 0 3930 1740 75043 32793 19147224 32827916 0 242668 32829592

253 9 dm-9 239 0 20186 184 25 0 200 12 0 192 200

253 10 dm-10 703 0 64306 684 23 0 184 12 0 400 696

253 7 dm-7 741 0 43834 528 119 0 952 1580 0 368 2108

253 4 dm-4 260 0 22994 252 33 0 264 116 0 224 368

253 5 dm-5 366 0 34842 272 41 0 328 60 0 256 332

/ # cat /home/proc/diskstats

202 0 xvda 6212 33 425673 5836 2771887 2216353 298620632 25439208 0 2564456 25430964

202 1 xvda1 6071 33 422689 5768 2771887 2216353 298620632 25439208 0 2564400 25429588

202 26368 xvdcz 105870081 212837 9940085376 2480486256 1192091 697385 66912224 5938028 0 580534080 2486424980

202 26369 xvdcz1 105869863 212837 9940082336 2480486196 1192090 697385 66912216 5938012 0 580534016 2486424972

253 0 dm-0 41425 0 331424 38904 1009060 0 8072480 4098328 0 2614444 4137304

253 1 dm-1 106029534 0 9939634352 2480661340 909971 0 58839256 22982732 0 579431268 2503647652

253 2 dm-2 106029532 0 9939634336 2480683400 909971 0 58839256 22983208 0 579435160 2503671112

253 3 dm-3 171 0 17482 76 35 0 280 36 0 76 112

202 80 xvdf 270 0 3930 1740 75043 32793 19147224 32827916 0 242668 32829592

253 9 dm-9 239 0 20186 184 25 0 200 12 0 192 200

253 10 dm-10 703 0 64306 684 23 0 184 12 0 400 696

253 7 dm-7 741 0 43834 528 119 0 952 1580 0 368 2108

253 4 dm-4 261 0 23250 252 41 0 328 116 0 224 368

253 5 dm-5 366 0 34842 272 41 0 328 60 0 256 332

Reply all

Reply to author

Forward