Hi All
AMI: 1967.6.0
Kubernetes: 1.13.4
Kops: 1.12
Instances: m5.4xlarge
On occasion we hit an issue when the PVC is unable to format and mount the EBS correctly. The issue is caused by the linux_mount.go finding an atari partition table on a new EBS disk.
Have anyone else seen the issue at all?
Apr 08 12:44:38 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:44:38.925248 7059 mount_linux.go:212] Unmounting /var/lib/kubelet/pods/dfe4a274-59fb-11e9-bd5c-06625ead958c/volumes/kubernetes.io~secret/default-token-pwptd
Apr 08 12:44:39 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:44:39.026085 7059 mount_linux.go:212] Unmounting /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-01fb7dcaf98ae76a6
Apr 08 12:45:29 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:29.771559 7059 mount_linux.go:147] Mounting cmd (systemd-run) with arguments ([--description=Kubernetes transient mount for /var/lib/kubelet/pods/27874958-59fc-11e9-bd5c-06625ead958c/volumes/kubernetes.io~secret/default-token-pwptd --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/27874958-59fc-11e9-bd5c-06625ead958c/volumes/kubernetes.io~secret/default-token-pwptd])
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.518942 7059 mount_linux.go:441] Checking for issues with fsck on disk: /dev/xvdbr
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.533718 7059 mount_linux.go:454] `fsck` error fsck from util-linux 2.32
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.533742 7059 mount_linux.go:460] Attempting to mount disk: ext4 /dev/xvdbr /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.533758 7059 mount_linux.go:147] Mounting cmd (systemd-run) with arguments ([--description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a --scope -- mount -t ext4 -o defaults /dev/xvdbr /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a])
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: E0408 12:45:45.542277 7059 mount_linux.go:152] Mount failed: exit status 32
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.542297 7059 mount_linux.go:515] Attempting to determine if disk "/dev/xvdbr" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/xvdbr])
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.549685 7059 mount_linux.go:518] Output: "DEVNAME=/dev/xvdbr\nPTTYPE=atari\n", err: <nil>
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.549700 7059 mount_linux.go:556] Disk /dev/xvdbr detected partition table type: atari
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:47.121767 7059 mount_linux.go:441] Checking for issues with fsck on disk: /dev/xvdbr
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:47.136512 7059 mount_linux.go:454] `fsck` error fsck from util-linux 2.32
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:47.136525 7059 mount_linux.go:460] Attempting to mount disk: ext4 /dev/xvdbr /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:47.136538 7059 mount_linux.go:147] Mounting cmd (systemd-run) with arguments ([--description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a --scope -- mount -t ext4 -o defaults /dev/xvdbr /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a])
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: E0408 12:45:47.156508 7059 mount_linux.go:152] Mount failed: exit status 3
Thanks in advance
Rohith