Kubernetes PVC Atari partition

[Rohith]

Hi All

AMI: 1967.6.0

Kubernetes: 1.13.4

Kops: 1.12

Instances: m5.4xlarge

On occasion we hit an issue when the PVC is unable to format and mount the EBS correctly. The issue is caused by the linux_mount.go finding an atari partition table on a new EBS disk.

https://github.com/kubernetes/kubernetes/blob/c27b913fddd1a6c480c229191a087698aa92f0b1/pkg/util/mount/mount_linux.go#L512-L563

blkid records and atari partition which fails the code in https://github.com/kubernetes/kubernetes/blob/c27b913fddd1a6c480c229191a087698aa92f0b1/pkg/util/mount/mount_linux.go#L555-L560 "unknown error" and the loop continues. So the disk never gets formatted.

Numerous searches on kops, kubernetes, coreos came up with nothing, the closest we came to was https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=896171?? ..

Have anyone else seen the issue at all? 

Apr 08 12:44:38 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:44:38.925248    7059 mount_linux.go:212] Unmounting /var/lib/kubelet/pods/dfe4a274-59fb-11e9-bd5c-06625ead958c/volumes/kubernetes.io~secret/default-token-pwptd
Apr 08 12:44:39 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:44:39.026085    7059 mount_linux.go:212] Unmounting /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-01fb7dcaf98ae76a6
Apr 08 12:45:29 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:29.771559    7059 mount_linux.go:147] Mounting cmd (systemd-run) with arguments ([--description=Kubernetes transient mount for /var/lib/kubelet/pods/27874958-59fc-11e9-bd5c-06625ead958c/volumes/kubernetes.io~secret/default-token-pwptd --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/27874958-59fc-11e9-bd5c-06625ead958c/volumes/kubernetes.io~secret/default-token-pwptd])
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.518942    7059 mount_linux.go:441] Checking for issues with fsck on disk: /dev/xvdbr
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.533718    7059 mount_linux.go:454] `fsck` error fsck from util-linux 2.32
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.533742    7059 mount_linux.go:460] Attempting to mount disk: ext4 /dev/xvdbr /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.533758    7059 mount_linux.go:147] Mounting cmd (systemd-run) with arguments ([--description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a --scope -- mount -t ext4 -o defaults /dev/xvdbr /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a])
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: E0408 12:45:45.542277    7059 mount_linux.go:152] Mount failed: exit status 32
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.542297    7059 mount_linux.go:515] Attempting to determine if disk "/dev/xvdbr" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/xvdbr])
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.549685    7059 mount_linux.go:518] Output: "DEVNAME=/dev/xvdbr\nPTTYPE=atari\n", err: <nil>
Apr 08 12:45:45 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:45.549700    7059 mount_linux.go:556] Disk /dev/xvdbr detected partition table type: atari
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:47.121767    7059 mount_linux.go:441] Checking for issues with fsck on disk: /dev/xvdbr
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:47.136512    7059 mount_linux.go:454] `fsck` error fsck from util-linux 2.32
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:47.136525    7059 mount_linux.go:460] Attempting to mount disk: ext4 /dev/xvdbr /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: I0408 12:45:47.136538    7059 mount_linux.go:147] Mounting cmd (systemd-run) with arguments ([--description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a --scope -- mount -t ext4 -o defaults /dev/xvdbr /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2c/vol-09a87c5dc8b43995a])
Apr 08 12:45:47 ip-10-250-6-69.eu-west-2.compute.internal kubelet[7059]: E0408 12:45:47.156508    7059 mount_linux.go:152] Mount failed: exit status 3

Thanks in advance

Rohith

[wael gabsi]

Hi Rohith,

i have exactly the same issue. Did you resolve it ?

thanks and regards,

Wael