CentOS 7.3 -> 7.4 Yum Upgrade Issue - Root Filesystem Mounts Read Only on Reboot [Solved]

[Les Bell]

After this morning's (Australia time) CentOS yum update and reboot my web server (running on GCE) failed to restart. A little probing around revealed that the root filesystem was now mounted readonly, so that a bunch of services were unable to run. To cut a long story short, dmesg ended with the line:

XFS (sda1): unknown mount option [barrier=0]

and I discovered that /etc/fstab had the line

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / xfs     defaults,barrier=0 1 1

It seems that the latest version of xfsprogs expects the option to be "nobarrier", rather than "barrier=0". To remount the root filesystem and allow /etc/fstab to be edited, I used the command

mount -n -o remount,defaults /dev/sda1 /

and then used vi to edit the line to read


UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / xfs     defaults,nobarrier 1 1




After rebooting the server, everything is back to normal. The barrier option is used to force ordered writes to disk and avoid data loss on power failure. Disabling it provides a performance gain on systems which have battery-backed cache memory on the disk controllers, and I presume that it's not necessary on GCE's logical drives which are carved out of a SAN. My local CentOS 7 installs don't have this option, so I'm presuming it's specific to the GCE CentOS 7 images, and I'm posting here to save others some trouble-shooting time.




--- Les








[Navi Aujla (Google Cloud Support)]

Hello Les Bell, 

Thank you for posting the detailed information along with the fix and as well as opening an issue[1] on Public issue tracker for the same. 

As mentioned in the issue thread, I was not able to replicate the described behavior with the new source image "centos-7-v20170829", however I was able to replicate the same behavior when used the Centos source image ""centos-7-v20160301". This may be an issue for the certain images. I am verifying with the backend team and will post an update on issue thread [1]. 

[1] https://issuetracker.google.com/65658554

[Les Bell]

Thanks, Navi,

Thanks for checking the latest image, where it sounds as though the issue has been fixed. I should have mentioned that my system was built from an image back in August 2015, so it is probably an issue only for users with older, existing servers performing updates. The fix is fairly easy and hopefully panicked admins who Google the error message will find this.

Best,

--- Les

[Navi Aujla (Google Cloud Support)]

Les,

Thank you. Yes, it will definitely help. 

To add to it, You can just remove the `barrier=n` option in "/etc/fstab" file to make it look like following. 

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / xfs     defaults 1 1

Note: “defaults 1 1” option may vary depending upon the image in use and UUID will be different as well. *Only “barrier=n” option need to be removed.

I am waiting on the confirmation from the backend team regarding this issue and will post further updates on the issue thread [1]. However, feel free to post here as well to discuss further. 

[1] https://issuetracker.google.com/65658554

[Navi Aujla (Google Cloud Support)]

I have received update from backend team. I have updated the issue thread [1] with detailed information and posting the update here as well.

Issue: 

Compute Engine VM instances running RHEL 7 or CentOS 7 public images published on GCP prior to rhel-7-v20170816 (centos-7-v20170816), on restart, start with the root filesystem mounted in read only mode after Red Hat released updated packages as part of the RHEL 7.4 release on July 31, 2017 and CentOS 7 release 1708 on Sept 14, 2017. The update included an updated kernel package (version 3.10.0-693) which affected an existing mount option used for XFS filesystems in RHEL 7 and CentOS 7 causing root filesystem to mount in the read-only mode. 
------
Remediation Steps:

Google has released an updated gce-package update that fixes this automatically for VM instances running RHEL 7 or CentOS 7 images between rhel-7-v20160418 (centos-7-v20160418) and rhel-7-v20170719 (centos-7-v20170719) as a part of a gce-package update. Instances created using RHEL 7 or CentOS 7  images before rhel-7-v20160418 (centos-7-v20160418) require a manual fix. 

Fix for instances using RHEL 7 or CentOS 7 images between rhel-7-v20160418 (centos-7-v20160418) and rhel-7-
v20170719 (centos-7-v20170719)

- Run `sudo yum -y update` to update all installed packages including the `gce-disk-expand` package which 
 contains the fix.

Fix for instances using using RHEL 7 or CentOS 7  images before rhel-7-v20160418 (centos-7-v20160418):

-  Remove the “barrier” mount option in the `/etc/fstab` file by looking for `barrier=1` or ‘barrier=0’ and removing that option (the only mount option needs to be `default`). You can run following command to do this:

sudo sed -i 's/defaults,barrier[^ ,]*/defaults/' /etc/fstab

For example: In “/etc/fstab”

             UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / xfs defaults,barrier=1 0 0

To manually fix the problem remove the `barrier=n` option to make this line look like the following (UUID will be different):
             UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / xfs defaults 0 0

Note: “defaults 0 0” may vary depending upon the image in use and UUID will be different as well. *Only “barrier=n” option need to be removed. 

Remediation steps for affected instances:

- If VM instance root filesystem is already started in read-only mode after Kernel update, you can perform following 
 options: 

      gcloud compute ssh YOUR_INSTANCE --command "sudo mount -o remount,rw /dev/sda1 /"

After the filesystem is mounted in read write mode again, follow the fix step to remove the 'barrier=n' option from the '/etc/fstab' file as mentioned above.

[1] https://issuetracker.google.com/65658554

[Les Bell]

Thanks, Navi - good outcome.

--- Les

[Patrick Michael O'Leary]

Wow, Thank you.

I would never have found that!

Perhaps someday I could take you to the pub for some chips. 

G'Day.

Patrick

On Thursday, September 14, 2017 at 5:31:38 AM UTC-4, Les Bell wrote:

[Anders Brown]

Thanks so much for publishing this fix. Saved me a lot of frustration and stress. A godsend.

[Sener Koban]

Hi,

I also ran into this, and, unfortunately, I cannot remount the disk using SSH as described.

# gcloud compute ssh <INSTANCENAME> --command "sudo mount -o remount,rw /dev/sda1 /"

sudo: sorry, you must have a tty to run sudo

Is there any other way to access the fs, e.g. using serial console and boot into single user mode or something similar?

Otherwise I'll have to create copy of the disk, fix the issue and recreate my instance, right?

Cheers

Sener

 

[Sener Koban]

Hi,

Found another way to deal with it. It is possible to boot into single user mode and edit the file "/etc/fstab", deleting the barrier=0 option.

1) Enable the serial console 

# gcloud compute instances add-metadata <INSTANCENAME> --metadata=serial-port-enable=1

2) Connecto to the console

# gcloud compute --project=<PROJECTNAME> connect-to-serial-port <INSTANCENAME> --zone=<ZONE>

3) Reboot your instance

4) Use arrow-keys to pause at the boot loader, and choose the Kernel, press 'e'

5) Use arrow-keys and find the line beginning with linux16, and replace 'ro' with 'rw init=/sysroot/bin/sh', boot using CTRL+x

6) Now you have a prompt :-)

# chroot /sysroot/

# vi /etc/fstab

# reboot -f

Cheers

Sener

[Vincent Murphy]

Landed here from Google search "XFS (sda1): unknown mount option [barrier=0]."

Thank you Les Bell.

[Yahir Alejandro]

Hi Sener,

I'm unable to reproduce this and I was wondering if I'm missing something.

Is this possible without having a local account before launching the console session?

If possible, how did you rebooted your machine?

Thanks for the info!

[Tyler Gillispie]

You are Awesome Less Bell.  Thank you.  This helps; caught me right before I started pulling my hair out.