Corrupted Persistent Disk on gcloud compute VM

[Adam McFarlin]

Hi folks,

First some info:

Machine Type: n1-standard-2 (2 vCPUs, 7.5 GB memory)

Additional Disks Type: Standard persistent disk

Additional Disks Mode: Read/Write

Additional Disks When Deleting Disks: Keep disk

Last week I mounted a persistent Disk to a gcloud vm, saved some work to it, then moved onto another project. When I came back today to review the data on the persistent disk, I found it had been unmounted. I tried mounting as before and received this error...

$ mount -o discard,defaults /dev/sdb /mnt/disks/data

mount: /dev/sdb: can't read superblock

That doesn't looks good... I started to investigate

$ lsblk

NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT

sdb      8:16   0 1000G  0 disk 

sda      8:0    0   10G  0 disk 

└─sda1   8:1    0   10G  0 part /

$ fdisk -l

Disk /dev/sdb: 1000 GiB, 1073741824000 bytes, 2097152000 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

$ cat /etc/fstab

LABEL=cloudimg-rootfs / ext4 defaults 0 0

I'm guessing the disk was corrupted somehow, so perhaps I can repair the disk? 

$ fsck.ext4 -v /dev/sdb

e2fsck 1.42.13 (17-May-2015)

/dev/sdb has unsupported feature(s): metadata_csum

e2fsck: Get a newer version of e2fsck!

I'm well past my abilities at this point... why is the e2fsck installed not compatible with the standard persistent disk... more important how could the mounted persistent disk get corrupt in the first place? And any ideas on how to resolve?

Thanks so much for your help.

[Adam McFarlin]

EDIT: this is Ubuntu 16.04

[Justin Reiners]

Depending on the filesystem you most of the time, cannot repair while the drive is in use, try taking a snapshot, create a new disk from it, mount the disk on another instance, run fsck from there.

updates can very well cause some things to get out of date (or lack of updates)

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/94d35a42-2c76-4ec3-b661-e3b712f9528d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adam McFarlin]

Thanks Justin, will try and report back.

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.

[Justin Reiners]

I was just going to ask that. Thanks. You can still take a snapshot of the bad disk, and maybe repair it to working order on another tiny instance.

Then just shut down the original VM, remove the root disk, and add the new one created from the snapshot, and boot it back up. Then you can keep your IPs.

--

© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/24c3c885-34ac-4f60-829f-bed34a95c653%40googlegroups.com.

[Adam McFarlin]

Thanks Justin, but the snapshot recovered disk and new VM have the exact same problems as before...

I don't think this is an issue with the VM, I think this is an issue with the disk itself, or somehow Google Cloud corrupted my disk when doing an update.

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.

[Adam McFarlin]

Does anyone have any insight on how this might have happened? And how to resolve? Should we not rely on additional disks to be able to reliably persist data over 7 days?

[Adam McFarlin]

I seem unable to repair disks myself... e2fsck: Get a newer version of e2fsck!

[Adam McFarlin]

SOLVED: For anyone running into this later Google Cloud may change your additional disk underneath you in a way that is no longer compatible with your OS version. If you are running an "older" version of Ubuntu like me, spin up a VM with the latest Ubuntu, recover the additional disk with fsck.ext4 (assuming ext4), and move the data to google storage cloud, or out of the google system altogether. 

[Digil (Google Cloud Platform Support)]

Hello Adam,

I understand that you had an issue with your persistent disk on the Google Cloud Platform virtual machine instances. This specific issue of yours is not an intended behavior of the resource on the GCP platform. Because of this reason, this issue must be investigated deeply.

Please open a defect report using this private-issue thread, mentioning your project-number, instance name and detailed description of the error logs/report. I’ll make sure that this will be investigated and will provide you a fix(or the root cause of the issue) as soon as possible.

[Adam McFarlin]

I appreciate that Digil, I've made the private-issue