Disk is too full (important file on the disk)

[Khalid Sa]

Hello

This morning, I was trying to SSH into my instance but it seems that it is too full.  I received the following message:

Could not update /home/Name/.ssh/authorized_keys due to Disk is too full

 

Last night I used screen command to create some files which are expected to be large.

 

I have seen some answers for such a problem, but I am so worried since most of my phd research documents are stored on the disk. So I need to make sure that I am following the right instructions.

Snapshot

One of the solutions was to take a snapshot of the disk. However, in the documentations I found "If you take a snapshot of your persistent disk in a state where application data is cached without actually being written to disk, it might force a disk check or cause data loss”.

Since the ssh session was running all night, would it be inappropriate to take snapshot?

Increase disk size

I was also considering editing the instance to increase the disk size from 10G to 15G. But I am not sure if this will cause any data loss too. I don’t think there is a need for repartitioning the disk since I am using Ubuntu 14.

Startup script

Someone suggested deleting some files using a startup script. From what I understand, I can only create it at the instance creation time or by adding it to the storage unit which currently inaccessible.

 

Please recommend the safest solution

 

 

[George (Google Cloud Support)]

Hello Khalid,

As of 31 Mar 2016, you can resize a persistent disk online without stopping or rebooting the VM, without taking snapshots, and without having to restore it to a larger disk.

The blog post announcing the feature has the details, and you can see the docs for how to do this via the console:

Resize the persistent disk in the Google Cloud Platform Console:

1. Go to the VM instances page.
2. Click the name of the disk that you want to resize.
3. At the top of the disk details page, click Edit.
4. In the Size field, enter the new size for your disk.
5. At the bottom of the disk details page, click Save to apply your changes to the disk.
6. After you resize the disk, you must resize the disk partitions so that the operating system can access the additional space.

Or via CLI:

gcloud compute disks resize example-disk --size 250

I hope this helps.

Sincerely,

George

[Khalid Sa]

Thanks George for the quick reply
I tried increasing the disk size to 250G and the problem is still occurring.

ubuntu supports automatic resizing, so I have not resized the disk partitions manually

[Khalid Sa]

I believe I need to restart the instance for the new disk size to take effect.
When I press Stop, I receive this message: "Stop shuts down the instance. If the shutdown doesn't complete within 2 minutes, the instance is forced to halt. This can lead to file-system corruption. "

Is this normal??

[George (Google Cloud Support)]

Hello Khalid,

For backup purposes, you can take a snapshot of the disk before doing any changes. Once your changes are done, you can reboot the instance, either by stopping it and then starting it again which I recommend or by pressing the "Reset" button. Once the stop button is pressed, this message always shows which is expected.

I hope this helps.

Sincerely,

George

[Khalid Sa]

Thanks again!

After restarting it, the message "Disk is too full" does not appear any more.
Still, I cannot ssh the instance.
This is the last part of the serial console output :

Cloud-init v. 0.7.5 running 'modules:config' at Thu, 28 Jul 2016 16:20:24 +0000. Up 9.62 seconds.
 * Starting SSHGuard Server sshguard       [80G
[74G[ OK ]
 * Stopping Handle applying cloud-config [74G[ OK ]
 * Stopping cold plug devices [74G[ OK ]
 * Stopping log initial device creation [74G[ OK ]
 * Starting load fallback graphics devices [74G[ OK ]
 * Starting enable remaining boot-time encrypted block devices [74G[ OK ]
 * Stopping load fallback graphics devices [74G[ OK ]
 * Starting NTP server ntpd       [80G
[74G[ OK ]
open-vm-tools: not starting as this is not a VMware VM
landscape-client is not configured, please run landscape-config.
 * Restoring resolver state...       [80G
[74G[ OK ]
 * Stopping System V runlevel compatibility [74G[ OK ]
 * Starting execute cloud user/final scripts [74G[ OK ]
Cloud-init v. 0.7.5 running 'modules:final' at Thu, 28 Jul 2016 16:20:54 +0000. Up 39.94 seconds.
Cloud-init v. 0.7.5 finished at Thu, 28 Jul 2016 16:20:54 +0000. Datasource DataSourceGCE.  Up 40.01 seconds
Ubuntu 14.04.4 LTS InstanceName ttyS0
InstanceName  login:
       

[George (Google Cloud Support)]

Hello Khalid,

As per the serial-port output provided, it looks like the VM is asking for a login. In order to be able to input your login credentials, you need to enable interactive access on the serial console for the VM. You can have more information about this matter in this Help Center article.

I hope this helps.

Sincerely,

George