Can no longer SSH into GCP VM Instance after disk resized

[elena]

Hello,

I received a disk full message and after googling found that I can resize dynamically in the GCP dashboard under Disk so this is what I did.  Since I'm running Ubuntu I just rebooted to make sure changes took effect.  Once I reboot I could no longer SSH into my instance.  I tried another reboot.  Logging out of GCP and trying to log back in.  But now I can't even log into GCP Console.

I'm new to GCP and really not sure what is going on.  Can someone help me fix this issue ?  Please help!

Thank you

elena

[Justin Reiners]

What steps did you follow to repartition?

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/b5c59a9d-8ddb-4672-b559-27d31548648a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[elena]

Hi Justin,

Here are the steps I followed from StackOverflow:

https://stackoverflow.com/questions/22381686/how-can-size-of-the-root-disk-in-google-compute-engine-be-increased?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

Resize the persistent disk in the Google Cloud Platform Console:

1. Go to the VM instances page.
2. Click the name of the disk that you want to resize.
3. At the top of the disk details page, click Edit.
4. In the Size field, enter the new size for your disk.
5. At the bottom of the disk details page, click Save to apply your changes to the disk.
6. After you resize the disk, you must resize the disk partitions so that the operating system can access the additional space.

I skipped number 6 since I'm running Ubuntu and the changes should have taken effect automatically with a reboot.

Is there anything I can do now ?  I can't access my VM Instance in the GCP Console.

Thank you

elena

On Tuesday, June 12, 2018 at 5:51:49 PM UTC-4, Justin Reiners wrote:

What steps did you follow to repartition?

On Tue, Jun 12, 2018 at 4:13 PM, elena <elena....@gmail.com> wrote:

Hello,

I received a disk full message and after googling found that I can resize dynamically in the GCP dashboard under Disk so this is what I did.  Since I'm running Ubuntu I just rebooted to make sure changes took effect.  Once I reboot I could no longer SSH into my instance.  I tried another reboot.  Logging out of GCP and trying to log back in.  But now I can't even log into GCP Console.

I'm new to GCP and really not sure what is going on.  Can someone help me fix this issue ?  Please help!

Thank you

elena

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.

[Justin Reiners]

Google is having an issue with the console at the moment. I needed to scale production at 5pm and had to skip it because all of my vms are missing. But running. Have you tried ssh using the gcloud API?

gcloud compute ssh instancename

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/9ae3397f-b487-4d30-96dd-df2f1fe01455%40googlegroups.com.

[elena]

gcloud-cli gives an error: "Could not fetch resource" because the resource was not found.  I didn't know GCP Console was down. Does gcloud-cli work for you ? Checking GCP status just now it says the issue was resolved for everyone at 16:30 US/Pacific.   I sent feedback indicating its not resolved in my case.

https://status.cloud.google.com/incident/developers-console/18004

[Kord Campbell]

Strangely enough, this resizing issue happened to me yesterday on a box. After resizing and failing to get it to boot, I detached the drive from the instance to ensure I could keep the data. My next step would be to spin up a new box, mount that drive, and then resize the partition. At that point, I should be able to create a new instance with a larger drive and either copy it over, or simply map the drive to a directory and sim-link in the correct directory structure to restart the services.

I'm not in a huge hurry, but figured this might point you in a direction for recovery! Good luck.

K

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/9ae3397f-b487-4d30-96dd-df2f1fe01455%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/CAG9nfHyXrH3mB_HK-AcUWaWhup_XAeggntMLQuerCpLSsANjgg%40mail.gmail.com.

[elena]

Console and gcloud-cli are still not working for me.  

Kord, Thank you for the tips.  I'll give that a try if I can ever access my VM Instances again.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/9ae3397f-b487-4d30-96dd-df2f1fe01455%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.
To post to this group, send email to gce-dis...@googlegroups.com.

[Navi Aujla (Google Cloud Support)]

Hello Elena 

As per the incident summary report:" The issue with the Google Cloud Console not displaying Google Compute Engine Instances has been resolved for all affected users as of 16:30 US/Pacific." 

You can always verify the list of the compute engine instances in your project using the following command and check the instance status: 

 

gcloud compute instances list --project=[project-id]

In addition to Kord's suggestion, you can also enable connection to the serial console and login through the serial console directly and can verify the file system. 

That said, if you are still encountering problem with the Google Cloud Console not displaying Google Compute Engine Instances and gcloud not listing the instances or finding the resources, please open a private issue report using this link providing related information (eg: project id etc.) and we will verify and investigate. 

[elena]

After a few cli attempts I was prompted to update gcloud components.  Updating helped and I can now see my VM Instances but no access - ssh error port 22 (checked firewall settings and port is open).  I enabled serial console as you suggested and its looping the errors below non-stop.  I've replace my user id details to protect my account.  

 /sbin/prime-switch: 22: /sbin/prime-switch: /usr/bin/gpu-manager: not found

Jun 13 15:20:28 instance systemd[1]: lightdm.service: Main process exited, code=exited, status=1/FAILURE

Jun 13 15:20:28 instance systemd[1]: lightdm.service: Unit entered failed state.

Jun 13 15:20:28 instance systemd[1]: lightdm.service: Failed with result 'exit-code'.

Jun 13 15:20:28 instance systemd[1]: lightdm.service: Service hold-off time over, scheduling restart.

Jun 13 15:20:28 instance systemd[1]: Stopped Light Display Manager.

Jun 13 15:20:28 instance systemd[1]: Starting Light Display Manager...

Jun 13 15:20:28 instance systemd[1]: Started Light Display Manager.

Jun 13 15:20:32 instance google-accounts: INFO Removing user [GCP-USER-ID].

Jun 13 15:20:32 instance google_accounts_daemon[1704]: userdel: /etc/passwd.19226: No space left on device

Jun 13 15:20:32 instance google_accounts_daemon[1704]: userdel: cannot lock /etc/passwd; try again later.

Jun 13 15:20:32 instance google-accounts: WARNING Could not remove user [GCP-USER-ID]. Command '['userdel', '-r', 'GCP-USER-ID']' returned non-zero exit status 1.

Jun 13 15:20:32 instance google-accounts: ERROR Exception calling the response handler. [Errno 2] No usable temporary 

directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/'].#012Traceback (most recent call last):#012  File "/usr/lib

/python3/dist-packages/google_compute_engine/metadata_watcher.py", line 196, in WatchMetadata#012    handler(respon

se)#012  File "/usr/lib/python3/dist-packages/google_compute_engine/accounts/accounts_daemon.py", line 263, in Hand

leAccounts#012    self.utils.SetConfiguredUsers(desired_users.keys())#012  File "/usr/lib/python3/dist-packages/goo

gle_compute_engine/accounts/accounts_utils.py", line 284, in SetConfiguredUsers#012    mode='w', prefix=prefix, del

ete=True) as updated_users:#012  File "/usr/lib/python3.5/tempfile.py", line 679, in NamedTemporaryFile#012    pref

ix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)#012  File "/usr/lib/python3.5/tempfile.py", li

ne 269, in _sanitize_params#012    dir = gettempdir()#012  File "/usr/lib/python3.5/tempfile.py", line 435, in gett

empdir#012    tempdir = _get_default_tempdir()#012  File "/usr/lib/python3.5/tempfile.py", line 370, in _get_defaul

t_tempdir#012    dirlist)#012FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp

', '/usr/tmp', '/']

[Navi Aujla (Google Cloud Support)]

From the serial console output, it indicates that there is "No space left on device" indicating disk is full. However, as you had indicated disk was resized, make sure you had stop and start the VM instance. 

However, if you still encounter the same error messages under serial console,

Another approach that you can try:

1. Take a snapshot of the boot disk of instance in question (for backup purposes)[1].

2. Create a disk from the snapshot.

3. Create a new instance where you can ssh and attach the disk created in step 2 as secondary disk

4. Mount and access the disk on the newly created instance and verify the disk space, also can revert any changes made.

5. Once done, detach the fixed disk and create a new instance using the fixed disk as boot disk.

Note: Make sure to delete resources which are no longer required after the troubleshooting in order to avoid billing charges.

[Kord Campbell]

Just confirming that resizing a disk on a non-full/halted instance does work as expected, using --image "ubuntu-1604-xenial-v20180522".

I would assume the resize occurs after the instance boot. When the disk is full, the instance never boots fully (mine is giving errors with not having /tmp mounted).

I would suggest mounting the disk on another box to resize it, as indicated by Navi.

K

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/9f4d4900-b77a-4b2b-b721-91fc80a12565%40googlegroups.com.

[John Mc Donald]

Navi,

I'm in the same situation and but am unable to resize my disk for whatever reason. I have console access but cannot login because on startup mysql cycles through trying to start, even when I attach it to a new instance as a second drive. I've tried using a startup script to resize the disk but it doesn't seem to work. We've been down for 30 hours and I'm completely stuck on this issue. Any assistance you can provide would be greatly appreciated. 

[Navi Aujla (Google Cloud Support)]

Hello John, 

You may see the console logs moving when connecting to serial console directly, however pressing "Enter" key should get the login prompt. In addition, you can set up startup-script to stop the MySQL service in order to avoid MySQL startup cycles. 

Try stop/start the instance using the disk, which was resized? 

You can open a private issue report using this link providing all related information (eg: project id, instance name, disk name etc.) to verify and investigate any problem with the GCP. 

[John McDonald]

Navi,

Startup scripts DO NOT run, GCE giving the following error:

Aug 23 02:24:19 web-db systemd[1]: Starting Google Compute Engine Startup Scripts...
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]: Traceback (most recent call last):
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/bin/google_metadata_script_runner", line 11, in <module>
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     load_entry_point('google-compute-engine==2.4.1', 'console_scripts', 'google_metadata_script_runner')()
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/lib/python3/dist-packages/google_compute_engine/metadata_scripts/script_manager.py", line 104, in main
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     debug=bool(options.debug))
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/lib/python3/dist-packages/google_compute_engine/metadata_scripts/script_manager.py", line 66, in __init__
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     self._RunScripts(run_dir=run_dir)
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/lib/python3/dist-packages/google_compute_engine/metadata_scripts/script_manager.py", line 74, in _RunScripts
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     with _CreateTempDir(self.script_type, run_dir=run_dir) as dest_dir:
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     return next(self.gen)
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/lib/python3/dist-packages/google_compute_engine/metadata_scripts/script_manager.py", line 42, in _CreateTempDir
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     temp_dir = tempfile.mkdtemp(prefix=prefix + '-', dir=run_dir)
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     _os.mkdir(file, 0o700)
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]: OSError: [Errno 28] No space left on device: 'startup-bhkpted8'
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]: Error in sys.excepthook:
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]: Traceback (most recent call last):
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 146, in apport_excepthook
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     pr.write(f)
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/lib/python3/dist-packages/problem_report.py", line 402, in write
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     file.write(v.replace(b'\n', b'\n '))
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]: OSError: [Errno 28] No space left on device
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]: During handling of the above exception, another exception occurred:
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]: Traceback (most recent call last):
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 146, in apport_excepthook
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]:     pr.write(f)
Aug 23 02:24:20 web-db google_metadata_script_runner[2009]: OSError: [Errno 28] No space left on device

At this point I'm completely stuck, I've spent 4 days on our production server trying to get it running everything I do leads to an inability to perform the action because the disk is full. 

I need an engineer from Google to step in and resolve this issue. I am not provided the tools necessary to do this myself. We have lost tens of thousands in revenue because of this issue!

Please advise.

[Mehdi (Cloud Platform Support)]

Hello John,

I understand your situation. We will action to assist you accordingly. However before that, could you connect to serial console directly as Navi advised ? When you get the login prompt, you might be able to stop mysql cycles. In your latest message, I see most of errors are about "OSError: No space left on device". We first need to make sure if you can access the serial console directly.

[Anna Knezevic]

Hi Navi, Medhi,

I've managed to successfully resize my disk via another box, following the instructions here. The space error disappeared.

However, I am still getting the same error which runs in circles.

Sep 11 13:06:44 instance-19 systemd[1]: Starting Google Compute Engine Startup Scripts...
Sep 11 13:06:44 instance-19 systemd[1]: Reached target Cloud-init target.
Sep 11 13:06:45 instance-19 lightdm[3307]: /sbin/prime-switch: 22: /sbin/prime-switch: /usr/bin/gpu-manager: not found
Sep 11 13:06:45 instance-19 systemd[1]: lightdm.service: Main process exited, code=exited, status=1/FAILURE
Sep 11 13:06:45 instance-19 systemd[1]: lightdm.service: Unit entered failed state.
Sep 11 13:06:45 instance-19 systemd[1]: lightdm.service: Failed with result 'exit-code'.
Sep 11 13:06:45 instance-19 systemd[1]: lightdm.service: Service hold-off time over, scheduling restart.
Sep 11 13:06:45 instance-19 systemd[1]: Stopped Light Display Manager.
Sep 11 13:06:45 instance-19 systemd[1]: lightdm.service: Start request repeated too quickly.
Sep 11 13:06:45 instance-19 systemd[1]: Failed to start Light Display Manager.
Sep 11 13:06:45 instance-19 startup-script: INFO Starting startup scripts.
Sep 11 13:06:45 instance-19 startup-script: INFO No startup scripts found in metadata.
Sep 11 13:06:45 instance-19 startup-script: INFO Finished running startup scripts.
Sep 11 13:06:45 instance-19 systemd[1]: Started Google Compute Engine Startup Scripts.
Sep 11 13:06:45 instance-19 systemd[1]: Startup finished in 6.043s (kernel) + 28.861s (userspace) = 34.905s.
Sep 11 13:06:45 instance-19 google_accounts_daemon[1756]: Restarting sshd.
Sep 11 13:06:45 instance-19 systemd[1]: Stopping OpenBSD Secure Shell server...
Sep 11 13:06:45 instance-19 systemd[1]: Stopped OpenBSD Secure Shell server.
Sep 11 13:06:45 instance-19 systemd[1]: Starting Google Compute Engine Instance Setup...
Sep 11 13:06:46 instance-19 instance-setup: INFO Running set_multiqueue.
Sep 11 13:06:46 instance-19 instance-setup: INFO Setting /proc/irq/29/smp_affinity_list to 0 for device virtio1.
Sep 11 13:06:46 instance-19 instance-setup: INFO /proc/irq/29/smp_affinity_list: real affinity 0
Sep 11 13:06:46 instance-19 instance-setup: INFO Setting /proc/irq/30/smp_affinity_list to 0 for device virtio1.
Sep 11 13:06:46 instance-19 instance-setup: INFO /proc/irq/30/smp_affinity_list: real affinity 0
Sep 11 13:06:46 instance-19 instance-setup: INFO Queue 0 XPS=1 for /sys/class/net/ens4/queues/tx-0/xps_cpus
Sep 11 13:06:46 instance-19 systemd[1]: Started Google Compute Engine Instance Setup.
Sep 11 13:06:46 instance-19 systemd[1]: Starting OpenBSD Secure Shell server...
Sep 11 13:06:46 instance-19 systemd[1]: Started OpenBSD Secure Shell server.
Sep 11 13:07:50 instance-19 google-accounts: INFO Activating OS Login.
Sep 11 13:07:50 instance-19 google_accounts_daemon[1756]: Activating Google Compute Engine OS Login.

I am tempted to try to copy the debug file across the systems.

Would it work?

Anna

[Patrick (Cloud Platform Support)]

John,

Your disk is completely full at this point (hence the error messages form the serial console).

To connect to your instance using gcloud SSH, metadata needs to be written to the disk, there isn't enough space for that.

If you resize the GCE persistent disk, you should have more space, however, if the disk is completely full (which is likely) then there isn't enough space on the disk for the OS to update the partition table so even though the disk may be larger, the partition can't grow.

The only option is to take a snapshot of the disk, mount it on another VM (not as the boot disk) and resize it. Once the size has been increased, you can use it as the boot disk for a VM.

On that note, I highly recommend using a secondary disk to store your DB instead of the boot disk of your instance.

[Patrick (Cloud Platform Support)]

Anna,

are you able to connect to the instance through the serial port as Navi suggested? It will make debugging the errors easier.
Also, based on the output provided, you are getting error messages while running you startup script, but nothing indicates an issue that would block you from connecting via SSH.

Although, you do have OS Login enabled, make sure that your user account has one of the roles that allows you to connect to OS Login instances.

[Steve Larsen]

I am having nearly identical issues and have the console up at the login prompt but if all my users login with their google identities and don't set passwords how am I meant to use the serial console login now?

[Steve Larsen]

This is where I'm at currently

[pwolt...@google.com]

The serial console does require a linux username and password, which you are not provided when using Google managed SSH keys.

This being said, you can add a startup script to your instance [1] which will create a new user and password which will allow you to login here. You can find example scripts online, such as this one [2].

[1]: https://cloud.google.com/compute/docs/startupscript

[2]: https://www.cyberciti.biz/tips/howto-write-shell-script-to-add-user.html

[Steve Larsen]

That's great, very useful into. Thanks. I think in this case though because of the disk space issue the startup script is not able to be added to the machine and run.

I even tried to add a simple command to find a bunch of files to delete and it's unable to execute a two line command:

#!/bin/bash

find . -type f -name *.tgz -exec rm -f {} \;

[pwolt...@google.com]

If the disk is 100% full, that will be the case, there won't be enough room to update the appropriate files to create a new user. Your best bet is to either take a snapshot and create a larger disk or attach the disk to another instance and try to create some free space

[harshil...@gmail.com]

Hello Navi, Mehdi,

After resizing the boot disk on Ubuntu GCP instance, I am unable to open the SSH. 

Can you please help me to resolve this problem?

Thanks,

Harshil

[Jason]

Hi,

Groups is for discussions purposes and not for troubleshooting individual issues. I would highly suggest posting individual issues on serverfault where it has a high amount of visibility and users that can aid in troubleshooting your use case.

[Marcus Eagan]

Posting here late to the party in case someone else encounters the issue.

To address this error for team mbers that ran into it, I created a bigger box, mounted a snapshot of the full one, albeit with a larger disk as configured in the UI, and then used resize2fs, although that may not have been needed anyway.

The mounted disk was able to make use of expanded capacity.