Google App Engine Standard - OSError: [Errno 28] No space left on device

[Linh T.]

Hi guys,

Currently, i have met issue about disk data on GAE.

test@gae-5482:~$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
none             30G   24G  4.4G  85% /
test@gae-5482:~$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
none             30G   20G  8.2G  72% /
test@gae-5482:~$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
none             30G   22G  6.6G  77% /
test@gae-5482:~$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
none             30G   26G  2.1G  93% /
test@gae-5482:~$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
none             30G   27G  1.1G  97% /
test@gae-5482:~$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
none             30G   30G     0 100% /

After i test performance, API (hosted on GAE) -> Datastore.
I call API from local (my laptop).

I check with cmd: df -h /tmp. Disk be full 30/30gb and then I not do anything, anh re-check df -h /tmp more times, data disk changed more time.
I don't know that why data disk changed and how to delete or flush it or increase size for data disk (GAE standard does not have this feature).

Please help me!
Thanks,

[George (Cloud Platform Support)]

Does your disc belong to a compute engine instance? Do you make calls to the Datastore API? Do you actually use the G Suite? More information is needed on your disc and your app's architecture. 

If you use the app engine, what is the output of the gcloud info command? 

[Linh T.]

Hi George,

Thanks for reply!

Does your disc belong to a compute engine instance?

No, i don't use compute engine.

If you use the app engine, what is the output of the gcloud info command?

gcloud infoGoogle Cloud SDK [201.0.0]Platform: [Linux, x86_64] ('Linux', 'cs-6000-devshell-vm-0a549b2f-e336-4f48-bd86-a2d2e6923d35', '3.16.0-6-amd64', '#1 SMP Debian 3.16.56-1+deb8u1 (2018-05-08)', 'x86_64', '')Python Version: [2.7.13 (default, Nov 24 2017, 17:33:09) [GCC 6.3.0 20170516]]Python Location: [/usr/bin/python2]Site Packages: [Disabled]Installation Root: [/google/google-cloud-sdk]Installed Components: datalab: [20180503] kubectl: [] container-builder-local: [] pubsub-emulator: [2018.02.02] core: [2018.05.11] app-engine-java: [1.9.63] app-engine-python: [1.9.69] app-engine-php: [ ] docker-credential-gcr: [] app-engine-python-extras: [1.9.69] beta: [2017.09.15] gsutil: [4.31] cloud-datastore-emulator: [1.4.1] cbt: [] alpha: [2017.09.15] gcd-emulator: [v1beta3-1.0.0] app-engine-go: [] bq: [2.0.33]

Do you actually use the G Suite? 

No, i also don't use G Suite.

 Do you make calls to the Datastore API?

No, i don't use Datastore API. i write flask app, which hosted on GAE. This is code in flask.
And i use ab benchmark from local to test API. i try to test with ab -c 10 -n 10000 -k https://....

from google.appengine.ext import ndb

class UserController():

token_jose = None

def __init__(self):

pass

def create_user(self):

usr = Account(first_name='test', last_name='test')

usr.put()

class Account(ndb.Model):

first_name = ndb.StringProperty()

last_name = ndb.StringProperty()

 

On Friday, May 18, 2018 at 12:40:00 AM UTC+7, George (Cloud Platform Support) wrote:

More information is needed on your disc and your app's architecture.

Web app -> GAE Standard (host API) -> Datastore.

Thanks,

[Attila-Mihaly Balazs]

Hello,

Just to clarify - you're using Google App Engine (GAE) Flexible, right? Because using GAE standard you can't SSH into the instances which are running your code.

Could you list the top 20 files for example (by size) from your /tmp directory? Something like:

du -a /tmp 2>/dev/null | sort -gr | head -n20

Cheers,

Attila

[Linh T.]

Hi Attla,

I use GAE Standard, screenshot from dashboard. 

Could you list the top 20 files for example (by size) from your /tmp directory? Something like:

du -a /tmp 2>/dev/null | sort -gr | head -n20

This is output:

128 /tmp36 /tmp/dotnet-installer32 /tmp/tmpLL2RSP32 /tmp/dotnet-installer/.dotnet20 /tmp/dotnet-installer/.dotnet/optimizationdata16 /tmp/tmp.FCMuxpIW8F16 /tmp/metadata.go16 /tmp/dotnet-installer/.dotnet/optimizationdata/2.0.012 /tmp/dotnet-installer/.dotnet/optimizationdata/2.0.0/debian.9-x648 /tmp/tmppwp2Ip8 /tmp/tmp.FCMuxpIW8F/configurations8 /tmp/dotnet-installer/.dotnet/optimizationdata/2.0.0/debian.9-x64/dotnet4 /tmp/tmux-10004 /tmp/tmp.FCMuxpIW8F/configurations/config_cloudshell-253224 /tmp/tmp.FCMuxpIW8F/active_config4 /tmp/npm-427-3430096d4 /tmp/hsperfdata_root4 /tmp/dotnet-installer/.dotnet/2.0.0_MachineId.dotnetUserLevelCache4 /tmp/dotnet-installer/.dotnet/2.0.0_IsDockerContainer.dotnetUserLevelCache4 /tmp/b63773568.patch

Thanks,

[Linh T.]

Hi Attila,

I re-send output (file attached)

[Attila-Mihaly Balazs]

Hello Linh,

Thank you for the update. Do I understand correctly that the machine which is having problems with the free space is your cloud shell instance? (ie. a small virtual machine which is allocated for you to play around with different tools - https://cloud.google.com/shell/docs/)

If so, please note that this is a temporary instance allocated to run light-weight tasks: https://cloud.google.com/shell/docs/limitations

A quick look seems to indicate that the .NET framework installer had left some big temporary files there.

Attila

[Linh T.]

Hi Attilia,

Thanks for your information.

A quick look seems to indicate that the .NET framework installer had left some big temporary files there.

Yes, i see it. But GAE standard, i use Python and i only deploy small code (Python Flask). And i only test ab benchmark from local (my laptop): ab -c 10 -n 10000 -k
and then tmp on disk data be full 30/30 and then I stop test and not do anything, but tmp disk size be changed more times: 30/30 -> 27/30 -> 22/30 -> 24/30. Today, i login, and see 22/30.

 

If so, please note that this is a temporary instance allocated to run light-weight tasks: https://cloud.google.com/shell/docs/limitations

I also have meet 1 issue:

If you do not access Cloud Shell for 120 days, we will delete your home disk. You will receive an email notification before we do so and simply starting a session will prevent its removal. 

If i deploy API on GAE production, and then API be stable, and i never or rarely change/deploy code => i don't need login Cloudshell after code be stable. If i miss email or i handover to somebody,...Code will be deleted on GAE. From my point view, GAE Standard make us not focus on server monitor, but "If you do not access Cloud Shell for 120 days, we will delete your home disk.", we still must to take time for it.

Thanks,

 

[Attila-Mihaly Balazs]

Hello Linh,

You don't need CloudShell to use/deploy/run AppEngine. So the space running out on CloudShell has no effect on how/if the Google Appengine Standard application is running.

Attila

[George (Cloud Platform Support)]

It may be worthwhile checking some examples of everyday uses, intended for Cloud Shell, as detailed on the Cloud Shell "Examples" document.