Out of space error when deploying VM

[Matt Hanson]

Hello,

I've got a VM built from a Dockerfile, the build all happens fine, both on it's own or if spawned by 'gcloud preview app deploy ...'

It gets to the end where it says 'Successfully built'

However, when doing this with gcloud, it then starts giving me a ton of errors like this:

ERROR: Failed to generate layer archive: write /var/lib/docker/graph/_tmp/7f20216219b569e64ac06dbb121792c0bf2e2e1d47492ef14a1664cb5fdf9ae7/577820362: no space left on device

At the end it then says:

Unable to push the image to/from the registry: "Failed to generate layer archive: mkdir /var/lib/docker/graph/_tmp/e581bf80f74ce1c98909d0f588132c8bc561695093399e634fe431c38c2d5b9c: no space left on device"
If you would like to report this issue, please run the following command:
gcloud feedback

It then sometimes properly deletes the VM on GCE, but other times it just hangs and I have to manually delete it through the dev console.

I'm running on Ubuntu 14.04, I have docker 1.8.1, and the latest gcloud components. Docker info:

Containers: 6
Images: 63
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 75
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-46-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 8
Total Memory: 23.46 GiB
Name: clavius
ID: BBJV:U7VU:BWQI:G4EZ:5QB3:U2IZ:ZG3G:X6KF:GCQT:GHJO:YZ5U:M5YN
Username: matthewhanson
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

I am not running out of space on my root drive where /var/lib/docker is located, nor do I have any problem with inodes.

Thanks in advance for any help you can provide, this is proving to be a difficult and elusive issue.

matt

[Patrice (Cloud Platform Support)]

Hi Matt,

Do you mind including your app.yaml as well so we can look into this?

Cheers!

[Matt Hanson]

Sure, here's my app.yaml file

###############
runtime: python27
threadsafe: true
api_version: 1
vm: true

handlers:
- url: /.*
script: miru.api.app

[Patrice (Cloud Platform Support)]

Hey Matt,

So looking a bit into this, my first guess would be that, while your PC has enough room, the image built doesn't, so your "VM" doesn't have enough memory even if your computer does.

Do you mind running gcloud preview app deploy (or run, for that matters) with "--verbosity debug"?

In general, if your error is really a size issue on your VM, I would try giving it a bigger hard drive. You can specify the size of your VM by using "resources" in your app.yaml, like so:

resources:

    cpu: 5

    memory_gb: 2

    disk_size_gb: 24

I think what may be happening is that, since the default size (if you don't specify any) is 10GB, it's possible that while your image has 23.6 GB, when you build it with gcloud, it only allocates 10 gigs to your VM, and therefore would cause these issues.

Can you try adding the resources configuration to your yaml, trying again, and telling me if this helps?

Cheers!

[Matt Hanson]

Thanks Patrice,

I tried upping the size but got the same result. Here's the output around one of the errors with debug verbosity:

INFO: Image e8f0aff7d819 already pushed, skipping

INFO: Pushing

ERROR: Failed to generate layer archive: mkdir /var/lib/docker/graph/_tmp/7ce1c1718f7a273b54fca0a8dd2a5e6480f655dc069ff521606e9820a68b2d2a: no space left on device
INFO: Exception Unable to push the image to/from the registry: "Failed to generate layer archive: mkdir /var/lib/docker/graph/_tmp/7ce1c1718f7a273b54fca0a8dd2a5e6480f655dc069ff521606e9820a68b2d2a: no space left on device" thrown in ProgressHandler. Retrying.
INFO: Starting new HTTPS connection (1): appengine.gcr.io
DEBUG: "GET /v2/ HTTP/1.1" 404 1428
INFO: Starting new HTTPS connection (1): appengine.gcr.io
DEBUG: "GET /v1/_ping HTTP/1.1" 200 None
INFO: Starting new HTTPS connection (21): 104.197.120.5

[Patrice (Cloud Platform Support)]

Hi Matt,

Just as a check, you gave my suggested 24 gigs, or more (just to be safe)? Theoretically, you can go up to 1024, so trying with the max value just to make sure 24 isn't just a bit "too tight" might be of use here. I based myself on your docker info for 24 gigs, maybe you'll need slightly more.

In any case, could you add your dockerfile here? Maybe there's something iffy in there?

Cheers

[Matt Hanson]

I tried it with 50GB, although my built image (built just using docker) is only 6GB anyway.

My Dockerfile uses another image we made, so I'm trying to replicate this problem with a simpler config, and will post shortly.

[Matt Hanson]

Ok, I tried with our basic Docker image, and that uploaded fine (snhq/basic-gae). So it looks like it's something with this Dockerfile that is causing the problem:

#####################################
FROM snhq/basic-gae

RUN apt-get update
# install numpy/scipy
RUN apt-get install -y libatlas-base-dev gfortran
RUN pip install numpy
RUN pip install scipy

# install PIL
RUN apt-get install -y libjpeg-dev libpng-dev
RUN pip install pillow

# opencv install derived from this blogpost:
# http://rodrigoberriel.com/2014/10/installing-opencv-3-0-0-on-ubuntu-14-04/

# install opencv deps
RUN apt-get update
RUN apt-get -y install libopencv-dev build-essential cmake git libgtk2.0-dev pkg-config python-dev python-numpy libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff4-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev libtbb-dev libqt4-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils

#RUN apt-get install -y software-properties-common
RUN apt-get remove libav-tools
#RUN apt-get install -y python-software-properties
#RUN add-apt-repository ppa:mc3man/trusty-media
RUN apt-get update && apt-get install -y ffmpeg

# install opencv
ENV version="3.0.0"
WORKDIR /tmp
RUN wget https://github.com/Itseez/opencv/archive/${version}.zip
RUN unzip ${version}.zip
RUN rm ${version}.zip
RUN wget https://github.com/Itseez/opencv_contrib/archive/3.0.0.zip
RUN unzip ${version}.zip

WORKDIR /tmp/opencv-${version}
RUN mkdir /tmp/opencv-${version}/build

WORKDIR /tmp/opencv-${version}/build

# http://stackoverflow.com/a/27406016/699026
#RUN apt-get install -y qt5-default

# http://stackoverflow.com/a/27347104/699026
# http://answers.opencv.org/question/25824/installation-problems-build_new_python_support/
RUN cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D WITH_FFMPEG=OFF -D WITH_CUDA=ON -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-${version}/modules -D WITH_IPP=OFF -D HAVE_QT5=OFF ..

RUN make -j $(nproc)
RUN make install
RUN echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf
RUN ldconfig

RUN rm -rf /tmp/opencv*

WORKDIR /

[Patrice (Cloud Platform Support)]

Hey Matt,

I looked a little bit further into your error and I have a question. Are you using the "--docker-build=remote" flag in your build (or simply not using the "--docker-build", as this is the default behavior)? It is possible this is the issue you're currently facing here is that the remote environment doesn't have enough memory available (I've seen it handle long dockerfiles in weird ways). Using a local build for your deployment might be an alternative solution here.

Lemme know if it solves your issue.

Cheers

[Matt Hanson]

I'm not using the "--docker-build" flag at all. I will try --docker-build=local and report back. Thanks,

matt

[Matt Hanson]

Actually I don't think this will work as they are private images, and I get an error that the base image isn't found.

[Patrice (Cloud Platform Support)]

Hi again Matt,

Thank you for being so thorough and looking into all the separate steps like that.

At this point, I think this will require some backend investigation on our part to figure out where this is holding up.

Do you mind posting the most basic "helloworld" example (or something equally trivial), but with your Dockerfile to the Issue Tracker? This way, we'll have steps to reproduce and I'll have someone from the managedVM team look into more details into this.

Thank you in advance