App Engine and Container Builder are making deployments faster

[Justin Beckwith]

TL;DR - Deployments are about to get faster; you don’t need to do a thing.  

The App Engine Flexible Environment (previously know as Managed VMs) has always used Docker to package and run your App Engine FE applications.  At first we required developers to install Docker, and build containers locally before publishing to App Engine.  Last year we improved that process by automatically provisioning a virtual machine in your project, and performing the Docker build and push during deployment -  eliminating the need to have Docker installed locally.  

In the next release of the Google Cloud SDK, we’re making it even better by switching all Flexible Environment builds to our shiny new service, Google Cloud Container Builder.  Container Builder provides a fast and secure way to build Docker images and publish them to Google Container Registry.

You don’t need to do anything to start using the new service. The only change you should notice is faster builds and deployments.  If you do run into problems, please let us know by responding to the thread!  You can always fall back to the in-project VM model by setting the app/use_cloud_build flag in gcloud to false:

gcloud config set app/use_cloud_build true|false

We recognize that build time and reliability is important for App Engine FE, and we’re hoping this change makes everyone's lives just a little easier.

Thanks!

--

Justin Beckwith | Google Cloud Platform | @justinbeckwith | http://jbeckwith.com

[pdknsk]

I'm noticing an improvement, but it's relative. Deployment is still slow, and slower* than using local docker. I ran some tests with the three different ways to deploy.

The time reported is from the second run each, to not include setup time and to take advantage of caching. It's the best-case scenario of no changes between runs.

remote (old): 6m44

remote (new): 5m23 

local: 3m24

What I noticed is that the new builder doesn't cache containers. When it was first announced I thought that it's basically like local docker running remotely, with caching in cloud storage. It doesn't seem to be. With local docker, an unchanged Dockerfile doesn't trigger a build process. With the remote builder it does.

I also tried the scenario of a Dockerfile with changed RUN command.

remote (old): 7m18

remote (new): 6m5

local: 8m36

Now local loses here, because it downloads packages und uploads container deltas at non-Google throughput. This scenario doesn't happen regularly though, depending on how often you change your Dockerfile. More commonly it just picks up app/ dir changes, which is a small delta of 100K or so.

[pdknsk]

PS. This is the Dockerfile I used. For the second test, I just changed the pinned lxml version.

FROM gcr.io/google_appengine/python-compat

RUN apt-get update -yq \

 && apt-get install --no-install-recommends -yq libxslt1.1 libxml2 \

 && apt-get install --no-install-recommends -yq gcc libxslt1-dev libxml2-dev python-dev \

 && pip install -Uq lxml==3.4.4 google-api-python-client \

 && apt-get purge --auto-remove -yq gcc libxslt1-dev libxml2-dev python-dev

ADD . /app

[Justin Beckwith]

Glad to hear we could shave over a minute off of your builds! For the questions on caching, I've added Andrew and Scott :)

--
You received this message because you are subscribed to the Google Groups "google-cloud-sdk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-s...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-cloud-sdk.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sdk/afca56a8-0f61-431d-ac4c-efd7cbe52dd8%40googlegroups.com.

[Andrew Jessup]

Thanks for the detailed feedback pdknsk. You're quite right, we don't (yet) support caching of build artifacts - though it's not out of scope for the service at some point. I'd encourage you to request this feature on the App Engine issue tracker which will help us consolidate feedback around this feature request.

In the interim, have you tried speeding up builds by creating a new base in Google Cloud Repositories off gcr.io/google_appengine/python-compat that includes libxml, and using that in your Dockerfile? I'd be curious to see what performance improvements in application builds you see.

--

Andrew Jessup | Product Manager, Google Cloud Platform | jes...@google.com 

[Justin Beckwith]

If this is blocking you right now, the easiest work around is to fall back to the previous build system using:

gcloud config set app/use_cloud_build false

Scott & Andrew may have a better idea of what's up with the timeout.

On Fri, Mar 25, 2016 at 5:31 PM, Carter Maslan <car...@camio.com> wrote:

Our deployment is failing with this:

ERROR: (gcloud.preview.app.deploy) Error Response: [4] DEADLINE_EXCEEDED

Any tips on fix/workaround for that?

Here's the tail end of the deployment messages:

-----------

Successfully built 78753090e089

PUSH

The push refers to a repository [appengine.gcr.io/gcloud/<appid>.<module>.<version>] (len: 1)

Sending image list

Pushing repository appengine.gcr.io/gcloud/camio-test.default.mongoose (1 tags)

Image 4bfc20f0ed66 already pushed, skipping

Image 3bc7cdb36864 already pushed, skipping

Image 096d9403d234 already pushed, skipping

ERROR: (gcloud.preview.app.deploy) Error Response: [4] DEADLINE_EXCEEDED

--
You received this message because you are subscribed to the Google Groups "App Engine Managed VMs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-manage...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

@CarterMaslan

car...@camio.com
650-248-2466

[pdknsk]

> In the interim, have you tried speeding up builds by creating a new base in Google Cloud Repositories off gcr.io/google_appengine/python-compat that includes libxml, and using that in your Dockerfile? I'd be curious to see what performance improvements in application builds you see.

I just tried. Works well, and cuts down deployment time by almost 100s in my case, according to builder logs.

Of course this requires docker installed, and the whole purpose of the remote builder is to avoid that. Unless you wrestle directly with the REST API of the Container Builder. It does not seem impossible to have this happen transparently in the background in some way. Or if not transparently, then on request. Basically what I did manually, but automatically and via Container Builder, including an automatic rewrite (or print-out) of the Dockerfile. (This should probably not be done recursively.)

But no matter how fast the docker-related part gets, there is still the other component: VMs. According to the serial console output, the VM takes about 90s to boot fully once it's provisioned, if things go well. It can take double that. I notice that one culprit is this.

Pulling GAE_FULL_APP_CONTAINER: [...]

Done pulling app container

Which is blocking and takes between 20s and 120s. Seems like a lot, for what appears to be a network operation. (It's Google!) I thought that maybe it's not the network, but the 10GB disk I'm using. Neither 100GB nor a faster VM helped.

I use manual scaling with multiple instances, and there can be huge variance even with the same configuration.

With 100GB and n1-standard-2.

(1) Mar 27 01:40:14 Pulling GAE_FULL_APP_CONTAINER: [...]

(1) Mar 27 01:41:30 Done pulling app container

(2) Mar 27 01:40:01 Pulling GAE_FULL_APP_CONTAINER: [...]

(2) Mar 27 01:40:37 Done pulling app container

And 10GB and f1-micro.

(1) Mar 27 01:53:07 Pulling GAE_FULL_APP_CONTAINER: [...]

(1) Mar 27 01:53:50 Done pulling app container

(2) Mar 27 01:53:12 Pulling GAE_FULL_APP_CONTAINER: [...]

(2) Mar 27 01:53:51 Done pulling app container

PS. The variance makes the benchmarks in my previous post somewhat useless.

[pdknsk]

I've noticed an opportunity for concurrency, as the docker-part and VM-part are run strictly sequentially. According to the logs, it takes the VM about 30s to get from being "inserted" to the point where it pulls the container. It could already start as the docker-part runs, and then wait for the container to be ready.

[Justin Beckwith]

Greetings Erik,  

This seems unrelated to the container build change, but I'll reach out off of the thread.  That obviously doesn't sound right.  

Thanks!

On Thu, Mar 31, 2016 at 11:16 AM, Erik Troan <e...@pendo.io> wrote:

We deploy go based runtimes (using the appengine runtime, not a custom one) and builds have gotten significantly slower and less reliable for us. Setting "use_cloud_build" to false doesn't seem to make a difference. 

Until this most recent change we were using appcfg.py to deploy our mvms, so I am comparing that process to using gcloud preview app deploy. They now take 5-10 minutes each, and fail around 30% of the time.

Erik

--

You received this message because you are subscribed to the Google Groups "App Engine Managed VMs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-manage...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pdknsk]

Unless you wrestle directly with the REST API of the Container Builder.

I withdraw that sentence: no wrestling required. It's very simple actually. Still, I wouldn't mind an addition to gcloud.

$ gcloud build-container <dir> <name> 

[montasir]

We keep getting this error when we deploy, even the Cloud Storage JSON API is enabled

Failure setting up GCS logging: failed to create GCS logging client: error creating logfile: googleapi: Error 403: Access Not Configured. Cloud Storage JSON API has not been used in project xxxxxxxxx before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/storage_api/overview?project=xxxxxxxxxxxx then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured
{
metadata: {
severity:
"INFO"
projectId:
"xxxxxxxxxxxx"
serviceName:
"cloudbuild.googleapis.com"
labels: {…}
timestamp:
"2016-04-10T14:58:07.894227359Z"
projectNumber:
"xxxxxxxxxxxxxx"
}
insertId:
"96b6799d-5933-42b5-8861-137b2a398c09-0"
log:
"cloudbuild"
}

gcloud version 104.0.0

PS: Cloud Storage JSON API is enabled

[Zachary Newman]

Hi montasir@,

Do you mind filing an issue over at <https://code.google.com/p/google-cloud-sdk/issues/list>? That's a better venue for us to provide support with bugs.

In the meantime, can you try disabling the API and re-enabling it? 

Cheers,

Zack

--

You received this message because you are subscribed to the Google Groups "google-cloud-sdk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-s...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-cloud-sdk.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sdk/7bce4af2-b290-4c7d-85a1-50f3fbe654b9%40googlegroups.com.

[Montasir Asaira]

OK I'll open case there, and yeah disabling/enabling it made no difference.