Running additional docker containers alongside my GAE app?

[ingo]

I built a small Golang app and deployed it to GAE's Flexible Environment. Right now this will create a single Golang Docker container remotely at deploy time. For simple projects similar to this one what is the best practice to add more dependencies that run alongside the app  (like statsd)? I want to continue to run my app in a container on GAE. But I want to run publicly hosted Docker images alongside it (e.g. this one). I am looking for a way to run those multi-Docker instances still within GAE to take advantage of high level of abstraction. And if possible I want to avoid having to integrate with a Docker Registry. 

Is this feasible on GAE or do I have to switch over to Google Compute Engine or Google Container Engine?

Thanks!

[Nick (Cloud Platform Support)]

Hey Ingo,

The Flexible Environment (from this point forward, "Flex") is a kind of bridge between App Engine and Compute Engine. Flex apps run within a Docker container on a Compute Engine instance or pool of instances, and requests to the app via appspot.com are load-balanced to go to any one of these instances (depending on your scaling settings, the pool's size and growth behaviour will be dependent).

Basically, our SDK (gcloud) allows you to specificy either "vm: true" (in the case of "runtime: go/python27/java/php55/etc.") or "runtime: custom" in your app.yaml or appengine-web.xml (for java) config file.

In the case of "vm: true", your traditional App Engine app will be deployed using a special compatibility runtime to a Flex environment, and in the case of "runtime: custom", the Dockerfile in your app's root directory will be used to build the image of the resulting container which is deployed to the Flex Environment (you would need to specify the base image using the Dockerfile FROM directive, and ensure a web server process is started and listening on the appropriate port). 

So, your Flex app is already running on Compute Engine. Therefore, you could use "runtime: custom", along with the appropriate go compatibility base image, and simply specify the install / configuration instructions for statsd in your Dockerfile. Each instance of your application would have an installed and running copy of statsd, in that case. Although you may also want to simply deploy a Compute Engine instance running statsd which is reachable via its internal network IP from your app runtime instances, since that appears to be more in line with how statsd is meant to be deployed, as a standalone service within your network.

Does this help answer your questions? Let me know if anything is unclear or remaining to be discussed.

Cheers!

Nick
Cloud Platform Community Support  

[ingo]

Hey Nick,

thanks for your response! I will give your "runtime: custom" suggestion a try to see if I can stay fully within GAE.

Some comments:

• I tried to avoid installing additional dependencies into the app's Docker container. This might be acceptable for a single additional dependency. But it mixes concerns and won't scale for larger numbers of dependencies.
  
• You are saying my app runs on GCE already. I can see how that is the case for the instances themselves. But GAE comes with a few features GCE does not have built-in AFAIK: built-in load balancer, versions, traffic splitting, per-version domain names, direct access to the datastore and memcache, Cloud Debugger/Tracer, log aggregation. What of those things would I loose when migrating my example app from GAE to plain GCE? 
  
• How does this problem relate to Google Container Engine? Does this sound like a good use case for Container Engine to you?
  

Thanks,

Ingo

[Nick (Cloud Platform Support)]

Hey Ingo,

Yes, installing dependencies on each container will affect deployment time along with disk storage and CPU / RAM to one extent or another, so should be considered carefully. 

As far as the GAE services in the context of a Flex app, the JVM, python environment, golang libraries etc. in the Flex context will connect to the services by their own means, presenting the same surface to user code. Each runtime is slightly different in how connection to App Engine APIs is provided and which APIs are available, and you should check this docs page as the starting point of an investigation.

As far as services operating above the instance level, such as HTTP(S) load balancing, versioning, traffic splitting, per-version domain names, Cloud Debug/Trace, logging, and the rest, these operate in a similarly-transparent way, often requiring no extra work or tuning, although the underlying implementation details are of course distinct and there may be some differences here and there. The documentation is a great place to start, and if more concrete questions come up, don't hesitate to post them! It would be hard to list all aspects of each of these services, so I hope this high-level comment is sufficient to give a picture of the parallel-yet-distinct nature of Flexible and traditional environments.

As far as GCE purely considered and Container Engine, these are much more toward the traditional web-server model than App Engine is. You're basically provisioning virtual servers which can run web-server software or any other kind of software. Container Engine is a higher level of abstraction than individual VMs which allows you to provision and manage fleets of VMs (not usually that useful for startup apps, but becomes useful with scale). Within a python process on a GCE VM, for example, one can still interface with Cloud Platform / App Engine APIs using Google API client libraries or the Google Cloud client libraries more specifically, but the autoscaler, versioning, etc. of App Engine are not available, although there are Compute Engine parallels, such as the HTTP(S) Load Balancer.

I hope this has been helpful, let me know if you have any further questions!

Cheers,

Nick
Cloud Platform Community Support

[Stephen Deasey]

On Wed, May 11, 2016 at 3:55 AM, ingo <ingo.j...@gmail.com> wrote:
>
> But I want to run
> publicly hosted Docker images alongside it (e.g. this one). I am looking for
> a way to run those multi-Docker instances still within GAE to take advantage
> of high level of abstraction. And if possible I want to avoid having to
> integrate with a Docker Registry.

Deploy your custom docker image as a new service (module). You can
have up to 20 services:

https://cloud.google.com/appengine/docs/flexible/go/an-overview-of-app-engine

Leave your existing code deployed as it is, running as the 'default' service.

[ingo]

Hey Nick/Stephen,

Thank you very much for your responses!

I looked at Stephen's suggestion and created a new service/module within my GAE app that runs the docker-dd-agent Docker image. I updated my project on Github accordingly. Now my app consists of two modules:

* A: The default service which acts as the HTTP(S) front-end. This is a Golang app running within the flexible environment.

* B: A separate docker-dd-agent service based on a custom Dockerfile. This service is not accessible via HTTP.

Now I want that my default service (A) can communicate with the custom runtime (B) over TCP (port 8125 in this case). But after reading https://cloud.google.com/appengine/docs/flexible/custom-runtimes/build#listen_to_port_8080 I am not sure if custom runtimes are supported if they do not provide HTTP access. I disabled the (HTTP based) health check in B via app.yaml. But is this enough to allow communication from A to B?

Thanks,

Ingo

[Nick (Cloud Platform Support)]

Hey Ingo,

Custom runtimes can listen on any port specified in the app.yaml just like a regular Compute Engine instance, although these connections can't bee made through the appid.appspot.com domain or custom domains routing. You'll need to connect to the IP of the instance itself to be able to connect on a non-8080 port via something other than HTTP. The instance can hit the Compute Engine Metadata API to determine its external IP address like this:

curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip
As for health checks, you don't want to disable those, as that's vital to ensuring your app serves properly. 

I hope this answer was helpful!

Sincerely,

Nick
Cloud Platform Community Support

[ingo]

Hey Nick,

Thanks for those details! I will try this right away.

Going forward though, how do you suggest to communicate the external IP address from the custom runtime (B) to the Golang runtime A? B is a dependency which is based off a public docker container. Are you suggesting to add the curl command to the initialization of B and write the output of it to a shared location e.g. Memcache/Datastore?

If at all possible I would like to determine the IP address of B from A. It is a lot easier for me to change A or customize it's behaviour. Whereas I want B to stay very simple - a parameterized public docker image.

Thanks,

Ingo

[Nick (Cloud Platform Support)]

Hey Ingo,

The information can be retrieved from the App Engine by checking a common storage location which B could update (memcache, for example), or (and this is preferable) by having the docker container host a micro-sized HTTP server process which responds to OPTIONS requests by returning its external IP address after hitting the Metadata API to retrieve it. Both the client or A could hit this request handler on B via the usual appspot.com domain or custom domain to find its IP address in the response.

Cheers,

Nick
Cloud Platform Community Support