Run AppEngine development server without gcloud app run

[Günter Zöchbauer]

I still haven't found a satisfying solution for developing a Dart app for AppEngine managed VMs consisting of more than one local package.

Is there a way to run the AppEngine development server from my own script without `gcloud app run`? 

What things does `gcloud app run` do I would need to reproduce?

If there would be a way to pass arguments like `--volumes` or `--volumes-from` to Docker using `gcloud app run` this might work as well.

Are there any ideas or plans to improve this dev experience? 

[Anders Holmgren]

I'm having similar issues.
Tonight I started on a different path. I'm developing grinder scripts that will build the docker images in a separate build directory (eg /tmp/build).
First I clone / update using a file git url to my actual workspace. Then I run gcloud deploy there.

The main reason for this is that if you simply use ADD on your project dirs then it sucks in all the packages & test folders too and you end up creating a much bigger image than planned. I was running into issues where deploy failed because I had too many objects in the image.
This is still a WIP though.

I've been having similar thoughts to you in regards to bypassing gcloud. However i think what I really need is a better understanding of what it does around the basic docker parts.

--
You received this message because you are subscribed to the Google Groups "Dart Server-side and Cloud Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud+un...@dartlang.org.
Visit this group at http://groups.google.com/a/dartlang.org/group/cloud/.

[Günter Zöchbauer]

Currently my problem is primary about development (problems with deployment maybe later :-) )

Have you tried to add a .dockerignore file containing the directories/files you want to exclude?

[Gustav Wibling]

Hi Günter and Anders,

The only current workaround we have for using multiple local packages is to rewrite the Dockerfile to add the local packages. The rewritten version cannot rely on the google/dart-runtime, but must be based on the google/dart docker image.

I have attached a Dockerfile which does this as well as a pubspec.yaml using a local package.

Because we don't support nested packages you also have to change the directory structure for your project a bit. Basically the directory layout should be something like:

myproject/app.yaml

myproject/Dockerfile

myproject/app/             <- this is the equivalent of the "old" app dir minus the Dockerfile and app.yaml files.

myproject/pkg/             <- this is the new package directory where you put/develop your local packages.

Please take a look at the attached files and let us know if you run into issues.

Cheers,

/gustav

[Günter Zöchbauer]

Thanks for your response!

I figured this out already (see http://stackoverflow.com/questions/26883460)

I would rather not change the directory structure because I have a shared local package that I use from more than one application (currently two) and having two copies of a package that is under heavy development is not fun.

I have still a hard time to understand why it is so important for Docker to actively prevent such a scenario.

(no support for symlinks with ADD/COPY). I know their arguments but they are not very convincing.

Docker seems fine for deployment but they just don't care much about requirements during development.

Because I can't use `--volume` and `--volume-from` with gcloud it seems all possible workarounds are successfully prevented :-(

[Gustav Wibling]

Yes, using --volume/--volume-from is currently not supported in the gcloud commands. I have cc'ed the managed vm team to let them know about your request.

That said I am curious how you would like the --volume/--volume-from feature to work? AFAICT it would require deploying at least two containers? Do you do special setup when using the boot2docker vm or remote deployment?

Regarding the symlink issue you might work around that by writing a script which first copies you app/packages into a tmp directory with the above directory layout and then runs the gcloud preview app run command.

Cheers,

/gustav

[Günter Zöchbauer]

On Wednesday, November 19, 2014 3:53:03 PM UTC+1, Gustav Wibling wrote:

Yes, using --volume/--volume-from is currently not supported in the gcloud commands. I have cc'ed the managed vm team to let them know about your request.

That said I am curious how you would like the --volume/--volume-from feature to work? AFAICT it would require deploying at least two containers? Do you do special setup when using the boot2docker vm or remote deployment?

I'm just looking for a reasonable development workflow. 

Deployment is a completely different story. 

I'm fine with any workaround for deployment (copying files around, or whatever)  because this isn't necessary very often. Deploying more than one image won't be necessary.

But during development this has to be super fast and reliable for a short development cycle.

I expect that --volume/--volume-from would allow me to create a Docker image that mounts a directory (the shared local package) as volume (required the --volume argument) and then mount this volume in the image used by `gcloud preview app run` (require `--volumes-from sharedimage` argument)

I'm rather new to Docker but wasted already a lot of time figuring out how this situation can be solved.

As far as I understand using such a shared image is the only workaround (besides copying shared files into every project directory).

[Günter Zöchbauer]

On Wednesday, November 19, 2014 4:38:59 PM UTC+1, Günter Zöchbauer wrote:

On Wednesday, November 19, 2014 3:53:03 PM UTC+1, Gustav Wibling wrote:

Yes, using --volume/--volume-from is currently not supported in the gcloud commands. I have cc'ed the managed vm team to let them know about your request.

That said I am curious how you would like the --volume/--volume-from feature to work? AFAICT it would require deploying at least two containers? Do you do special setup when using the boot2docker vm or remote deployment?

I'm just looking for a reasonable development workflow. 

Deployment is a completely different story. 

I'm fine with any workaround for deployment (copying files around, or whatever)  because this isn't necessary very often. Deploying more than one image won't be necessary.

But during development this has to be super fast and reliable for a short development cycle.

I expect that --volume/--volume-from would allow me to create a Docker image that mounts a directory (the shared local package) as volume (required the --volume argument) and then mount this volume in the image used by `gcloud preview app run` (require `--volumes-from sharedimage` argument)

Here only `--volumes-from` needs gcloud support. The shared image has to be created before `gcloud preview app run` is executed. 

[Anders Holmgren]

Aahh I'd forgotten about dockerignore. That's likely exactly what I need
Thanks

[Anders Holmgren]

Thanks Gustav. I can't see the files on my phone but the layout is very similar to what I've ended up with.

I think the dockerignore is the missing ingredient for me. Btw if you haven't already I think it makes sense to provide a default dockerignore for dart projects

[Anders Holmgren]

Also you may want to consider a stagehand for appengine projects that includes the dockerignore etc

[Seth Ladd]

On Wed, Nov 19, 2014 at 1:54 PM, Anders Holmgren <andersm...@gmail.com> wrote:

Also you may want to consider a stagehand for appengine projects that includes the dockerignore etc

In the works! :)  https://github.com/google/stagehand/pull/154

[Anders Holmgren]

Awesome. Don't forget the dockerignore ;-)

[Günter Zöchbauer]

Awesome!

[Gustav Wibling]

Hi Günther,

One thing that you might do, instead of the volume export/mount, is to create a separate package image which contains your local package. You then use this new package docker image as a base for your application docker image.

Basically you create a Dockerfile to build your local package(s) into a separate image. The dockerfile would be similar to the attached which basically mimics the dart-runtime dockerfile except it is adding the package(s) to a well-known location, e.g. /packages/<packagename>

You build your docker package image (calling it e.g. local/packages) and then create an application dockerfile which is just

FROM local/packages

when you deploy your application it will have the local packages present at /packages/... since it inherits this from the base image.

Your application's pubspec.yaml file could then refer to these packages, by using "path: /packages/..." for the dependency or dependency_overrides.

Granted this approach still requires rebuilding both the packages image and the application image if the packages change, but I think this is the same if you use volumes. AFAICT the data volume containing your local packages will have to be updated if your package change requiring you to rebuild the data volume.

IMO the most seamless developer experience would be the first approach where you have your local packages in a directory next to your application and build just one docker image when deploying. It trades the multiple docker images for having a local copy of your packages which needs to be sync'ed, e.g. using git locally, with your primary copy.

Of course there are many ways to do this and if Docker had support for symlinks that would be ideal:)

Cheers,

/gustav

 

 

[Günter Zöchbauer]

Hi Gustav,

thanks a lot for looking into this issue!

I think I found a very elegant solution (at least for Linux)

http://superuser.com/a/842705/114341

Currently Docker at least doesn't complain anymore about the `linked` folder when using mount insteand of a symlink.

I want to test more but so far it works fine.

I'll post the complete solution when I'm sure all works as expected.

Günter

[Gustav Wibling]

Cool, that could be a fine workaround for linux users. Please keep me posted on how it turns out:)

Cheers,

/gustav

[Günter Zöchbauer]

Building the image works just fine now.

I use this Dockerfile:

FROM google/dart

WORKDIR /app
ENV DART_SDK /usr/lib/dart

ADD dart_run.sh /dart_runtime/

RUN chmod 755 /dart_runtime/dart_run.sh && \
 chown root:root /dart_runtime/dart_run.sh

ADD pubspec.yaml /app/
ADD pubspec.lock /app/
ADD docker/my_shared_package /my_shared_package
RUN pub get
ADD . /app/
RUN pub get --offline

## Expose ports for debugger (5858), application traffic (8080)
## and the observatory (8181)
EXPOSE 8080 8181 5858

CMD []
ENTRYPOINT ["/dart_runtime/dart_run.sh"]

Where `../my_shared_package` is mounted to `docker/my_shared_package` (see link above for more details)

Maybe there is room for improvement because now `docker/my_shared_package` is copied twice to the Docker image. One time explicitly to `/my_shared_package` and one time with `ADD . /app/` to `docker/my_shared_package` where it is redundant but I have to admit that I still don't fully understand the symlink issue with `pub get`.

At least this is working now :-)

[Gustav Wibling]

Happy to hear that it is working :)

I am not sure I understand your comment about the "pub get" symlink issue? AFAIK the issue is related to docker not following symlinks when tar'ing up the docker context, but maybe you are referring to another issue?

Cheers,

/gustav

[Günter Zöchbauer]

See https://registry.hub.docker.com/u/google/dart/ "Why run `pub get` twice" at the bottom of the page.

[Günter Zöchbauer]

Are you sure this works? 

... -v /myhosts/.pub-cache:/wherever/pub-cache -v /myhosts/projects:/app/

On Tuesday, December 9, 2014 5:30:34 PM UTC+1, daniel.domberger wrote:

Hi there everyone!

We are working on a project with various dependencies in private repos and dependencies provided via path. And of course we ran into the issues described in this thread and I played around with the workarounds and solutions provided here and on stackoverflow. I ended up with a Dockerfile that looks something like this:

FROM google/dart-runtime-base

WORKDIR /app

ENV PUB_CACHE /app/docker/mounts/pub-cache

ADD docker/mounts/path_deps/ /

ADD . /app/

RUN pub get --offline

It's not optimal, local deps are copied twice and the paths are not clean but it works. The only problem I have now is performance. Building the image after a code change takes somewhere around 45-60 seconds, without code changes with the use of cached images it's somewhere around 9s. I tried improving on this with not much success for quite a while. I reordered stuff and only copied selected directories which didn't help a lot. From what I've seen the most time consuming things when building the docker image are (a) copying the context and (b) every single step that is added to the Dockerfile where no cached image can be used adds a pretty much constant amount of time plus the time to execute this one step. I can optimize a bit on (a) but I can't really do anything about (b). As soon as there's any change in the code (be it the project or in any of the local dependencies) it starts to take really long so the development workflow is really tedious. Especially since we've got several modules that have to be started via gcloud.

This wasn't the case in previous gcloud versions with Dart < 1.8 when a Dockerfile for us looked like this:

FROM google/dart-appengine

ADD . /app/

My question now is what a gcloud with Dart >= 1.8 dev setup is supposed to look like to avoid these long container build times. And how did gcloud manage to get all the project files and dependencies in dart < 1.8 where we had a setup that was at least bearable in terms of build time.

For me any setup that takes longer than a few seconds to make code changes take effect is really tedious to work with. And from my perspective the only way to achieve this would be to use some way to tell gcloud to use docker volumes instead of rebuilding the image every time I change my code. I would imagine something that looks and behaves like the docker parameter -v:

gcloud preview app run dispatch.yaml ... -v /myhosts/.pub-cache:/wherever/pub-cache -v /myhosts/projects:/app/

Are there any plans to improve on the dev setup? How is it supposed to look and work right now?

Thanks and regards
daniel

[Günter Zöchbauer]

Dart needs a `packages` directory in the source directory with symlinks to a global dependencies cache and `pub get` is the command that analyzes dependencies and creates the "packages" directory and the symlinks.

This has to be recreated within the Docker image because ADD in the Dockerfile doesn't follow symlinks.

I have a working solution but it is still all very cumbersome and needs a lot of preparation work/workarounds, maintenance to make this work properly.

Just mounting the source folder into the Docker image during development would make this dead simple and fast. 

[Günter Zöchbauer]

On Tuesday, December 9, 2014 5:30:34 PM UTC+1, daniel.domberger wrote:

Hi there everyone!

We are working on a project with various dependencies in private repos and dependencies provided via path. And of course we ran into the issues described in this thread and I played around with the workarounds and solutions provided here and on stackoverflow. I ended up with a Dockerfile that looks something like this:

FROM google/dart-runtime-base

WORKDIR /app

ENV PUB_CACHE /app/docker/mounts/pub-cache

ADD docker/mounts/path_deps/ /

ADD . /app/

RUN pub get --offline

It's not optimal, local deps are copied twice and the paths are not clean but it works. The only problem I have now is performance. Building the image after a code change takes somewhere around 45-60

The most important thing is that the dependencies are not copied every time you modify the source but Docker uses the cache from the last run instead.

Currently I do this by creating a Docker image that only adds the dependencies and then use a Dockerfile in my Dart application like

From myown/dart-with-deps
ADD . /app/
RUN pub get --offline

But it also takes 45-60 seconds

For testing purposes I reverted back to the official Dart image (see also https://code.google.com/p/dart/issues/detail?id=21765#c11)

but I had to remove path dependencies and some other stuff to simplify my project to make it work with this approach.

`pub get` took only about 5 secs this way.

I have yet to figure out why it takes so much longer in my more complex setup. 

seconds, without code changes with the use of cached images it's somewhere around 9s. I tried improving on this with not much success for quite a while. I reordered stuff and only copied selected directories which didn't help a lot. From what I've seen the most time consuming things when building the docker image are (a) copying the context and (b) every single step that is added to the Dockerfile where no cached image can be used adds a pretty much constant amount of time plus the time to execute this one step. I can optimize a bit on (a) but I can't really do anything about (b). As soon as there's any change in the code (be it the project or in any of the local dependencies) it starts to take really long so the development workflow is really tedious. Especially since we've got several modules that have to be started via gcloud.

This wasn't the case in previous gcloud versions with Dart < 1.8 when a Dockerfile for us looked like this:

FROM google/dart-appengine

ADD . /app/

My question now is what a gcloud with Dart >= 1.8 dev setup is supposed to look like to avoid these long container build times. And how did gcloud manage to get all the project files and dependencies in dart < 1.8 where we had a setup that was at least bearable in terms of build time.

For me any setup that takes longer than a few seconds to make code changes take effect is really tedious to work with. And from my perspective the only way to achieve this would be to use some way to tell gcloud

I can't work when this takes to long. Until the app is ready for debugging I have forgotten what I was about to check. 5 seconds is already tedious. This just adds up with other things that have to be done to bring the app in a state so I can start debugging.

 

to use docker volumes instead of rebuilding the image every time I change my code. I would imagine something that looks and behaves like the docker parameter -v:

gcloud preview app run dispatch.yaml ... -v /myhosts/.pub-cache:/wherever/pub-cache -v /myhosts/projects:/app/

or

gcloud preview app run dispatch.yaml ... --docker-args"-v /myhosts/.pub-cache:/wherever/pub-cache -v /myhosts/projects:/app/"

or whatever.

I put this in a startup script anyway. I don't care about the syntax, there just has to be any way to do it.
 

[Anders Holmgren]

Yeah I agree the docker stuff is a real killer for the dev loop speed which is normally super fast w dart and one of the things I love about dart.

I'm luckily rather OC about minimising my dependencies on my paas and have two versions of my app. One that has no gcloud dependencies and works off in memory data. This is how I develop mostly and preserves the fast dev loop.

The second one has the full dependencies and must be run with docker and the gcloud scripts. And you guessed it takes a while to start.

It would be nice if gcloud had a mode where we could keep our server outside of docker and communicate with the gcloud resources like datastore and memcache that are inside a docker container that we keep running.

[Günter Zöchbauer]

On Thursday, November 20, 2014 1:00:09 PM UTC+1, Gustav Wibling wrote:

Hi Günther,

One thing that you might do, instead of the volume export/mount, is to create a separate package image which contains your local package. You then use this new package docker image as a base for your application docker image.

Basically you create a Dockerfile to build your local package(s) into a separate image. The dockerfile would be similar to the attached which basically mimics the dart-runtime dockerfile except it is adding the package(s) to a well-known location, e.g. /packages/<packagename>

You build your docker package image (calling it e.g. local/packages) and then create an application dockerfile which is just

FROM local/packages

when you deploy your application it will have the local packages present at /packages/... since it inherits this from the base image.

Your application's pubspec.yaml file could then refer to these packages, by using "path: /packages/..." for the dependency or dependency_overrides.

Granted this approach still requires rebuilding both the packages image and the application image if the packages change, but I think this is the same if you use volumes. AFAICT the data volume containing your local packages will have to be updated if your package change requiring you to rebuild the data volume.

IMO the most seamless developer experience would be the first approach where you have your local packages in a directory next to your application and build just one docker image when deploying. It trades the multiple docker images for having a local copy of your packages which needs to be sync'ed, e.g. using git locally, with your primary copy.

Of course there are many ways to do this and if Docker had support for symlinks that would be ideal:)

Cheers,

/gustav

 

Hi Gustav,

thanks for your ideas.

I have already considered/tried all these suggestions.

The problem is that the parts are permanently changing, so creating a base image that contains parts of it doesn't help because this causes lots of daily cumbersome maintenance work.

My app is split in three packages (client, server, shared) and I'm permanently developing in all three of them (at least two).

This makes this approach impossible for my local packages and is very cumbersome for 3rd party dependencies.

Dart and the IDE have their own ideas how the source directory layout should look like to work properly, which causes other limitations and problems if not followed.

As stated before, I don't care if this is necessary for deployment or integration testing but during development requiring any of this is just a pain.

Docker itself is a big PITA except maybe when it's used for deployment only. 

I read through enough Docker discussions to learn that deployment is the only thing they care about.

They don't care at all about how cumbersome their limitations are for development.

They can of course follow their own goals, but for me this makes Docker inappropriate for a development environment.

Now gcloud adds its own limitations (for example missing support for passing custom arguments to Docker) and the whole approach becomes merely a collection of limitations. 

I'll probably find a way to make it work but I hate to work around such a lot of artificial limitations just to be able to start developing.

Just adding support for passing custom arguments to Docker would simplify the setup a lot.

I don't expect any improvements from Docker so this is my only hope ...

Cheers

Günter

[Günter Zöchbauer]

On Wednesday, December 10, 2014 12:30:48 PM UTC+1, Anders Holmgren wrote:

Yeah I agree the docker stuff is a real killer for the dev loop speed which is normally super fast w dart and one of the things I love about dart.

I'm luckily rather OC about minimising my dependencies on my paas and have two versions of my app. One that has no gcloud dependencies and works off in memory data. This is how I develop mostly and preserves the fast dev loop.

The second one has the full dependencies and must be run with docker and the gcloud scripts. And you guessed it takes a while to start.

It would be nice if gcloud had a mode where we could keep our server outside of docker and communicate with the gcloud resources like datastore and memcache that are inside a docker container that we keep running.

This approach would be too good to be true so I didn't even bother to ask ;-)

There seems to be a local installable Datastore server available but there is no Dart API available to address it.

The remote API (googleapis_beta) doesn't allow to change the Server URL (Google servers hardcoded).

[Anders Holmgren]

Without having looked at the internals of the code, it strikes me as something doable.
The code running in your server would need to hit ports exposed via a running docker container that contains the test datastore, memcache etc

I'm no doubt overlooking something though.

In case it is of any value to you let me share my set up. I suffer from the same problems as you but maybe I'm not burdened quite as much

I have two git repos

1/ App
This contains all my code except anything that depends on appengine (gcloud etc). This is a fully functional app that just stores everything in memory.

It is broken into client, server and common (like yours is).

The client has interfaces for the objects that communicate with the server and I have test implementations for those that don't talk to the server. This allows me to user "run in Dartium" with no server.

I can also run with the server by plugging in the real implementations here

2/ AE App
This imports App and contains just a server project.
It is made up of
- datastore implementations of my Repository interfaces
- similar for memcache
- adaptor code for things like integrating appengine's context.user with shelf_auth etc

One thing I'm missing here is unit tests against datastore and memcache as I haven't researched how to set this up.

So in summary this gives me 3 ways to run in development

1/ Just browser => run in dartium
2/ full client server outside of docker, using in memory implementations of my Repository interfaces
3/ in docker with the full code including Datastore Repositories etc

I'm greedy though and would like a fourth way as mentioned

4/ outside of docker running against docker contained Datastore and memcache

Then a second or two to restart the server and I'm testing changes (unless of course I changed indexes etc).

It would also be nice to have the same logic that watches the files and restarts the appengine based server to run against this server so I don't even have to restart it.

And in the future it could pull in Peter's incremental compilation work for virtually instant updates ;-)

[Günter Zöchbauer]

@Anders

Seems it is working fast for me now

My Dockerfiles

base image

FROM google/debian:wheezy

MAINTAINER Günter Zöchbauer <guenter@gzoechbauer.com>

ENV DEBIAN_FRONTEND noninteractive
ENV DART_SDK /usr/lib/dart
ENV PATH $DART_SDK/bin:$PATH
ENV DART_VERSION 1.9.0-dev.0.0

RUN \
  apt-get -q update && \
  DEBIAN_FRONTEND=noninteractive && \
  apt-get install --no-install-recommends -y -q \
    apt-transport-https \
    apt-utils \
    apt-show-versions \
    ca-certificates \
    curl \
    git

## (removed net-tools sudo procps telnet apt-show-versions)
RUN \
  curl https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
  curl https://storage.googleapis.com/download.dartlang.org/linux/debian/dart_unstable.list > \
    /etc/apt/sources.list.d/dart_unstable.list && \
  apt-get update && \
  apt-get install dart=$DART_VERSION-1 && \
  rm -rf /var/lib/apt/lists/* && \

ADD dart_run.sh /dart_runtime/

RUN \
  chmod 755 /dart_runtime/dart_run.sh && \
  chown root:root /dart_runtime/dart_run.sh

WORKDIR /app

## Expose ports for debugger (5858), application traffic (8080)
## and the observatory (8181)
EXPOSE 8080 8181 5858

## Colorful bash prompt for `docker exec`
RUN \
  echo "alias la='ls -lahFLH --color --group-directories-first'" >> /root/.bashrc && \
  echo "PROMPT_COMMAND='PS1=\"\[\e]0;\W\007\e[38;5;69m\]\u@\h \[\e[38;5;36m\]\w\n\[\e[38;5;222m\]\[\e[38;5;47m\] $$\[\e[0m\] \"'" >> /root/.bashrc

CMD []
ENTRYPOINT ["/dart_runtime/dart_run.sh"]

############################################################
### Add the following lines to the Dockerfile that uses this image as base image

## local path dependencies
# ADD some_pkg /bwu_pkg # for each local dependency

##
#RUN pub get
#ADD . /app/
#RUN pub get --offline
############################################################

Dockerfile in my app package-root

FROM zoechi/bwu-dart-dev:latest

## Some convenient packages for `docker exec`
RUN \
 apt-get update && \
 DEBIAN_FRONTEND=noninteractive && \
 apt-get install -y \
 apt-show-versions \
 net-tools \
 sudo procps \
 telnet

## add path dependency mounted to `docker/bwu_server`
ADD docker/bwu_server /bwu_server

## print current Dart version and whether a newer Dart deb package is available
RUN \
 dart --version && \
 apt-show-versions dart

ADD pubspec.* /app/
RUN pub get

ADD . /app/
RUN pub get --offline

I don't know what caused the long-running `pub-get` and pre-build previously and why they are now gone but after the initial start 

a source file modification currently causes a delay of about 10 secs until the app is up again (pub get about 3 secs).

Currently the only hack is mounting the path dependency and ADDing it to an appropriate place inside the image.

All other deviations from the Dockerfile provided by the Dart team, are just for convenience or my personal taste.

Debugging still only works occasionally.

[Günter Zöchbauer]

Didn't use it much yet though, but tests look promising.

[Günter Zöchbauer]

Maybe someone finds this useful as well. 

I use this script to start `gcloud` which deals with problems I have after stopping `gcloud` with ctrl+c and ctrl+z by killing remnants which are kept running for unknown reasons.

#!/bin/bash

## Force download of new base image (was useful occasionally)
#docker pull $(awk '/^FROM[ \t\r\n\v\f]/ { print /:/ ? $2 : $2":latest" }' Dockerfile)

# kill older script if still running
if [ $(pgrep --exact -u ${USER} -f -c "/bin/bash tool/run.sh") -eq "2" ]; then
 pkill -SIGKILL --exact -u ${USER} -f -o "/bin/bash tool/run.sh"
fi

## change `some-app` with your app name
RUNNING_CONTAINER=`docker ps | grep some-app.default.dev1 | awk '{print $1}'`
if [ -n ${RUNNING_CONTAINER} ]; then
 docker stop ${RUNNING_CONTAINER}
fi

gcloud --verbosity debug preview app run app.yaml index.yaml

[Anders Holmgren]

Thanks Günter. I'll try this out on the weekend

[Søren Gjesse]

Anders,

There is actually an option for 4/, which is running the App Engine API Server directly. The API server is part of the App Engine SDK, and we are using it for running tests in the appengine package. If you look at https://github.com/dart-lang/appengine/blob/master/tool/run_tests.sh you will see that it expects the environment variable APPENGINE_API_SERVER.

The API server is in <Cloud SDK root>/platform/google_appengine/api_server.py

and takes a number of arguments. I just tested running it like this:

$ $CLOUD_SDK/platform/google_appengine/api_server.py \

   -A dev~test-application \

  --api_port 4444 \

  --high_replication \

  --datastore_path /tmp/datastore

To run an app engine application outside the normal development server requires that a number of environment variables are set. This worked for my application:

$ GAE_LONG_APP_ID=test-application \

  GAE_MODULE_NAME=default \

  GAE_MODULE_VERSION=version \

  GAE_PARTITION=dev \

  API_PORT=4444 \

  API_HOST=127.0.0.1 \

  dart bin/server.dart

In the Dart Editor you cannot set environment variables for each launch configuration, so they have to be set globally before starting the Dart Editor. In WebStorm it is possible to have run configuration specific environment variables.

This simple setup will of cause not support everything the normal development server support. Some of the issues are:

  * Only one application at the time as it is always listening on port 8080 (can easily be made configurable)

  * The users API (mocking this shouldn't be that difficult)

  * The modules API

  * No health-checks (should not be a problem)

  * All HTTP headers are direct from the client (no x-appengine- headers)

  * The admin web interface is not available

  * Probably other stuff as well

This is all experimental, but it is one solution for a simpler developer setup, which of cause does not match the deployment environment as closely as the development server.

Running the API Server using Docker is also possible as the image google/cloud-sdk with the Cloud SDK is on hub.docker.com.

Use the following Dockerfile

FROM google/cloud-sdk

EXPOSE 4444

ENTRYPOINT ["/google-cloud-sdk/platform/google_appengine/api_server.py", \

  "-A", "dev~test-application", \

  "--api_port", "4444", \

  "--high_replication", \

  "--datastore_path", "/tmp/datastore"]

Build and run

$ docker build -t api_server .

$ docker run -d -p 4444:4444 api_server

Change API_HOST above to 192.166.59.103 (of wherever your Docker containers are) and run.

Regards,

Søren Gjesse

---

Søren Gjesse

Software Engineer, Google Denmark

CVR nr. 28 86 69 84

[Günter Zöchbauer]

This is great stuff, thanks a lot Søren!

I assume this way even debugging from WebStorm will work (doesn't have remote debugging support yet).

Actually I already asked in another thread if this is possible, because I'm pretty sure that this is the most efficient and flexible dev setup for most development work. Glad to hear it is!

Have a nice Weekend

Günter

[Søren Gjesse]

Günter,

When running outside Docker you can just use normal debugging in WebStorm.

Regards,

Søren

[Tomasz Kubacki]

Doesn't work as of Google Cloud SDK 0.9.81 path for api_server.py  changed or it doesn't exists anymore.