Managed VM GOPATH issues

[Andrew Greene]

Hi,

I just updated to the new docker image gcr.io/google_appengine/go-compat, and I am running my app with:

ADD . /app

RUN GOPATH=/app/_gopath go build -tags appenginevm -o /app/_ah/exe

However, with the default GOPATH and GOROOT, they don't pick up my local files. For example,

cannot find package "github.com/golang/oauth2/google" in any of:

/goroot/src/github.com/golang/oauth2/google (from $GOROOT)

/app/_gopath/src/github.com/golang/oauth2/google (from $GOPATH)

Thanks,

Andrew

[Christopher Hesse]

This could be related to https://groups.google.com/forum/#!topic/google-appengine-go/eoSSKKN_VmI

Could you try deleting your Dockerfile and then deploying with appcfg.py or goapp?

[Andrew Greene]

Deploying with goapp works with Dockerfile 1, although I don't think the docker container is being updated with new packages. 

Deploying with gcloud works with Dockerfile 2, but my GOPATH seems wrong.

Deploying with gcloud doesn't work with Dockerfile 1 with error message:

INFO: /bin/bash: /app/_ah/build.sh: No such file or directory

ERROR: The command [/bin/sh -c /bin/bash /app/_ah/build.sh] returned a non-zero code: 127

Dockerfile 1:

FROM gcr.io/google_appengine/go-compat

RUN apt-get update && apt-get install -y \

curl \

build-essential \

git \

mercurial \

bzr \

ffmpeg \

libav-tools

ADD . /app

RUN /bin/bash /app/_ah/build.sh

Dockerfile 2:

FROM gcr.io/google_appengine/go-compat

RUN apt-get update && apt-get install -y \

curl \

build-essential \

git \

mercurial \

bzr \

ffmpeg \

libav-tools

ADD . /app

RUN GOPATH=/app/_gopath go build -tags appenginevm -o /app/_ah/exe

If I delete the dockerfile, gcloud serve generates the following:

FROM gcr.io/google_appengine/golang

COPY . /go/src/app

RUN go-wrapper download

RUN go-wrapper install -tags appenginevm

 which doesn't work since git is missing. I added apt-get requirements, but the build still fails because it tries to bootstrap my project by downloading my default module from github instead of using the local files, and, of course, doesn't have the necessary github permissions. Is there a way to tell gcloud serve to use my local default module files while serving another module?

Thanks,

Andrew

[Glenn Lewis]

I apologize for the deployment challenges.

There are many possible workarounds, and this should stabilize soon... but a quick one that should address most cases is to use this Dockerfile:

FROM gcr.io/google_appengine_qa/golang

COPY . /go/src/app

RUN go-wrapper download

RUN go-wrapper install -tags appenginevm

(Note the "google_appengine_qa".)

We will be making more announcements soon with more complete details.

-- Glenn

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Greene]

The go-wrapper command is still a problem because it tries to download my packages from github. Is there a way to bootstrap the container with my local files?

INFO     2015-04-29 14:51:58,685 vm_runtime_proxy_go.py:133] Go VM Deployment process failed: Docker build aborted: The command [/bin/sh -c go-wrapper download] returned a non-zero code: 1

ERROR    2015-04-29 14:51:58,686 instance.py:279] Docker build aborted: The command [/bin/sh -c go-wrapper download] returned a non-zero code: 1

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Glenn Lewis]

OK, this was not meant to be released yet, but we have had some sync problems with the Cloud SDK releases which caused these issues.

You basically need a way to bundle all your code into a single directory and then deploy it.  You can use a third-party vendoring tool or you could try something like this:

$ go get google.golang.org/appengine/cmd/aebundler

$ cat deploy.sh

#!/bin/bash -ex

rm -rf /tmp/tmpapp

mkdir /tmp/tmpapp

aebundler -o - | tar xf - -C /tmp/tmpapp

gcloud --verbosity debug preview app deploy /tmp/tmpapp

$ ./deploy.sh

-- Glenn

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

[Glenn Lewis]

I forgot to mention that if you try to go this route, your app's top-level package must be changed to "package main".

Please only try this if you are stuck deploying your Go app on Managed VMs.

-- Glenn

[Kun Li]

I'm running into the same problem and the deploy.sh works for me. Although I do need to specify yaml file instead of folder in the script.


I have two questions though during this process:


1 What's the difference between gcr.io/google_appengine_qa/golang and gcr.io/google_appengine_qa/gocompat?

2 Is there more complete details to be announced soon? Deploying go app on managed vm quickly became to be very problematic nowadays. It used to be smooth and trouble free.

[Glenn Lewis]

Re#1: "go-compat" uses the old, more-restrictive App Engine build toolchain.  "golang" uses the latest less-restrictive external Go 1.4.2 toolchain.

Re#2: Yes, I was hoping to make the announcement, but instead a larger announcement for all runtimes is being prepared, and will hopefully be delivered soon... but I don't know when.

Yes, the road has recently been bumpier than I had hoped, and this is because a number of different pieces have been changing at the same time.

I believe that all of this will be worth it because the new developer experience should be much easier, no longer requiring the need for VirtualBox, Boot2Docker, or Docker running locally.

-- Glenn

[Kun Li]

Hmmmm interesting, if your last sentence is even just half true, I think the wait will be worth it:)

[suau....@gmail.com]

I'm new to servers, Docker, vms and golang so excuse me if I'm totally wrong here.

But isn't ensuring the hosting environment to be replicable, one of the main motivations for using Docker ? I'm unsure if that is still the case if I understood RE#2) correctly.

To further explain what I mean:
I just went through the tutorials at https://cloud.google.com/nodejs/ , where each sample project is tested locally first before deployed to google app engine. While everything worked as described, the local part simply runs on my local node.js installation (no Docker involved), I could have deleted the Dockerfile and it would still run. In a non-tutorial context I would have absolutely no confidence that my container would still run properly once deployed.

My personal opinion as a rookie cloud developer: I'd rather set up docker once on my machine and have my code running in a container while developing, than always being worried about my golang version, env variables etc.

For the veteran developers, setting up Docker is even less of an issue and they'll probably have it installed anyways.

I hope this isn't taken the wrong way, I really appreciate all the hard work you're putting in there for a better developer experience.

Su-Au

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Glenn Lewis]

If you wish to develop locally with Docker, you may.  You can build an image and run a new container based on it, then test it out.

It was determined that for the majority of developers, the "Getting Started with Managed VMs" experience was painful due to the need for VirtualBox, boot2docker, and Docker.

What you are describing is more advanced usage... but by removing the requirement of these other components, the goal is to make the experience of using Managed VMs much nicer for everyone.

-- Glenn

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

[Karan Misra]

Hi Glenn,

I recently updated my app to the latest SDK, and my MVM is refusing to use the local packages from GOPATH.

I understand the solution provided by you, but how will it work in a local dev environment (expecting the image to be rebuilt when I save changes) ?

Regards,

Karan Misra

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Karan Misra]

Quite excited to see this happen. If you can give us a sneak peak, whats replacing boot2docker (docker) ?

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Glenn Lewis]

You can try it out right now following the instructions I posted.

It uses your standard Go toolchain to build and run your app locally (and totally ignores your Dockerfile).

When you deploy your app, a new image is built and deployed to production.

-- Glenn

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

[Karan Misra]

Sorry, which instructions are you referring to? I looked in the thread but only found instructions related to docker.

On Thursday, June 4, 2015 at 11:44:57 PM UTC+5:30, Glenn Lewis wrote:

You can try it out right now following the instructions I posted.

It uses your standard Go toolchain to build and run your app locally (and totally ignores your Dockerfile).

When you deploy your app, a new image is built and deployed to production.

-- Glenn

On Thu, Jun 4, 2015 at 11:11 AM, Karan Misra <kid...@gmail.com> wrote:

Quite excited to see this happen. If you can give us a sneak peak, whats replacing boot2docker (docker) ?

On Sunday, May 31, 2015 at 9:53:28 AM UTC+5:30, Glenn Lewis wrote:

Re#1: "go-compat" uses the old, more-restrictive App Engine build toolchain.  "golang" uses the latest less-restrictive external Go 1.4.2 toolchain.

Re#2: Yes, I was hoping to make the announcement, but instead a larger announcement for all runtimes is being prepared, and will hopefully be delivered soon... but I don't know when.

Yes, the road has recently been bumpier than I had hoped, and this is because a number of different pieces have been changing at the same time.

I believe that all of this will be worth it because the new developer experience should be much easier, no longer requiring the need for VirtualBox, Boot2Docker, or Docker running locally.

-- Glenn

On Sat, May 30, 2015 at 9:10 PM, Kun Li <likunar...@gmail.com> wrote:

I'm running into the same problem and the deploy.sh works for me. Although I do need to specify yaml file instead of folder in the script.


I have two questions though during this process:


1 What's the difference between gcr.io/google_appengine_qa/golang and gcr.io/google_appengine_qa/gocompat?

2 Is there more complete details to be announced soon? Deploying go app on managed vm quickly became to be very problematic nowadays. It used to be smooth and trouble free.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Karan Misra]

Is it this: https://groups.google.com/d/msg/google-appengine-go/R6q3Mqk5sIk/tK4pSfj7sBQJ

On Thursday, June 4, 2015 at 11:44:57 PM UTC+5:30, Glenn Lewis wrote:

You can try it out right now following the instructions I posted.

It uses your standard Go toolchain to build and run your app locally (and totally ignores your Dockerfile).

When you deploy your app, a new image is built and deployed to production.

-- Glenn

On Thu, Jun 4, 2015 at 11:11 AM, Karan Misra <kid...@gmail.com> wrote:

Quite excited to see this happen. If you can give us a sneak peak, whats replacing boot2docker (docker) ?

On Sunday, May 31, 2015 at 9:53:28 AM UTC+5:30, Glenn Lewis wrote:

Re#1: "go-compat" uses the old, more-restrictive App Engine build toolchain.  "golang" uses the latest less-restrictive external Go 1.4.2 toolchain.

Re#2: Yes, I was hoping to make the announcement, but instead a larger announcement for all runtimes is being prepared, and will hopefully be delivered soon... but I don't know when.

Yes, the road has recently been bumpier than I had hoped, and this is because a number of different pieces have been changing at the same time.

I believe that all of this will be worth it because the new developer experience should be much easier, no longer requiring the need for VirtualBox, Boot2Docker, or Docker running locally.

-- Glenn

On Sat, May 30, 2015 at 9:10 PM, Kun Li <likunar...@gmail.com> wrote:

I'm running into the same problem and the deploy.sh works for me. Although I do need to specify yaml file instead of folder in the script.


I have two questions though during this process:


1 What's the difference between gcr.io/google_appengine_qa/golang and gcr.io/google_appengine_qa/gocompat?

2 Is there more complete details to be announced soon? Deploying go app on managed vm quickly became to be very problematic nowadays. It used to be smooth and trouble free.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Glenn Lewis]

Yes.

-- Glenn

On Thu, Jun 4, 2015 at 11:23 AM, Karan Misra <kid...@gmail.com> wrote:

Is it this: https://groups.google.com/d/msg/google-appengine-go/R6q3Mqk5sIk/tK4pSfj7sBQJ

On Thursday, June 4, 2015 at 11:44:57 PM UTC+5:30, Glenn Lewis wrote:

You can try it out right now following the instructions I posted.

It uses your standard Go toolchain to build and run your app locally (and totally ignores your Dockerfile).

When you deploy your app, a new image is built and deployed to production.

-- Glenn

On Thu, Jun 4, 2015 at 11:11 AM, Karan Misra <kid...@gmail.com> wrote:

Quite excited to see this happen. If you can give us a sneak peak, whats replacing boot2docker (docker) ?

On Sunday, May 31, 2015 at 9:53:28 AM UTC+5:30, Glenn Lewis wrote:

Re#1: "go-compat" uses the old, more-restrictive App Engine build toolchain.  "golang" uses the latest less-restrictive external Go 1.4.2 toolchain.

Re#2: Yes, I was hoping to make the announcement, but instead a larger announcement for all runtimes is being prepared, and will hopefully be delivered soon... but I don't know when.

Yes, the road has recently been bumpier than I had hoped, and this is because a number of different pieces have been changing at the same time.

I believe that all of this will be worth it because the new developer experience should be much easier, no longer requiring the need for VirtualBox, Boot2Docker, or Docker running locally.

-- Glenn

On Sat, May 30, 2015 at 9:10 PM, Kun Li <likunar...@gmail.com> wrote:

I'm running into the same problem and the deploy.sh works for me. Although I do need to specify yaml file instead of folder in the script.


I have two questions though during this process:


1 What's the difference between gcr.io/google_appengine_qa/golang and gcr.io/google_appengine_qa/gocompat?

2 Is there more complete details to be announced soon? Deploying go app on managed vm quickly became to be very problematic nowadays. It used to be smooth and trouble free.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

[Karan Misra]

Thanks :D

Yes.

-- Glenn

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.