dev_appserver.py great build optimization with "goapp install"

[Esse]

Is there a way to configure dev_appserver or go-app-builder to run faster using installed packages and not rebuilding everything?

The command dev_appserver runs this while serving and catching a file change:

~/go_appengine/goroot/bin/go-app-builder -app_base ~/module -arch 6 -dynamic -goroot ~/go_appengine/goroot -nobuild_files ^^$ -unsafe -gopath ~/golang -binary_name _go_app -extra_imports appengine_internal/init -work_dir /var/.../.../.../...-go-bin -gcflags -I,~/go_appengine/goroot/pkg/darwin_amd64_appengine -ldflags -L,~/go_appengine/goroot/pkg/darwin_amd64_appengine x.go y.go module.go

It takes about 12 seconds to build a lot of dependencies.

But using goapp install we can circumvent installed packages:

$ goapp build (~12s)
$ goapp install (~12s)
$ goapp build (~0.6s)
... edit main package ...
$ goapp build (~ 1s)

How could we achieve this in dev_appserver.py?

Even if we could do that, would nasty things happen?

[Strom]

Oh man, I was prepared for some compilation slowdown with the new SDK bringing 1.6. I was not ready for this 12 second build after every edit. Really slows down my work.

[Admin UBA]

This is not related to Go versions but to building every static dependency over and over. I haven't benchmarked 1.5, but it was already slow.

You may try the new SDK and roll back anytime.

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/k-r_hJ7W1dw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Strom]

There never was a 1.5 on app engine. I haven't investigated this in depth, but the latest SDK with 1.6 is the first one where I noticed the compilation went from instant to 10+ seconds.

To unsubscribe from this group and all its topics, send an email to google-appengine-go+unsub...@googlegroups.com.

[Admin UBA]

True, 1.4
I will roll back and benchmark both today.
Anyway, I'm interested in making the dev_appserver use the installed packages. The only necessary step would be run install before running the server, and probably resetting at any additional package installed. I don't know the depths of the server yet, so the call for help. Will return with the benchmarks.

To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/k-r_hJ7W1dw/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

[Esse]

Here it is, 1.9.34 (1.4.2) vs 1.9.35 (1.6)
Well, not a good benchmark, but we can see the impact anyway.

$ goapp version
go version go1.4.2 (appengine-1.9.34) darwin/amd64

$ time goapp build -a
real 0m3.099s
user 0m6.263s
sys 0m1.304s

$ time goapp build
real 0m0.708s
user 0m0.540s
sys 0m0.145s

$ time goapp install -a
real 0m3.761s
user 0m6.758s
sys 0m1.468s

$ time goapp build
real 0m0.210s
user 0m0.129s
sys 0m0.067s

go-app-builder: build timing: 61×6g (4.555s total), 0×6l (0 total)

$ goapp version
go version go1.6 (appengine-1.9.35) darwin/amd64

$ time goapp build -a
real 0m18.144s
user 0m42.085s
sys 0m5.798s

$ time goapp build
real 0m18.084s
user 0m42.625s
sys 0m5.780s

$ time goapp install -a
real 0m20.125s
user 0m44.057s
sys 0m6.109s

$ time goapp build
real 0m0.709s
user 0m0.274s
sys 0m0.201s

go-app-builder: build timing: 61×compile (10.284s total), 0×link (0 total)

[Christian Ohler]

I don't know if this still works with the latest SDK (we're a few versions behind), but we've been running dev_appserver with this patch to enable incremental builds, which leads to a large speedup:

--- a/go_appengine/google/appengine/tools/devappserver2/go_application.py 2015-10-25 18:28:34.000000000 +1100

+++ b/go_appengine/google/appengine/tools/devappserver2/go_application.py 2015-10-25 16:42:59.000000000 +1100

@@ -247,10 +247,7 @@

     gab_stdout, _ = self._run_gab(gab_args, env={})

     return gab_stdout

-  def _build(self):

-    assert self._go_file_to_mtime, 'no .go files'

-    logging.debug('Building Go application')

-

+  def _build_with_gab(self):

     gab_args = [

         '-binary_name', '_go_app',

         '-extra_imports', 'appengine_internal/init',

@@ -263,6 +260,35 @@

     logging.debug('Build succeeded:\n%s\n%s', gab_stdout, gab_stderr)

     self._go_executable = os.path.join(self._work_dir, '_go_app')

+  def _build_without_gab(self):

+    executable = os.path.join(self._work_dir, '_go_app')

+    args = [os.path.join(_SDKROOT, 'goapp'), 'build', '-i', '-o', executable]

+    env = {

+      'GOROOT': GOROOT,

+    }

+    if 'GOPATH' in os.environ:

+      env['GOPATH'] = os.environ['GOPATH']

+    process = safe_subprocess.start_process(args,

+                                            stdout=subprocess.PIPE,

+                                            stderr=subprocess.PIPE,

+                                            env=env)

+    stdout, stderr = process.communicate()

+    if process.returncode:

+      raise go_errors.BuildError(

+        '(Executed command: %s)\n%s\n%s' % (' '.join(args),

+                                            stdout, stderr))

+    logging.debug('Build succeeded:\n%s\n%s', stdout, stderr)

+    self._go_executable = executable

+

+  def _build(self):

+    assert self._go_file_to_mtime, 'no .go files'

+    logging.debug('Building Go application')

+    ### HACK(ohler): _build_without_gab enables incremental builds.

+    if os.getenv('HACK_DEV_APPSERVER_USE_GO_APP_BUILDER') is None:

+      self._build_without_gab()

+    else:

+      self._build_with_gab()

+

   def maybe_build(self, maybe_modified_since_last_build):

     """Builds an executable for the application if necessary.

To make it work, we have an extra file main_appengine.go in the same directory as app.yaml, containing this:

// The code in this file would normally be generated by

// go-app-builder, but our hacked App Engine SDK bypasses that, so we

// need this file checked in.

// +build appengine

package main

// extra_imports from go_application.py

import (

_ "appengine_internal/init"

)

// main from go-app-builder/synthesizer.go

import (

internal "appengine_internal"

)

func main() {

internal.Main()

}

To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

[Esse]

Thank you Christian, you land me in the bulls eye lines.
I've tried this but it's not working. Maybe because I load various modules. I mean, it builds, but with no gain. Checking paths, the stack, etc. Still checking.

[Strom]

There is now a bug report for this https://code.google.com/p/googleappengine/issues/detail?id=13053

Please go and star the issue so that Google can see this slowdown is affecting us all.

[Dave Day]

Thanks for bringing this up; hopefully we can find a way to improve the experience.

The issue attached only talks about the slowness of `goapp build`, but this thread seems to be about dev_appserver.py/go-app-builder. It seems to me that the second is the one that's more interesting/important – particularly since we automatically rebuild after each save. Is that right?

--

[Admin UBA]

It's the same issue. But it's not for go 1.6 slowness, it's appserver build forcing to rebuild all packages every time.

You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/k-r_hJ7W1dw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

[Dave Day]

Actually, they're entirely separate issues – which is why I'm trying to understand the exact concerns.

The first email in this thread talks about modifying dev_appserver/go-app-builder so that it can install (cache) the build products between uses so it doesn't rebuild the entire tree every time.

That is an independent issue to the slow down Go saw between 1.4 and 1.6 (and which the Go community is working hard to improve for 1.7 and beyond).

It's also different between any difference in performance between `goapp build` and `go build`: dev_appserver.py builds your application in a completely different way. The linked issue only seems to talk about goapp, and not dev_appserver.py at all.

[Admin UBA]

True, I've missed the other issue not using devappserver. We have tested before in here calling goapp build in sequence as well, and I think the issue there is calling goapp build first, then goapp build -a later. Both will have high times. Then running go build in third, it uses the installed packages. Finally go build -a have the same times. Try running -a before running plain build. Goapp build have the same times as go build. I haven't seen any issue in it.

1.6 slower. Sure.

Now here. How to make devappserver use the installed packages... This will turn 40s of build to 1s each time I hit save.

[Strom]

My #1 concern is the slowness of dev_appserver.py. It was previously possible to edit code, then switch to a browser and have the new code being served already. Since about SDK 1.9.35 I have to wait for 10+ seconds until the browser loads the page, as the SDK server is busy recompiling. The timing difference is not explained away by the slower Go 1.6 compiler. Our main running theory here has been that it recompiles every single referenced package every time.

[David Komer]

Ditto... if I could go back and make the ticket specifically about dev_appserver.py I would :\

I thought I was isolating and clarifying the issue by noticing a slow build time between goapp vs. go, but in fact that was simply user error (mixing up "go get / go install" with "goapp get / goapp install"...)

The real issue is dev_appserver.py. Along with Esse and Strom and I'm sure many others - the inability to quickly switch windows and see changes is excruciating. We can't be the only ones seeing this, and it's a real showstopper for time-sensitive projects.

Is there anything we can do to help? Since goapp isn't the bottleneck, where is the bottleneck happening?

[David Komer]

Though on the plus side, since "goapp test" works pretty well - I'm now writing all those tests I told myself I'd do later :P

[Admin UBA]

Christian have shown us some way inside go_aplication.py, but I was unable to make it work. It just kept ignoring installed packages. It may have worked in previous versions.
Is someone literate in the go-app-builder?

[Christian Ohler]

FWIW, our current patch is below.  We're on 1.9.36.  You need an extra file main_appengine.go as described in my previous email.  Our team has been using this for many months with no problems, so we know it works well, at least in our setup.  Make sure you leave HACK_DEV_APPSERVER_USE_GO_APP_BUILDER unset – setting it gives you the old behavior.

--- a/go_appengine/google/appengine/tools/devappserver2/go_application.py 2015-10-25 18:28:34.000000000 +1100

+++ b/go_appengine/google/appengine/tools/devappserver2/go_application.py 2015-10-25 16:42:59.000000000 +1100

@@ -247,10 +247,7 @@

     gab_stdout, _ = self._run_gab(gab_args, env={})

     return gab_stdout

-  def _build(self):

-    assert self._go_file_to_mtime, 'no .go files'

-    logging.debug('Building Go application')

-

+  def _build_with_gab(self):

     gab_args = [

         '-binary_name', '_go_app',

         '-extra_imports', 'appengine_internal/init',

@@ -263,6 +260,37 @@

     logging.debug('Build succeeded:\n%s\n%s', gab_stdout, gab_stderr)

     self._go_executable = os.path.join(self._work_dir, '_go_app')

+  def _build_without_gab(self):

+    executable = os.path.join(self._work_dir, '_go_app')

+    args = [os.path.join(_SDKROOT, 'goapp'), 'build', '-i', '-o', executable]

+    env = {

+      'GOROOT': GOROOT,

+      'CGO_ENABLED': '0',

+    }

+    if 'GOPATH' in os.environ:

+      env['GOPATH'] = os.environ['GOPATH']

+    process = safe_subprocess.start_process(args,

+                                            stdout=subprocess.PIPE,

+                                            stderr=subprocess.PIPE,

+                                            cwd=self._module_configuration.application_root,

+                                            env=env)

+    stdout, stderr = process.communicate()

+    if process.returncode:

+      raise go_errors.BuildError(

+        '(Executed command: %s)\n%s\n%s' % (' '.join(args),

+                                            stdout, stderr))

+    logging.debug('Build succeeded:\n%s\n%s', stdout, stderr)

+    self._go_executable = executable

+

+  def _build(self):

+    assert self._go_file_to_mtime, 'no .go files'

+    logging.debug('Building Go application')

+    ### HACK(ohler): _build_without_gab enables incremental builds.

+    if os.getenv('HACK_DEV_APPSERVER_USE_GO_APP_BUILDER') is None:

+      self._build_without_gab()

+    else:

+      self._build_with_gab()

+

   def maybe_build(self, maybe_modified_since_last_build):

     """Builds an executable for the application if necessary.

--

[Esse]

I made the hack work past compilation. It is a blast. But, as I have modules, I don't know exactly what is going wrong. The modules do compile, are bonded to the right port, but the modules init() function seems to be not called. So the application responds 404 on any route I try and also logs 404.

INFO module.py:788] moduleA: "GET /resources/id HTTP/1.1" 404 19
INFO module.py:788] moduleB: "GET /resources/id HTTP/1.1" 404 19

Have you seen something like this? 

[Christian Ohler]

We use multiple modules, too.  What .go files (names and content) do you have in the directory of the app.yaml of each of your modules?  We have just two minimal files, main_appengine.go and main_non_appengine.go:

main_appengine.go:

// The code in this file would normally be generated by

// go-app-builder, but our hacked App Engine SDK bypasses that, so we

// need this file checked in.

// +build appengine

package main

// extra_imports from go_application.py

import (

_ "appengine_internal/init"

_ "our/modules/moduleA"

)

// main from go-app-builder/synthesizer.go

import (

internal "appengine_internal"

)

func main() {

internal.Main()

}

main_non_appengine.go:

// `go test -race` expects a package named "main" to

// contain a "main" function, so we define a dummy one

// here.

// +build !appengine

package main

func main() {

panic("not meant to be called")

}

The package our/modules/moduleA then contains a single file:

app.go:

package server

import (

"net/http"

)

import (

// modules of our app that install request handlers in a global mux

_ "our/app/handlers/featureA"

_ "our/app/handlers/featureB"

_ "our/app/handlers/featureC"

)

func init() {

http.Handle("/", ...global mux...)

}

--

[Dave Day]

Thanks all for the feedback.

We've implemented a change in dev_appserver.py now so that it will reuse build products between builds, which should cut down the compile portion of the time significantly – each incremental change should only have to recompile a small handful of packages.

I'll ping this thread once a new SDK with the changes has been released (it might be a few weeks yet).

Thanks,

Dave

[David Komer]

WOOHOOOOO!!!

[szte...@gmail.com]

Christian, I was trying to rearchitect my files when the news arrived.

Thank you for all the help!

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Esse]

Can't wait anymore! Hehe.
Thank you

[Strom]

SDK 1.9.40 has come out now, and to me it still looks slow and the changelog doesn't mention anything either. Was this change forgot to put in or where did it disappear?

[Dave Day]

On 28 July 2016 at 13:29, Strom <xxs...@gmail.com> wrote:

SDK 1.9.40 has come out now, and to me it still looks slow and the changelog doesn't mention anything either. Was this change forgot to put in or where did it disappear?

The change hasn't been forgotten, it's just taking a little while to roll out.

The good news is that most of the change is actually already in the 1.9.40 SDK, so if you're happy to make a local modification it should be possible to try it out immediately. Basically, there's a new flag in the go-app-builder to enable incremental builds. So, you just need to modify go_application.py to pass this flag:

In $SDK_ROOT/google/appengine/tools/devappserver2/go_application.py, find the _get_base_gab_args func, and add an extra arg:

  gab_args = [

      _GAB_PATH,

      '-app_base', application_root,

      '-arch', arch,

      '-dynamic',

      '-goroot', GOROOT,

      '-nobuild_files', '^' + str(nobuild_files),

      '-unsafe',

      '-incremental_rebuild',  ### Add this new line

  ]

Restart dev_appserver/goapp serve to pick up the change.

Hopefully you'll soon be enjoying much quicker build iterations. Let me know how it goes.

[Admin UBA]

Two modules running:

From 21,672 ms

go-app-builder: build timing: 65×compile (10.824s total), 0×link (0 total)

go-app-builder: build timing: 65×compile (10.848s total), 0×link (0 total)

To: 380 ms

go-app-builder: build timing: 64×skip (27ms total), 1×compile (173ms total), 0×link (0 total)

go-app-builder: build timing: 64×skip (29ms total), 1×compile (151ms total), 0×link (0 total)

Is it snowing? Because I feel like Xmas morning.

Thanks! Congratulations!

[David Komer]

Thanks!

[Strom]

Unfortunately it doesn't seem to have any effect, I still get over 10 sec compile times.

I modified the go_application.py file exactly as instructed, and I have verified that the -incremental_rebuild flag does indeed get passed to go-app-builder.exe.

[Dave Day]

Can you break the build (something like a typo in a variable name) and reporting what timings you're seeing?

Unfortunately, while we can re-use the build products from the compile step, there's nothing we can really do to make the link stage quicker. The link stage can take multiple seconds to complete.

--

You received this message because you are subscribed to the Google Groups "google-appengine-go" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

[Admin UBA]

I've seen one little issue, nothing of too much concern, and don't know exactly how to reproduce (some times happens, some times it doesn't). Simple as this: hit save too fast altering something during the previous build may cause the builder to lose track of some package.a. But, not always the edited file. Also, this happens when there is lots of small packages. Just get out of the dev_appserver process and restart. Actually I've put together some small packages and it stopped heppening. As I said, just happens. Not sure of anything.

Other thing is to get out of Dropbox. Actually this is a recommendation from git to not break the repository with other developers in. But I'm alone. After disabling Dropbox the response time of the build reduced a lot as well.

On Wed, Aug 17, 2016 at 7:19 PM <dar...@darshancomputing.com> wrote:

Upgrading to 1.9.40 and adding `'-incremental_rebuild',` to `gab_args` in _get_base_gab_args in $SDK_ROOT/google/appengine/tools/devappserver2/go_application.py seems to work as intended for me.  Rebuilds after saving files are now very quick instead of very slow.  Thanks!

[Strom]

Successful build takes ~14.5 seconds, broken build takes ~9 seconds. These timings seem to be the same for both with and without the -incremental_rebuild flag.

[dar...@darshancomputing.com]

Upgrading to 1.9.40 and adding `'-incremental_rebuild',` to `gab_args` in _get_base_gab_args in $SDK_ROOT/google/appengine/tools/devappserver2/go_application.py seems to work as intended for me.  Rebuilds after saving files are now very quick instead of very slow.  Thanks!

On Wednesday, August 17, 2016 at 2:45:41 AM UTC-7, Dave Day wrote: