The Performance of the SDK development server has become unbearable

[PK]

Over the years Google App Engine has made tremendous progress on adding features and making the production environment more robust. All this is very much noted and appreciated even though they came along with the recent steep price hike.

At the same time though, the SDK server has been getting slower and slower and the pre-deployment development cycle has become really tedious for anything but the smallest projects. I want to remind to Google that history has shown that in the platform space you do not become successful unless you capture the developers mindset and you will not capture the developers unless you give a descent development environment to get their job done. For some this means, a fancy IDE, but almost for everybody this primarily means an environment that does not slow you down.

So, I highly recommend, among all the other great things in the roadmap to devote some effort to go back and look at the SDK performance and fix the low hanging fruits as soon as possible. I and others, have posted notes with examples of performance degradation from release to release---that went mostly unanswered by the GAE team---but I think that with 1.6.0 it has become really unbearable.

Thanks for reading through,
PK
http://www.gae123.com

PS My experience is with the Python SDK only, I cannot talk about the Java or GO

[Rohan Chandiramani]

+1

[Timofey Koolin]

I have big startup time dev server. 

It will be fast when I drop datastore file and use  --use_sqlite parameter.

2011/11/16 Rohan Chandiramani <maste...@gmail.com>

+1

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/xqLXe-vABTIJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Blog: www.rekby.ru

[Desmond Brand]

Most of the slowness I experience comes from allowing dev_appserver.py to serve static assets. It's possible to get a big speed boost by placing a reverse proxy in from of the dev server. Serve static files directly from the proxy, and pass through dynamic requests to the dev server as usual. I was able to drop my pageload time from 35 s to 2 s by doing this. Instructions are on my blog: http://desmondbrand.com/blog/2011/11/15/speed-up-your-app-engine-dev-server-with-an-apache-reverse-proxy/

[PK]

Rekby: Using --sql_lite does not work for me because to this day there is no parity between the --sql_lite version and the production datastore. Others and I have filed issues about it. Others have reported that sqllite helps with large datasets. It used to be that the SDK datastore would get very slow as the dataset was getting larger, but with 1.6.0 it has become slow and unpredictable/cranky even with tiny test datasets.

Desmond: Proxying static content is a useful and necessary technique for any real world app developed with the GAE SDK. I have been developing a real-world app with GAE for four years and I could have not survived without it.... I first wrote about this here  back on July 10th 2010. I do miss the days when this was my only major performance issue.

[Andrius A]

+10 totally agree. big pain and impossible to develop using backends + queues + channels, it takes all my day, night and weekends!

On 16 November 2011 22:12, PK <p...@gae123.com> wrote:

Rekby: Using --sql_lite does not work for me because to this day there is no parity between the --sql_lite version and the production datastore. Others and I have filed issues about it. Others have reported that sqllite helps with large datasets. It used to be that the SDK datastore would get very slow as the dataset was getting larger, but with 1.6.0 it has become slow and unpredictable/cranky even with tiny test datasets.

Desmond: Proxying static content is a useful and necessary technique for any real world app developed with the GAE SDK. I have been developing a real-world app with GAE for four years and I could have not survived without it.... I first wrote about this here  back on July 10th 2010. I do miss the days when this was my only major performance issue.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/AEabqH7w2M4J.

[King]

Has there been any response from Google about this yet?  Perhaps in another thread or the chatrooms? I have noticed a huge drop in performance with the python 1.6.0 dev server.

[Robert Schuppenies]

Is it possible to reproduce the issues you are seeing, for example with a code snippet or an app? If so could you file a ticket, attach the snippet/app and describe how we can reproduce it?

 

FWIW, I personally haven't experienced any performance degradation, but that only indicates I am doing something differently from you.

cheers,

robert

On Wed, Nov 16, 2011 at 2:20 PM, King <sirh...@gmail.com> wrote:

Has there been any response from Google about this yet?  Perhaps in another thread or the chatrooms? I have noticed a huge drop in performance with the python 1.6.0 dev server.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/5eQIDomPdwwJ.

[King]

Hi Robert,

I haven't been able to figure out exactly what is the issue, so that others could reproduce it. It seems like it's happening for all my requests, so that's not helping me narrow it down at the moment.  I'll try and track down which code is the slowest, and see if I can get enough information to open a ticket.

[Wilson MacGyver]

is it across the board or just gae-python? I haven't noticed any problem
with gae-java or gae-go.

--
Omnem crede diem tibi diluxisse supremum.

[PK]

Hi Robert,

I am glad to see somebody from Google showing interest in the problem. Please take a look at the benchmarks I reported in this thread on Sep 6 and just updated with results for 1.6.0

This is a test case from a test suite of a big app. I cannot cut and paste but is totally reproducible on both my mac and ubuntu systems. Same code base, I just change the GAE SDK version from 1.5.2 to a later one, run the tests and get 100%+ performance degradation!!!

Having said that, I am in the Mt View area and open to a number of options to work with Google to get to the bottom of this:

1. come in the Mt. View campus with the code
2. show you through a gotomeeting/webex session
3. run the tests with instrumentation code you provide and send you the results

Let me know....

Thanks,
PK

[PK]

But, what is even more annoying and new with 1.6.0 is that very often the datastore decides to go out for lunch in the middle of executing a query. Look for example at this one: about a minute to run a really simple query that returns one entity, in a datastore that I just created and maybe it has 200 entities of about 15 kinds. The Domain kind has two entities!!!

INFO     2011-11-17 03:39:47,371 query.py:260] SWQ: Executing 'SELECT *  FROM Domain WHERE dom_names = :1' ['ls']
INFO     2011-11-17 03:40:38,751 query.py:264] SWQ: 1

[Robert Schuppenies]

Thanks PK. Do you happen to have a appstats screenshot for things like that? 

From what I read so far, it seems the speed issues are either related to serving static content or datastore, so that's already some valuable insight.

For datastore, 'vladdrac' already did some sleuthing and put his results in http://code.google.com/p/googleappengine/issues/detail?id=6005. Do you think what you are seeing could be related?

For the slow serving of static content, is that an issue for only Windows or are people seeing this on other platforms, too? If so, could you elaborate and maybe provide a reproducible test scenario?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/nE4DRRHUdC8J.

[Matt Jibson]

I saw equivalent behavior with the SDK on Windows and OpenBSD when
serving images from the datastore. I had a page with multiple
images.get_serving_url (with and without resizing) uses, and those
increased page load times by some seconds per image in the SDK.

[Mickey]

Robert: I posted the following comment to http://code.google.com/p/googleappengine/issues/detail?id=5134, but it may be more useful here...

I didn't notice this until after 1.6.0 came out. It seems as if the following two lines of code is a lot slower than before.

The first runs fast (1/10th of a second): a query that looks for child Books of the_user entity.
The second runs much slower (4 seconds): a query that looks for Books in which the_user is in the Book's ReferenceProperty

DEBUG    2011-11-17 05:44:02,299 main.py:483] 1
some_books = Book.all().ancestor(the_user).filter('good =', True).fetch(30)
DEBUG    2011-11-17 05:44:02,336 main.py:485] 2
other_book = the_user.books.filter('good =', False).get()
DEBUG    2011-11-17 05:44:06,661 main.py:487] 3

....plus it seems like writing to the datastore takes a lot longer than before.

[PK]

Robert:

1. The static content issue I know about is an issue on all platforms and has primarily to do with the single threaded nature of the dev server. See here about what I mean: http://www.gae123.com/articles/gaet/fir-tim-exp.html

2. Issue 6005 is definitely part of the what I see, I updated the bug, thanks for pointing me to that direction

3. However, the issue of the dev server being extremely slow on certain queries that appears to be new in 1.6.0 is a separate issue. I filed issue 6355 to track this.

[PK]

Hi Robert,

I Updated issue 6355 with appstats data, let me know if this is what you want to see.

Thanks,
PK

[PK]

I want to close the loop on this thread with some positive notes. For the past few days, I have been working with a super fast dev environment thanks to the workarounds I describe below, follow the links for the kudos. In addition, to how the server feels during the development cycle, the application regression tests now run three times faster.

Here is a summary of workarounds for 1.6.0. Please do star the issues mentioned if you want to see them fixed by the Google team in a future release. Patches are at the end.

1. Comment out the SetupIndexes line in dev_appserver.py (see/star issue 6005 for the reason)
2. Turn off all dev server attempts to write logs into the datastore, this seems to be an upcoming feature that currently just slows down development (see/star issue 6355 for this)
3. Finally, this is not new and was already worked around in my environment, but if you have a realistic application you need to proxy the serving of static content (images/javascript/css) if you want to get any work done. I use apache for that and a mini framework I have built. I describe the issue more here. It might also help in the long term, if you star issues 343, 6005 and 4720.

One more piece of good news for me, sqllite started working with my application. I do not remember when was the last time I tried it and would not pass my test cases and/or hang once in a while; now it mostly works. Unfortunately, when I tried to upload a lot of data it locked pretty quickly, so it is not as great as I thought but these are steps in the right direction.

I want to acknowledge Robert S. help from Google who followed up on this thread and updated the bugs in the database.

PATCHES:
1. +++ google_appengine_1.6.0/google/appengine/tools/dev_appserver.py     
@@ -2628,7 +2629,7 @@
-        dev_appserver_index.SetupIndexes(config.application, root_path)
+        #dev_appserver_index.SetupIndexes(config.application, root_path)

2. +++ google_appengine_1.6.0/google/appengine/tools/dev_appserver.py   
@@ -1435,11 +1435,12 @@

     sys.stderr = LoggingStream()

-    logservice._global_buffer = logservice.LogsBuffer()
+    logservice.AUTOFLUSH_ENABLED = False
+    #logservice._global_buffer = logservice.LogsBuffer()

-    app_log_handler = app_logging.AppLogsHandler(
-        logservice.logs_buffer().stream())
-    logging.getLogger().addHandler(app_log_handler)
+    #app_log_handler = app_logging.AppLogsHandler(
+    #    logservice.logs_buffer().stream())
+    #logging.getLogger().addHandler(app_log_handler)

     os.environ.clear()
     os.environ.update(env)
@@ -1514,9 +1515,9 @@
     sys.stdout = old_stdout


-    logservice_stub._flush_logs_buffer()
+    #logservice_stub._flush_logs_buffer()
     sys.stderr = old_stderr
-    logging.getLogger().removeHandler(app_log_handler)
+    #logging.getLogger().removeHandler(app_log_handler)

[Calvin]

On Wednesday, November 23, 2011 4:38:48 PM UTC-8, PK wrote:

3. Finally, this is not new and was already worked around in my environment, but if you have a realistic application you need to proxy the serving of static content (images/javascript/css) if you want to get any work done. I use apache for that and a mini framework I have built. I describe the issue more here. It might also help in the long term, if you star issues 343, 6005 and 4720.

Do you have more details on how you do this?  Do you just do a redirect to the Apache server when a static asset is requested from dev_appserver?

[Tim Hoffman]

Hi PK

Not sure what you call a lot of data.  I have successfully imported 500,000 entities into the sqlite backend.

The trick I found was to specify single thread during the bulk upload.  I found with multiple threads the dev server

would ultimately lock up or the bulk upload would fall over with lots of errors.

Rgds

Tim

[Mickey]

Big thanks PK. That patch for Issue 6335 made a HUGE difference in the performance of the dev server.  I wasn't even aware that logging was identified as an issue in SDK 1.6.0 until this post.  Everything else I read seemed to point to the SetupIndexes problem, and that wasn't effecting me.

[Andrin von Rechenberg]

+1 on speeding up  dev_appserver.py

On Thu, Nov 24, 2011 at 4:49 AM, Mickey <bmcc...@gmail.com> wrote:

Big thanks PK. That patch for Issue 6335 made a HUGE difference in the performance of the dev server.  I wasn't even aware that logging was identified as an issue in SDK 1.6.0 until this post.  Everything else I read seemed to point to the SetupIndexes problem, and that wasn't effecting me.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/LRi97gavywIJ.

[Desmond Brand]

I wrote more detailed instructions for how to set this up:

http://desmondbrand.com/blog/2011/11/15/speed-up-your-app-engine-dev-server-with-an-apache-reverse-proxy/

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/M5yP6ScsCC4J.

[Ikai Lan (Google)]

Desmond, thanks for that! I think nginx is a bit lighter weight and easier to configure:

https://help.ubuntu.com/community/Nginx/ReverseProxy

nginx is a bit easier to configure than Apache, especially if you're just using it for development. Here's a sample reverse proxy config:

server {
        listen   80;
        server_name  foo.bar.no foo.bar.yes foo.bar.ok;

        access_log  /var/log/nginx/access.log;


        location / {
                proxy_pass      http://172.27.0.2/;
                include         /etc/nginx/proxy.conf;
        }
}

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[Vivek Puri]

Ikai,

Is it possible to bundle this into the sdk itself. All this is way too
much custom config to deal with. I already loose my email sending
capabilities from localhost(needed for testing purposes) every time i
upgrade to new version of sdk.

On Nov 29, 1:56 pm, "Ikai Lan (Google)" <ika...@google.com> wrote:
> Desmond, thanks for that! I think nginx is a bit lighter weight and easier
> to configure:
>
> https://help.ubuntu.com/community/Nginx/ReverseProxy
>
> nginx is a bit easier to configure than Apache, especially if you're just
> using it for development. Here's a sample reverse proxy config:
>
> server {        listen   80;        server_name  foo.bar.no
> foo.bar.yes foo.bar.ok;        access_log  /var/log/nginx/access.log;
>       location / {                proxy_pass      http://172.27.0.2/;
>               include         /etc/nginx/proxy.conf;        }}
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai
>

> On Fri, Nov 25, 2011 at 5:36 PM, Desmond Brand <d...@desmondbrand.com>wrote:
>
>
>
>
>
>
>
> > I wrote more detailed instructions for how to set this up:
>

> >http://desmondbrand.com/blog/2011/11/15/speed-up-your-app-engine-dev-...

>
> > On 23 November 2011 18:43, Calvin <calvin.r...@gmail.com> wrote:
>
> >> On Wednesday, November 23, 2011 4:38:48 PM UTC-8, PK wrote:
>
> >>> 3. Finally, this is not new and was already worked around in my
> >>> environment, but if you have a realistic application you need to proxy the
> >>> serving of static content (images/javascript/css) if you want to get any
> >>> work done. I use apache for that and a mini framework I have built. I

> >>> describe the issue more here<http://www.gae123.com/articles/gaet/fir-tim-exp.html>.
> >>> It might also help in the long term, if you star issues 343<http://code.google.com/p/googleappengine/issues/detail?id=343>,

[Ikai Lan (Google)]

Vivek,

I get you. There are definitely parts of the SDK and dev appserver that we can improve, and this is one of them. 

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[PK]

Hi Ikai,

Thanks for acknowledging this.

Good software gets better but great software also does nor regress first. As painful as is to configure static resource serving, it is by now a known evil and I have automated it. Hunting down regressions is unexpected and totally unproductive. Getting back to where this thread started I would like to hear that you will also/first add some benchmarks in your internal SDK testing process so that huge performance regressions like the ones described here will be less likely to happen again.

Thanks for your attention,

PK

[Robert Schuppenies]

On Wed, Nov 30, 2011 at 11:33 AM, PK <pa...@gae123.com> wrote:

Hi Ikai,

Thanks for acknowledging this.

Good software gets better but great software also does nor regress first. As painful as is to configure static resource serving, it is by now a known evil and I have automated it. Hunting down regressions is unexpected and totally unproductive. Getting back to where this thread started I would like to hear that you will also/first add some benchmarks in your internal SDK testing process so that huge performance regressions like the ones described here will be less likely to happen again.

You're right PK and we will look into this.

cheers,

robert

[Vivek Puri]

I got the proxy working on my end and it has definitely helped. Local
pages are loading in around 20 seconds, which were earlier taking 60+
seconds.

[PK]

Very happy to report that "1.6.1 prerelease" has addressed the performance regressions reported here.  Many thanks to everybody who starred the issues and to Robert and the rest of the Google team who followed up.

PK

http://www.gae123.com

[Andrius A]

I would like to add to that as well: developing backends in local environment is almost impossible because it is really slow running and there is no documentation how to do that using SDK?

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/zbXjoh77m9YJ.