GAE cold start time still too long

[Anders]

This has been discussed before but the problem still remains. It seems
that GAE is no longer in a preview release version (as far as I can
see). Having a cold start initiation time of 10 seconds is a major
bottleneck.

Imagine if it took 10 seconds to load for example the Google Search
index page in your browser. It doesn't sound like a very long time,
but today that kind of load time for an index page is very poor
performance.

I understand that GAE cannot at the moment hold all applications hot/
warm, because that would require a lot more resources I assume. But I
think the cold start time needs to be brought down to a maximum of
around 2 seconds.

It actually doesn't matter in many cases if an application is used by
millions of users every day or only seldom by a few people. The load
time for webpages is usually extremely important regardless the amount
of traffic to a website. Each user's experience counts.

[Brandon Thomson]

Are you using python or java? what is your framework?

[Anders]

I'm using Python. And only Python's standard library that's in GAE. My
guess is that the cold start problem is similar in the Java version.

The cold start time has improved but now and then loading the index
page takes frustratingly long time. That's poor quality for both end
users and developers.

[Eli Jones]

First, when I log into appengine.google.com, it still says "this is a preview release" with the "preview release" part being in bright red letters... so.. take that to mean what you want.

Second, you haven't mentioned what exactly the code for your index page does? What is it loading?  How much caching are you doing?

Caching isn't just for entities from the datastore.. you can and should also cache html or page templates or whatever else you can..  Also, you shouldn't dump a bunch of imports at the top of your code.. only import specific modules as needed within the code.

Though, it's hard to make suggestions without knowing exactly how your code works. (You may be doing all of the above things.. or feel like you are doing them.)

A lot of the annoying restrictions that people complain about, to me, are inherent limitations to having a highly scalable infrastructure.. the restrictions are there to force you to learn to code (from the beginning) for the App Engine environment.

Granted, your issue may just be with intermittent but slow cold start times.. is the cold start reasonable in general for you.. but occasionally hits that frustrating point? (Again, hard to know anything without knowing how your code is structured.)

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Anders]

And, I forgot to mention, the index page only uses the datastore and
memcache in secondary Ajax calls, except for users who are signed in.
For html pages I use the Django version that's included in GAE.

[Anders]

Strange, I don't see the preview release notice on
appengine.google.com any longer. So I assumed that GAE was no longer
in preview release version. And on the SDK download page it now says:
"Please note: The App Engine SDK is under active development, please
keep this in mind as you explore its capabilities."

I guess I could cache the html generated by Django, at least for the
index page, if that library is more heavy to load than the Memcache
API. I don't know exactly how Python works, but shouldn't it be
possible for GAE to always have the standard frameworks always loaded
into memory for all applications to share?

On Feb 21, 3:17 am, Eli Jones <eli.jo...@gmail.com> wrote:
> First, when I log into appengine.google.com, it still says "this is a
> preview release" with the "preview release" part being in bright red
> letters... so.. take that to mean what you want.
>
> Second, you haven't mentioned what exactly the code for your index page
> does? What is it loading?  How much caching are you doing?
>
> Caching isn't just for entities from the datastore.. you can and should also
> cache html or page templates or whatever else you can..  Also, you shouldn't
> dump a bunch of imports at the top of your code.. only import specific
> modules as needed within the code.
>
> Though, it's hard to make suggestions without knowing exactly how your code
> works. (You may be doing all of the above things.. or feel like you are
> doing them.)
>
> A lot of the annoying restrictions that people complain about, to me, are
> inherent limitations to having a highly scalable infrastructure.. the
> restrictions are there to force you to learn to code (from the beginning)
> for the App Engine environment.
>
> Granted, your issue may just be with intermittent but slow cold start
> times.. is the cold start reasonable in general for you..
> but occasionally hits that frustrating point? (Again, hard to know anything
> without knowing how your code is structured.)
>

> > google-appengi...@googlegroups.com<google-appengine%2Bunsu...@googlegroups.com>

[Eli Jones]

I have a testapp set up that I use to see if any goofy ideas I come up with have any merit.

So you could try to create a simplified version of your page that used the same imports and django templates as the live one.  Then create two different test pages.. one where you experiment with caching different things.. and then try to compare the cold start times.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Anders]

I doubt that replacing calls to Django with calls to memcache would
improve the cold start time significantly, although I could be wrong.
What I guess happens in a cold start is that the application code has
to be loaded through the network into a server and then instantiated
before serving a webpage. Using the memcache would then not improve
the initial loading of applications. Only if Django is taking a long
time to load during a cold start would the use of memcache make much
difference.

On Feb 21, 3:37 am, Eli Jones <eli.jo...@gmail.com> wrote:
> I have a testapp set up that I use to see if any goofy ideas I come up with
> have any merit.
>
> So you could try to create a simplified version of your page that used the
> same imports and django templates as the live one.  Then create two
> different test pages.. one where you experiment with caching different
> things.. and then try to compare the cold start times.
>

> > <google-appengine%2Bunsu...@googlegroups.com<google-appengine%252Buns...@googlegroups.com>

[Eli Jones]

Hmm.. only way to ever be sure is to test. I don't use Django so I am completely useless for suggestions.

It seems that 500 ms or less should be your expected cold start time for python.

If you are experiencing cold starts higher than that, then something in your code could be taking up that time.. so there might be ways to improve the cold start time.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Anders]

When my application is loaded into a server the performance is
excellent. Loading a webpage into the browser then only takes a
fraction of a second, even with the use of Django templates and calls
to the datastore. I only use the memcache for one Ajax call that is
made frequently, and that has improved that call a lot. Using the
memcache to replace calls to Django could probably improve the
performance a bit in my case, but I doubt that it would have any
significant impact on the cold start time.

Here is a blog post with a solution using static files:

http://blog.vikispot.com/2010/02/dynamic-homepage-with-static-html-and.html

Using entirely static html files could perhaps work. But the
generation of dynamic content would still take a long time in a cold
start. Anyway, I think the cold start time for GAE should be improved
rather than having thousands of developers having to come up with
messy workarounds.

On Feb 21, 4:15 am, Eli Jones <eli.jo...@gmail.com> wrote:
> Hmm.. only way to ever be sure is to test. I don't use Django so I am
> completely useless for suggestions.
>
> It seems that 500 ms or less should be your expected cold start time for
> python.
>
> If you are experiencing cold starts higher than that, then something in your
> code could be taking up that time.. so there might be ways to improve the
> cold start time.
>

> > > > <google-appengine%2Bunsu...@googlegroups.com<google-appengine%252Buns...@googlegroups.com>
> > <google-appengine%252Buns...@googlegroups.com<google-appengine%25252Bun...@googlegroups.com>

[Tim Hoffman]

HI

If you don't need to start the django stack to serve a page from cache
then you should be able to deliver a page from a warm instance in
20-40ms
and if you need to start an instance and only import memcache and
retrieve the cached page you are looking at about 200ms.

I am doing this on a number of sites (don't use django but use bfg,
but a complete cold start with nothing in cache
is typically around 8-10 seconds. And even the main page is made up
of many cachable bits) so its really rare
we ever have to deal with a full stack startup with nothing in cache.

T

[Anders]

Yes, it's the complete cold start I mean. Rarely nowadays the cold
start time is as much as 10 seconds for me, but occasionally it's
several seconds too long. Maybe it doesn't happen so frequently as to
be a real problem, but it's still frustrating when suddenly the index
page loads like some overbloated dot com site from the late 90s.

[Takashi Matsuo]

Please make sure that your app doesn't load all the views on the
memory when your index page is accessed.
Loading all the views in your memory at startup will cause very long
cold startup time.

Perhaps you can use profiling for detecting what is the bottleneck.

AFAIC, 10 secs cold-start is too long for any appengine/python application.

--
Takashi Matsuo
Kay's daddy

> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Devel63]

I have complained about this many times in the past, so it's only fair
that I should write to congratulate Google. For my app, anyway, start
times have improved amazingly over the past couple of weeks. I still
wish there were no bootup times, but it's actually acceptable now. A
big improvement! I wonder if Google is now pre-compiling the code?

At the same time, I see that memcache is now expiring MUCH more
quickly. That's too bad, but for me, for now, that's an acceptable
tradeoff, if indeed it is a tradeoff.

On Feb 21, 6:57 am, Takashi Matsuo <matsuo.taka...@gmail.com> wrote:
> Please make sure that your app doesn't load all the views on the
> memory when your index page is accessed.
> Loading all the views in your memory at startup will cause very long
> cold startup time.
>
> Perhaps you can use profiling for detecting what is the bottleneck.
>
> AFAIC, 10 secs cold-start is too long for any appengine/python application.
>
> --
> Takashi Matsuo
> Kay's daddy
>

[Anders]

Hmm... Good point. The file handling my index page actually has a lot
of imports to other pages. Like a main entry script for several pages.
If Python loads all the imports, even when they are not used, then
unused imports create an unnecessary overhead. I will move those
imports to a separate file.

On Feb 21, 3:57 pm, Takashi Matsuo <matsuo.taka...@gmail.com> wrote:
> Please make sure that your app doesn't load all the views on the
> memory when your index page is accessed.
> Loading all the views in your memory at startup will cause very long
> cold startup time.
>
> Perhaps you can use profiling for detecting what is the bottleneck.
>
> AFAIC, 10 secs cold-start is too long for any appengine/python application.
>
> --
> Takashi Matsuo
> Kay's daddy
>

[Anders]

I'm no expert on cloud computing, but I find it strange that it should
take several seconds to instantiate an application. Is the Java
version of GAE faster than the Python version when it comes to cold
start? In Java you basically just need to have the classloader load
the bytecode files for the application. Doesn't Python work in a
similar way?

To load an application over the network into the memory of a server
cannot take much time, unless the application is monstrously huge. So
there must be some other kind of bottleneck involved. Maybe to hook up
an application to the datastore, memcache etc is what takes a lot of
time during a cold start?

[GAEfan]

Yes, I have been getting "Deadline Exceeded" errors again, starting
about Friday. I thought this was fixed, as I hadn't seen them for a
while. But they are back again. :-(

AFAIK, it is the cold-start loading of Django that is the cause of the
delay.

[Blake]

Same here - my crontabs are failing, starting in the past week.

[Anders]

But are not deadline exceeded errors something that happens only after
an application has been instantiated, i.e. the execution of a request
taking too long time, not including the startup time for the whole
application?

And that Django can cause several seconds of extra delay doesn't sound
plausible.

No, I think this is something in the GAE infrastructure itself,
independent of Java or Python etc. Google should look into it, lest
IBM will beat them with a WebSphereCloud or something like that, or
Sun Microsystem with a JavaCloud, or Sony with a PlayStationCloud, or,
God forbid, Microsoft with a DOSCloud. ;-)

[Chris Prinos]

I see the same thing (DeadlineExceeded) over the past couple of days
with a cron job that writes a few bytes of date to a single table. It
had been working fine for weeks, no code changes. Over the past 10
hours, 6% of these cron requests have failed because of the ap engine
errors.

Chris

[Anders]

Ok, maybe the cron calls include the cold start time in the deadline
calculation. But that's just a guess. I recently implemented a cron
job, and it works fine, although I haven't checked all the calls since
they are not critical for my application. In the dashboard only the
latest cron call is displayed it seems.

[Wooble]

On Feb 22, 12:42 pm, Anders <i...@blabline.com> wrote:

> And that Django can cause several seconds of extra delay doesn't sound
> plausible.

In the early days of App Engine, there were complaints that django
couldn't be used because of the 1000 file limit. This is pretty much
the definition of a bloated framework.

[Anders]

Yikes, more than 1000 files sounds pretty big. At least for a template
engine. Maybe Django is capable of a lot of stuff. I only use a few
very simple template statements. It would be fairly easy for me to
replace Django with some manual much more lightweight code, but on the
other hand, the Django performance is fast when my application is
loaded, and I still doubt that Django has much to do with the long
delays during cold starts. The same with JSP. And why would Django, or
code for rendering JSP, need to be loaded every time for each
application? Let's say that one server has 100 applications loaded
using Django. Does this mean that the same Django (large) code is
duplicated 100 times in the server?!

[Robert Kluin]

Hey Andres,
If you are only using Django for templates, you might consider
looking into other template systems. We have been using Mako and are
very happy with it.

And I _think_ the answer to your question is that the framework is
loaded 100 times. That is just a guess based on my understanding from
other posts.

Robert

[Anders]

But is Mako included in GAE? I use the Django version that's included
in GAE.

It would be fun, although likely unnecessary to write my own template
code. I would then only use one template expression, such as
[[ name ]] and then do a simple, lightning-fast string replacement of
these expressions in the template. No if statements, no for loops or
anything fancy like that. :-)

On Feb 23, 5:15 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> Hey Andres,
>    If you are only using Django for templates, you might consider
> looking into other template systems.  We have been using Mako and are
> very happy with it.
>
>   And I _think_ the answer to your question is that the framework is
> loaded 100 times.  That is just a guess based on my understanding from
> other posts.
>
> Robert
>

[Robert Kluin]

Mako is not included, but it is easy to include yourself. Just grab a
copy (http://www.makotemplates.org/), then include the mako directory,
from under lib, inside in the root of your app.

If you are just needing some simplistic substitutions you could always
look at Python string templates:
http://docs.python.org/library/string.html#template-strings. I would
probably not do anything very involved with them, but for returning a
simply status line or something similar they might be work.

Robert

[Anders]

Python template strings look interesting. That's the kind of simple
string replacement I was thinking about. Could be worth testing and it
seems to be included in version 2.5. I am new to Python and have
mostly programmed in Java for many years, but when I started using GAE
only Python was available. Python is actually quite nice for web
applications. If I develop a new GAE application maybe I will choose
Python again instead of Java.

I don't think it would make much difference to the cold start time,
but could improve the general performance a little bit.

On Feb 23, 6:23 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> Mako is not included, but it is easy to include yourself.  Just grab a
> copy (http://www.makotemplates.org/), then include the mako directory,
> from under lib, inside in the root of your app.
>
> If you are just needing some simplistic substitutions you could always
> look at Python string templates:http://docs.python.org/library/string.html#template-strings.   I would
> probably not do anything very involved with them, but for returning a
> simply status line or something similar they might be work.
>
> Robert
>

[Anders]

I have now replaced Django with Python string templates and removed
the Django import from the index page and in one other page. It made a
noticeable difference in webpage load speed in the SDK. Cool. Maybe it
will also improve the cold start time a bit but I haven't checked that
yet.

On Feb 23, 6:23 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> Mako is not included, but it is easy to include yourself.  Just grab a
> copy (http://www.makotemplates.org/), then include the mako directory,
> from under lib, inside in the root of your app.
>
> If you are just needing some simplistic substitutions you could always
> look at Python string templates:http://docs.python.org/library/string.html#template-strings.   I would
> probably not do anything very involved with them, but for returning a
> simply status line or something similar they might be work.
>
> Robert
>

[Max]

Hi all, 

you may want to take a look at this site

http://www.keepyourappwarm.com/

this will periodically ping your site to prevent cold start

[JH]

Wow, this is a little funny as Google has stated they do not approve
of this method several times in the past. However, this
keepyourappwarm app actually runs on GAE.

[Max]

Hi JH, 

Yes it's actually running on GAE

Do you have some link on Google has stated they do not approve of this method several times

Would like to take a look.

Thanks,