Weekend project: keep your app warm

[Max]

Hi all, 

I just spent several hours developed an application to periodically ping your application so that your app(s) will be kept warm. The url is http://www.keepyourappwarm.com

You will need to login with your Google account to manage your warmers. The mechanism is simply using HTTP Head method

Several things on the desk to implement:

1, add site verification to make sure the site keep warming is owned by login user

2, add email alert when url is not available

3, keep track of RTT / downtime for better monitoring performance

Let me know if you have some other ideas.

It's cheaper compared with AlwaysOn plan and (I believe) it's more stable compared with Warm Up Request solution

So far I haven't found any app engine doc restricts me from doing this. Please please correct me if I am wrong.

[Max]

homepage is empty at the moment, you can use 

http://www.keepyourappwarm.com/manage

to access management console directly

I will add some explanation / comparison later on

[Jeff Schnitzer]

Lame.

Whether or not it is against the letter of the law, it's certainly
against the spirit of appengine. If your little "hack" becomes
popular, the likely consequence is that Google will have to devote
resources to engineering (or gaming) around it. I do not approve.

If you need always-on behavior, don't be a cheapskate. Pony up the
$9/mo, it's cheap hosting. If you want free, live with a little cold
startup time.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/tUkzLvDZjJcJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

[bejayoharen]

While I agree with jeff to a large extent -- you should pay for
hosting rather than gaming it -- I could still see this being useful.
Instead of using it to "warm up" your app, it would be nice to have a
third party performance monitor that could, for example, monitor your
app's uptime, latency, etc, and notify you when there is trouble (just
because there is an SLA doesn't mean it's always up).

I followed the link and don't see how to actually use the app (no
download or signup). A great project would be one that worked from a
third party site (non-cloud hosting would suffice), checked your site
periodically, and contacted you (eg sent you a text message using
twillio) if there was any trouble.

Bjorn

[Barry Hunter]

On Sun, Jul 10, 2011 at 5:17 PM, bejayoharen wrote:
> ....

> A great project would be one that worked from a
> third party site (non-cloud hosting would suffice), checked your site
> periodically, and contacted you (eg sent you a text message using
> twillio) if there was any trouble.

There are only about a billion of them already
http://www.google.com/search?q=monitoring+website

[Jeff Schnitzer]

I'd really love something specific to appengine that watches my logs
and emails/txts me when something untoward happens. Errors, avg
latency going up, task queues backing up, traffic dips, traffic
spikes, etc. Of course, it would have to run outside of appengine!

Jeff

[Max]

Hi Bejayoharen, 

Great thanks for the feedback.

As I said this is just several hours work so sorry for the UI. I agree it's not quite comprehensive. 

You can login with your Google Account by clicking top-right link "Log in with your Google Account to manage your warmers"

You will be able to create / delete / browse your warmers after login. (just click Overview | Manage | Create to switch)

[Max]

hi Jeff, 

AFAIK, lots of GAE developers are using cron job to *warm up*, including me. You can search in this group, many ppl are doing this. 

Also, if we want something more than a periodic ping, like those use cases you mentioned, then we probably don't want all these logic stay with in our business logic. This is the real benefit of this project and now is just the first step. But since such monitoring tools will enable "warm up" feature as a side effect, I will need to know this groups' idea before going further. 

Again, I don't intend to go against always on instance solution. I am just sharing my work so that other developers don't have to setup cron jobs every time. 

[Jeff Schnitzer]

Using cron to keep your app warm is not sanctioned either.  If Google wanted to give you a way to keep your app running, they'd offer it as a feature and charge for it.

If these hacks become commonplace, GAE engineers will be retasked to fighting them and this will further delay new features that I care about like next-generation queries.  Furthermore, the collateral damage of closing these loopholes might impact all our apps (such as, hypothetically, a surcharge for crons more frequent than 30 mins).

Furthermore, if you hope to build a real business on top of appengine, you're getting off on the wrong foot by alienating the folks at Google.  Don't expect to get your questions answered in this forum, and don't be surprised if your app gets blocked in some way.

Read this:  http://en.wikipedia.org/wiki/Tragedy_of_the_commons

Jeff

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ht3qDc6Z984J.

[Stephen Johnson]

Hi Jeff,

I believe that with the new pricing that may have already happened. If you keep your app "warm" with no one using it you're going to be paying for it. A lot of users will probably want their instances to shutdown faster rather than stay up and be either billed or use up their free quota.

Stephen

[Max]

Agree with your point on tragedy of the commons. The thing is this is not a new concept. It's already a common practice among many appengine developers.  Without my application, one can still write a dummy cron in 5 mins. Given that they (unfortunately including me at the moment) will do it anyway, my app is saving 5 min dev time and 1 cron job execution per minute for each single url. 

Of course, this is definitely a suboptimal solution. But it should be Google rather than us to do something on this. I don't think Google has to build some intelligence to detect this behavior, but at least they should give use some guideline in forum or GAE document. For example, multiple deployment of same app is explicitly documented here(point 4.4), but I can't find any term applicable to my case. Instead I saw this same topic been discussed here for many times but no Googler ever give a negative comment. I will disable my app if I get guideline officially or un-officially.

From other side, I don't think it's fair for a site gain a better performance just because more users are using it. From my point of view, such regenerative feedback is even worse than your tragedy of the commons.

[jeffrey_t_b]

It must not be too bad from Google's perspective, as they include an
example of a "recache" job in the GAE java cron documentation:

http://code.google.com/appengine/docs/java/config/cron.html

That said, I'm not sure why anyone would necessarily want to do this,
given the new pricing.

On Jul 11, 3:08 pm, Jeff Schnitzer <j...@infohazard.org> wrote:
> Using cron to keep your app warm is not sanctioned either.  If Google wanted
> to give you a way to keep your app running, they'd offer it as a feature and
> charge for it.
>
> If these hacks become commonplace, GAE engineers will be retasked to
> fighting them and this will further delay new features that I care about
> like next-generation queries.  Furthermore, the collateral damage of closing
> these loopholes might impact all our apps (such as, hypothetically, a
> surcharge for crons more frequent than 30 mins).
>
> Furthermore, if you hope to build a real business on top of appengine,
> you're getting off on the wrong foot by alienating the folks at Google.
>  Don't expect to get your questions answered in this forum, and don't be
> surprised if your app gets blocked in some way.
>
> Read this:  http://en.wikipedia.org/wiki/Tragedy_of_the_commons
>
> Jeff
>
>
>
>
>
>
>
> On Sun, Jul 10, 2011 at 10:44 PM, Max <thebb...@gmail.com> wrote:
> > hi Jeff,
>
> > AFAIK, lots of GAE developers are using cron job to *warm up*, including
> > me. You can search in this group, many ppl are doing this.
>
> > Also, if we want something more than a periodic ping, like those use cases
> > you mentioned, then we probably don't want all these logic stay with in our
> > business logic. This is the real benefit of this project and now is just the
> > first step. But since such monitoring tools will enable "warm up" feature as
> > a side effect, I will need to know this groups' idea before going further.
>

> > Again, I don't intend to go against *always on instance* solution. I am

[Phil Young]

Isn't the end of point 4.4 applicable here...

"or otherwise access the Service in a manner intended to avoid incurring fees"

...as you are avoiding paying the $9/month for an always running instance?

Phil

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/xFonPhtvbLEJ.

[Max]

my understanding is, you can't hack a service for free if it's supposed to be paid. By using my application, you still use / pay for your normal instance rather than always on instance. so this term is not applicable to this case.

[Jeff Schnitzer]

I can't believe I'm still writing about this... at the very least
you're hacking around the $9/mo fee for an always-on instance.

The "free tier" of appengine works because all those zillions of
little test apps and experiments that people create don't actually
occupy resources beyond a small amount of disk space. If you create a
magic button that keeps even the unused apps resident, the cluster
will fill up with terabytes of idle ram. If you want to skeeve the
system to keep your own app resident, keep quiet about it and hope
there aren't enough other people doing this to force Google to close
the loophole.

BTW, I don't see why this calculus would change at all with the new
pricing scheme. You could still have a lot of zombie apps eating up
their free 24 instance-hours.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/JaGmQlWQKj8J.

[Barry Hunter]

On Tue, Jul 12, 2011 at 1:26 AM, jeffrey_t_b <jbo...@gmail.com> wrote:
> It must not be too bad from Google's perspective, as they include an
> example of a "recache" job in the GAE java cron documentation:

If you are recaching something every 2 minutes, there is a certain
expectation that you going to be needing it uptodate all the time. So
if your app gets traffic all day long, thats a good practice.

But if your app is not that popular its just bad design.

... in short its just an example of what you can do. Its not shown as
a 'best practice' in every possible situation. Just because you can,
doesn't mean you should.

>
> http://code.google.com/appengine/docs/java/config/cron.html
>
> That said, I'm not sure why anyone would necessarily want to do this,
> given the new pricing.

Not sure how the New Pricing comes into this.

Keeping an app 'warm' just means keeping an instance always loaded.
And you get 24 hours of instance time free. So all its doing is making
sure you using all your free quota, not costing you anything as such.

>
>
> On Jul 11, 3:08 pm, Jeff Schnitzer <j...@infohazard.org> wrote:
>> Using cron to keep your app warm is not sanctioned either.  If Google wanted
>> to give you a way to keep your app running, they'd offer it as a feature and
>> charge for it.
>>
>> If these hacks become commonplace, GAE engineers will be retasked to
>> fighting them and this will further delay new features that I care about
>> like next-generation queries.  Furthermore, the collateral damage of closing
>> these loopholes might impact all our apps (such as, hypothetically, a
>> surcharge for crons more frequent than 30 mins).
>>
>> Furthermore, if you hope to build a real business on top of appengine,
>> you're getting off on the wrong foot by alienating the folks at Google.
>>  Don't expect to get your questions answered in this forum, and don't be
>> surprised if your app gets blocked in some way.
>>
>> Read this:  http://en.wikipedia.org/wiki/Tragedy_of_the_commons

Exactly. Google could be spending their time lowering the instance
startup time. Rather than dealing with the resource consumption of
lots of people keeping their app 'warm' just in case.

Keeping an app warm is a 'workaround'. And workarounds nearly always
distract the developers from fixing the actual issue.