Looking for ideas on eliminating instance use

[Joshua Smith]

Here's my situation:

- I'm running an app as a public service, and I'd rather not have to charge for it. Our google overlords have not given me any way to duck the new pricing for my do-gooder app, so I'm looking to game the system if I can. :)

- My app is used infrequently by humans, in short bursts. Normal use would probably stay under the new free quotas, because the instance-alive-time x number-of-bursts-per-day < 24 hours.

- There are kiosks which show information from my app. The kiosk gets a dynamic HTML page in one go, the page includes many DIVs to display to the user over the course of 4-5 minutes or so. When it's gotten through all the DIVs, it refreshes from the server, to pick up any new updates.

In the old pricing model, this wasn't a problem. But in the new model, the kiosk "heartbeats" are keeping an instance alive. That burns up my free quota.

When a human uses the app to browse or update data, I have to pay for it 100% of the time.

So I'm wondering if anyone here has an idea of a different way to deal with the Kiosks, to allow my instances to go away completely between human interactions.

One idea I had was that when a user interaction happened, I could compute the Kiosk page and push it someplace. I can't push it to the Kiosk itself, of course, so I'd have to push it someplace that the Kiosk could hit it without spinning up a new instance in GAE. GAE has an edge cache, but I don't think there's any way for me to prime that. I could push to a different GAE app that just handles the Kiosks, but that sounds like willfully creating multiple apps to avoid quota charges, which is verboten.

I could push it to Amazon S3. That's not awful, and I've certainly done hybrid GAE/AWS apps before. But a two cloud solution is certainly not my first choice on the complexity front.

A half-assed solution would be just to have the Kiosk go through the DIVs several times before refreshing. That's not a horrible solution, but I'm hoping some clever developer here has a better idea…

Thoughts?

-Joshua

[Barry Hunter]

Dont you get one instance (well two until multi-threading) free? - so
you can have a dynamic instance always loaded. (ie you get 24 instance
hours free a day)

So providing what you want can be served by this instance, you dont
have any (instance) changes.

As for saving the file, you could put it in the blobstore. The handler
to serves the send_blob should be very quick and not tie up the
instance for long.

Or Google Storage for developers (G's answer to S3) - that avoids the
instance totally.

You might have to optimize the code around user-interaction requests,
to make sure again they stay on the single instance. But set a high
minimum latency, and that shouldnt be an issue

(or i've missed something)

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

[Raymond C.]

Be noted that Google Storage is in PREVIEW.  Did GAE teach us sth?

[Joshua Smith]

On Sep 2, 2011, at 11:10 AM, Barry Hunter wrote:

Dont you get one instance (well two until multi-threading) free? - so
you can have a dynamic instance always loaded. (ie you get 24 instance
hours free a day)

You don't get 2 until multi-threading.  You get one.  They are just reducing the price of the second one.

And under the new system any $ > 0 = $108 / year

Hey google overlords: How about instead of charging half for the pre-2.7 period, you bump the free quota to 48 hours until 2.7 comes out?  I bet that will eliminate a lot of the kvetching here from the used-to-be-free crowd.  (And charging half until November didn't seem to ameliorate the angst of the $Ks/mo crowd anyway.)

So providing what you want can be served by this instance, you dont
have any (instance) changes.

If two users happen to hit the site at around the same time, I'm pretty much guaranteed to get > 1 instance.  With one running 24/7, there is no room to handle this bump.

(Plus, there are the scheduler bugs everyone is reporting, which drive up the instance count even more.)

As for saving the file, you could put it in the blobstore. The handler
to serves the send_blob should be very quick and not tie up the
instance for long.

How long doesn't matter.  I need the instance to be untouched by the kiosks, to avoid the 24/7 life thing.

Or Google Storage for developers (G's answer to S3) - that avoids the
instance totally.

I'll have to look into that.  I have a lot more experience with S3, but if there is a nice GAE/GS bridge, perhaps that would work for me...

You might have to optimize the code around user-interaction requests,
to make sure again they stay on the single instance. But set a high
minimum latency, and that shouldnt be an issue


(or i've missed something)

I don't think it's really possible to get GAE to only serve with one instance.  It's just not in the nature of the scheduler algorithm.

And in my case, I really don't need it to.  Here are the stats for the most recent month (only counting humans):

That should easily fit under the quotas, even if it has to spin up a bunch of instances for each human.

But only if I get the Kiosks to stop burning all my quota.

I'll look into GS for Developers…

-Joshua

[Bay]

Google does not honor the settings on latency tolerance. There is no way to force only a single instance to be spawned.

[Joshua Smith]

Zing!

On Sep 2, 2011, at 11:31 AM, Raymond C. wrote:

Be noted that Google Storage is in PREVIEW.  Did GAE teach us sth?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/CqsIGSJRnh0J.

[Barry Hunter]

Hmm, from what I understand it would be possible to keep one instance. Thats all Free apps get anyway. 

I've setup against an app not currently serving other traffic two cronjobs, on a server here. every 5 minutes, hits a URL (thats your kiosk) - and then once an hour runs " ab -n 30 -c 2 -t 15 http://....appspot.com/..." to similuate a user.

Have also set minumum maximum idle instances (1) and max pending latency. So far it has only one instance. Even when requests reached 4 rps. 

Another idea, you get 9 hours of backend free a day too. Could you shunt the 'users' to actully be done via your backend. Leaving the single frontend instance to serve kiosk traffic?

[Joshua Smith]

On Sep 2, 2011, at 11:46 AM, Barry Hunter wrote:

Hmm, from what I understand it would be possible to keep one instance. Thats all Free apps get anyway. 

I've setup against an app not currently serving other traffic two cronjobs, on a server here. every 5 minutes, hits a URL (thats your kiosk) - and then once an hour runs " ab -n 30 -c 2 -t 15 http://....appspot.com/..." to similuate a user.

Have also set minumum maximum idle instances (1) and max pending latency. So far it has only one instance. Even when requests reached 4 rps. 

I set my real app to min idle = 1, max latency is 1sec yesterday.  Here's the result:

The scheduler REALLY doesn't want to run only 1 instance.  As you can see, it really wants to run 2.  And I really don't want the users to have a totally sucky user experience, either.  I'd like them to get enough instances to serve them well for the few minutes a day they are using the system.

Another idea, you get 9 hours of backend free a day too. Could you shunt the 'users' to actully be done via your backend. Leaving the single frontend instance to serve kiosk traffic?

If I can, then I clearly don't understand what a backend is.

Can you elaborate on this idea?

-Joshua

[Barry Hunter]

Another idea, you get 9 hours of backend free a day too. Could you shunt the 'users' to actully be done via your backend. Leaving the single frontend instance to serve kiosk traffic?

If I can, then I clearly don't understand what a backend is.

Can you elaborate on this idea?

Its another type of instance :)

http://code.google.com/appengine/docs/python/backends/overview.html

". Dynamic backends come into existence when they receive a request, and are turned down when idle; they are ideal for work that is intermittent or driven by user activity."

"Backends do not automatically scale in response to request volume. Instead, you specify the number of instances of each backend"

http://code.google.com/appengine/docs/python/config/backends.html

"An integer between 1 and 20 indicating the number of instances to assign to the given backend. Defaults to 1 if unspecified."

Can make a backend public

http://code.google.com/appengine/docs/python/backends/overview.html#Public_and_Private_Backends

Can queue requests

http://code.google.com/appengine/docs/python/config/backends.html#Request_Handling

... so based on all that information. the frontend instance, could upon getting a visitor, spin up a backend (unless there already is one) and redirect the user to it. 

A backend - will *strictly* adhere to the 1 instance limit. The dynamic backend will then be torn down when not used for a few minutes. 

... so providing you get under 9 hours of user interaction in total on the site, it could use a free backend. (But milling is by the hour, so cant be lots of split in lots of bits over the day :( )

(still requires that your kiosk requests and the occasiaonl redirect, can exist on one instance. )

 

[Barry Hunter]

There is another mitigating issue, that hopefully should be making
this all academic anyway.

The $50 credit. That should cover costs for about 5 months on a *low
taffic* website.

Which in theory should be enough for Multi-threading and the scheduler
to be fixed* . At which case app can return to free quotas :)

* So that it will stick to one instance, even for a reasonable amount
of traffic. I'm sure this is possible and feasible.

(I'm tempted to signup for the multi-threading trail, just to see what
sort of QPS a multi-threaded instance can handle. )

[Joshua Smith]

Good point. Although in my case, I'm expecting to burn a lot of that $50 migrating to HR once the new tool gets blessed.

I dug into the Google Storage and S3 options, and they both suck as solutions because the free tiers go away after year 1. (Plus the authorization stuff is mind-numbingly complex.)

I think my plan right now is to sit tight. The architecture I have looks like it should work when they fix the scheduler, and allow multi-threading in Python. Switching to HR is going to be a huge PITA, because my use case is exactly the one that "eventual consistency" screws up (user posts a meeting, then expects to see it in the list of meetings; silly users). But there's a gun to my head, so I guess I'll spend a weekend on it. Sorry kids.

-Joshua

[Joshua Smith]

On Sep 2, 2011, at 12:59 PM, Barry Hunter wrote:

On Fri, Sep 2, 2011 at 5:46 PM, Joshua Smith  wrote:

 Switching to HR is going to be a huge PITA, because my use case is exactly the one that "eventual consistency" screws up (user posts a meeting, then expects to see it in the list of meetings; silly users).

Can't Entity groups be used to ensure consistency?
http://code.google.com/appengine/docs/java/datastore/hr/overview.html

Put all a users meeting in a single group (based of the User Entity).
Then all the meetings will be shown

Not possible.  Meetings are associated with boards, so they could be parented to the board, but they'd still not necessarily show up on the views that span boards (upcoming meetings for the town, on the home page, for example).  And boards can span towns (regional agencies), so parenting everything to the town doesn't work either.

I could just parent everything to a dummy root, but that would throw transaction collision errors if two users are doing updates on the system at the same time.

Also, changing to a new ancestor scheme will make the migration of the data just that much more complex.  I'm already going to have to deal with the fact that google has search results that include keys, and therefore I'm going to have to fixup every key in every handler to make sure it's got the right app-id in it.

Alternativly
A trick I used a long time ago, to deal with replication lag between
mysql servers. When someone added a new item, store it in the session
(as well as writing to master). then when run the query against the
slave (which might be stale) can just tack the saved item on the end.
Within appengine would use memcache. Not sure if its an appropriate
fix for this issue on appengine, but it worked just fine for me then.
(and is very little code)

Yeah, that's what the GAE docs recommend, and is exactly the "huge PITA" I was referring to.  (It's a huge PITA because I'm going to have to go look at every query and figure out what weird things the user might see, and how to fix it.)

Or another option that I'm seriously considering is to simply warn the user whenever they update or post a meeting that this change may not appear in lists for a while.

As was pointed out on another thread: GAE is encouraging us to ruin the user experience in many subtle ways these days…  It's like John Sculley is back there pulling the strings or something...

-Joshua

[Barry Hunter]

On Fri, Sep 2, 2011 at 5:46 PM, Joshua Smith wrote:

>  Switching to HR is going to be a huge PITA, because my use case is exactly the one that "eventual consistency" screws up (user posts a meeting, then expects to see it in the list of meetings; silly users).

Can't Entity groups be used to ensure consistency?
http://code.google.com/appengine/docs/java/datastore/hr/overview.html

Put all a users meeting in a single group (based of the User Entity).
Then all the meetings will be shown

Alternativly

A trick I used a long time ago, to deal with replication lag between
mysql servers. When someone added a new item, store it in the session
(as well as writing to master). then when run the query against the
slave (which might be stale) can just tack the saved item on the end.
Within appengine would use memcache. Not sure if its an appropriate
fix for this issue on appengine, but it worked just fine for me then.
(and is very little code)

>

[peterk]

I think this doesn't strictly relate to the OP's specific questions,
but I just wanted to brain dump some strategies I'm thinking of for my
next app to minimise front end instances.

Requests to my app are going to be doing one of two things, more or
less. Modify/Create data or read data.

The first type of request, the write will be put on a task queue and
it'll return immediately.

The queue will be handled by a backend instance. Now this costs a bit
upfront, but the key advantage is that I basically get a lot of
control over the rate at which my 'write queue' is processed.
AppEngine won't spin up new front end instances to handle this for me.
I get control over how fast and at how much cost my writes will be
handled. There may be more of a lag before writes manifest in the
datastore, but I think I can tolerate that in my app.

For read requests, I hope to use lots of caching. App caching, then
memcache. But also I'm thinking it might be possible to use the edge
cache to cut out a proportion of requests altogether. Let's say I have
a collection of data at /user/1221/activityfeed. When a (ajax) client
accesses that it'll either hit a edge cache (and not worry my
instances at all), or if it does worry my instances, will hopefully
predominantly get fetched out of cache.

My only worry is about stale data in the edge cache. But I'm thinking
you could turn the read into a two step process. When the activity
feed is updated it would set a new cache buster for the feed (in app/
memcache/datastore). So when someone comes to read it, they ask for
the URL to read from first (which will include the cache buster). Then
they ask for the URL directly, which hopefully the edge cache will
handle if someone's already hit that URL once before.

You might wonder why bother with that first step when it could return
the activity feed directly, from cache even perhaps - but I figure a
lot more cachebusters will fit in memcache/app-cache than full
activity feeds, and will be much faster to access (and thus tie up an
instance for less time). If the proxy cache holds up, you could read a
full feed from it (for free) for the cost of the fetch of a small
string from app/memcache. This could be particularly good for shared
collections - data being read by multiple users.

Still brewing over all this, but right now I think that's how I'm
going to approach my next app. Cut out as many (large) read requests
at the proxy cache as I can, throttle writes using a backend setup I
can control the cost of.

Still though - a little worried about Google's 'quietness' on the
somewhat undocumented edge cache feature!

On Sep 2, 5:59 pm, Barry Hunter <barrybhun...@gmail.com> wrote:
> On Fri, Sep 2, 2011 at 5:46 PM, Joshua Smith  wrote:
> >   Switching to HR is going to be a huge PITA, because my use case is exactly the one that "eventual consistency" screws up (user posts a meeting, then expects to see it in the list of meetings; silly users).
>

> Can't Entity groups be used to ensure consistency?http://code.google.com/appengine/docs/java/datastore/hr/overview.html

> >> For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.