Switched to HRD - So far not impressed

[Daniel]

I've been running a server for my game SkyWords on AppEngine for a little over a year now and never switched over to an HRD server.  I was under the impression it cost more but recently learned it doesn't.. so I made the switch. 

Prior to switching I had my server running really smoothly, most days I'd get zero server errors, some days a small handful.  I swtiched in hopes to leverage off python27 and possibly bring the cost of my server down.  Since switching I'm getting a lot (hundreds) or errors a day now. 

First I made the switch to python27 and found I was getting multiple server errors (mostly DeadlineExceededErrors) so I switched back to python25.  I'm still getting multiple (hundreds) of DeadlineExceededErrors using python25... Have others found this? could the servers be slower even when using python25 causing me to bump up against deadline errors?

I'm also finding some of my cron jobs are failing on my backend server.  The logs are showing errors with references to python27, it appears my backend is running python27, not python25?  I made sure to kill and recreate the backend after double checking my app is set to python25 and still seeing the python27 errors.  How do I ensure the backend server is also running python25?

I'm really bummed, I can't go back and now I've got to spend serval nights debugging what I expected to be a smooth transition. Any advice is appreciated

Daniel

[Brandon Wirtz]

Did you skip the day in 4^th grade they taught scientific method?

You don’t change 6 variables at once. Move to HR.

Test.

Move to Python 27

Test.
Move to Thread Safe

Test.

 

Somewhere in each of those you might want to revise code.  And tweak the scheduler settings. You might want to have more than once version of your app so you can test 2 things at once, like scheduler and Thread safe.

 

[Daniel Kramer]

Yes, I get it. I was anxious to try python27 and jumped the gun early.  But I am back to just the HRD switch without python27/threadSafe.  I was not surprised to see issue with python27, I didn't expect the errors with just switching to HRD.  I also didn't expect to see my backend server appearing to run python27 when my app is set to python25

-- 
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Brandon Wirtz]

Not sure on the python27 when 25 expected.  Your Timeouts are the scheduler. There are a couple of tuning guides out there (a few by me even) for avoiding DEE’s

 

Likely getting the scheduler reeled in will make it so you can jump back to 27 Threadsafe.

[Daniel Kramer]

I think I have the backend sorted out.  I did a quick search for a scheduler guide by you but came up short.

--------------------
Daniel Kramer

[vlad]

Daniel keep in mind that you data is still co-hosted with dozens or hundreds on other apps on the same physical node. That is what makes performance volatile not M/S vs HRD. HRD did solve the problem for writes somewhat but not for reads. Fundamentally if an app which runs on the same node is really active *your* app's performance will suffer. So it is dump luck. Since you got a new appid when migrating you got new "neighbors" so to speak - apps which share bigtable node with your app.

[Daniel Kramer]

 Interesting, I hadn't though about it that way.  Thanks for the info

On Mar 16, 2012, at 5:54 PM, vlad wrote:

Daniel keep in mind that you data is still co-hosted with dozens or hundreds on other apps on the same physical node. That is what makes performance volatile not M/S vs HRD. HRD did solve the problem for writes somewhat but not for reads. Fundamentally if an app which runs on the same node is really active *your* app's performance will suffer. So it is dump luck. Since you got a new appid when migrating you got new "neighbors" so to speak - apps which share bigtable node with your app.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/7NEAU3WOrG8J.

[Waleed Abdulla]

I thought HRD was supposed to solve the bad neighbor problem. As I read somewhere (don't have the reference handy right now), when your app issues a read request, it's sent to multiple datastore replicas in parallel and the app takes the data from whichever datastore replies first. 

[vlad]

I did not get an impression that HRD addressed the root cause of bad neighbor problem in any way. That is not to say it is bad because it improved data  availability. It is the price hike that was key to reducing bad neighbors. Especially, the clamp down on free quota. That was clearly targeted toward all those half baked apps, many even  abandoned, which keep eating GAE resources.

[Jeff Schnitzer]

On Sat, Mar 17, 2012 at 6:27 PM, Waleed Abdulla <wal...@ninua.com> wrote:
> I thought HRD was supposed to solve the bad neighbor problem. As I read
> somewhere (don't have the reference handy right now), when your app issues a
> read request, it's sent to multiple datastore replicas in parallel and the
> app takes the data from whichever datastore replies first.

I need to re-watch the "more 9s please" video to be sure, but I
believe "first reply wins" only works when you are using eventual
consistency mode - ie, when issuing a query on an indexed property
(always eventual) or doing a get-by-key in Consistency.EVENTUAL mode.
If you do a get-by-key in Consistency.STRONG mode (the default), you
will still need a quorum.

Jeff

[Brandon Wirtz]

> If you do a get-by-key in Consistency.STRONG mode (the default), you will
still need a quorum.

Oooh I didn't know/think about this. I don't need strong. I bet I can
switch modes and save money. I'm not sending you any of that saved money.
But I'll buy you a drink in Phoenix :-)

(unless you are coming to work for me in PHX then I'll give you some of that
money)

[Daniel]

Just to follow up.  It appears my app has settled down and appears to be running once again at around the same speed/price as prior to the HRD switch.  There was one error that kept cropping up which a small code change fixed.  It's odd as I never saw that error prior to the switch and I'm on python25 currently.  In any case I think I'm in a good spot.  I'll let this run for a few days before playing with python27.

[Alfred Fuller]

Most of the time 'first reply' works even when using strong consistency because we cache the state of entity groups in all replicas. Though to read from a replica, it has to be up to date. So setting eventual consistency will help you if an entity group is written to a lot (likely not up to date) or rarely read (cache miss).