My expirencie with python 2.7 and others

[Felippe Bueno]

Hi,

I will try to be objective:

The problems that I found:

1 - Seems that datetime.datetime.strptime is not threadsafe (https://groups.google.com/group/google-appengine-python/browse_thread/thread/efbcffa181c32f33/48afebe467861674?pli=1)

I needed to put a call to it on appengine_config.py following  http://bugs.python.org/issue7980

2 - webapp2 follow PEP333, it seems that's why you need to specify str().encode('ascii') when sending response headers.

I mean, if you do not specify it, str() will be the default coding  (utf8 in my case).

3 - I'm not sure if before python2.7/webapp2, it was ok send ',' on cookie header. But, it was working on python2.5 and webapp.

With python2.7 and webapp2 it stoped working. I replace it with '_' on cookie header.

Now, the real problems:

On python 2.5:

most requests to my app uses only memcache.incr()/get()/set(). Some times the only thing that I need to do is an incr().

I'm using a global dict to cache things, and it was working pretty well.

My instances latency is something from 120ms to 200ms.

incr() is running ~5ms, same to get/set. Not more than 20ms

My app is CPU intensive, and is consuming ~180 CPU seconds/sec running at 1300  req/sec

When I turn on appstats I'm able to see that my code (between the rpc calls) runs from 80ms to 120ms, some times <20ms.

After changing to python 2.7 I noticed that inc()/get()/set() (memcache) takes 100ms+  to run.

My code takes 1000ms+  to run.

Cpu seconds is 350 CPU seconds/sec running at 1300  req/sec.

Instances latency increased to ~500ms

In my opinion http://code.google.com/p/googleappengine/issues/detail?id=6323 should be reopened. 

It's true that the latency is  much better before issue 6323 be closed, but still not acceptable.

Also, it's true that when using python 2.7   threadsafe=yes, the scheduler starts half of instances when running on python 2.5.

But again, the latency is too high.

By the way, I still not comfortable with the instance scheduler.

Another thing that I noticed, is that when I change from python2.7 to python2.5 (appengine admin/versions), I get strange memcache errors:

class Reference has no attribute '__new__'

Traceback (most recent call last):

  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__

    handler.get(*groups)

  File "/base/data/home/apps/xxxxxxx/xxxxxxxxx/xxxxxxxviews.py", line 29, in get

    self._get()

  File "/base/data/home/apps/xxxxxxx/xxxxxxxxx/xxxxxxx/views.py", line 83, in _get

    output = rules.Process(nid, h, data, r, self.request)

  File "/base/data/home/apps/xxxxxxx/xxxxxxxxx/xxxxxxx/rules.py", line 73, in Process

    match, location_info = geoip.GeoIP_Org.match(params, rule, geoip_ck)

  File "/base/data/home/apps/xxxxxxx/xxxxxxxxx/xxxxxxx/matchers/geoip.py", line 101, in match

    return klass.compute_match(params, rule, cookie, storageclass=models.GeoIPOrg)

  File "/base/data/home/apps/xxxxxxx/xxxxxxxxx/xxxxxxx/matchers/geoip.py", line 38, in compute_match

    geoip_row = klass.fetch_row(storageclass, params)

  File "/base/data/home/apps/xxxxxxx/xxxxxxxxx/xxxxxxx/matchers/geoip.py", line 68, in fetch_row

    cached_row = memcache.get(klass._row_key(addr))

  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 563, in get

    results = rpc.get_result()

  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result

    return self.__get_result_hook(self)

  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 628, in __get_hook

    self._do_unpickle)

  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 271, in _decode_value

    return do_unpickle(value)

  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 404, in _do_unpickle

    return unpickler.load()

  File "/base/python_runtime/python_dist/lib/python2.5/pickle.py", line 852, in load

    dispatch[key](self)

  File "/base/python_runtime/python_dist/lib/python2.5/pickle.py", line 1077, in load_newobj

    obj = cls.__new__(cls, *args)

AttributeError: class Reference has no attribute '__new__'

I'm also trying to contact premier account sales team, but without success.

It is very frustrating when you don't have to who ask for help.

I know that there are a lot of good souls here :-) but most of time people are busy trying to solve their own problems.

Making long story short,

I need to choose between, save money (half of instances) or run fast. That should make sense, but in my opinion, 

python 2.7 should comes to save us money, running at same speed of python 2.5.

Am I wrong ?

Thank you for "listening me" :-)

Cheers

[Brandon Wirtz]

>I need to choose between, save money (half of instances) or run fast. That should make sense, but in my opinion, 

>python 2.7 should comes to save us money, running at same speed of python 2.5.

>Am I wrong ?

 

Yes you are wrong.  Thread safe only saves you money if you are waiting on API’s enough to save money.

 

If you look at your Dashboard and 85% of your CPU usage is API’s you will likely go faster and save money with Thread safe.

if you are 85% CPU, unless you always have 10 instances you may save a little money but you will go slower.

 

If you are at 50% API’s and always have 20 instances, thread safe will make you a lot faster and cut your bill in half.

 

 

[Felippe Bueno]

Thanks for your answer Brandon.

Let me see if I understand.

I'm like only 0.1% API calls CPU.

You are saying that threadsafe is not a good idea to me ?

I mean, instances were cuted to half, but latency was the double of python 2.5.

Thanks again.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Brandon Wirtz]

You only get the benefit of threads if you have unused CPU cycles. If you are only spending .1% of the time waiting on API’s than you don’t have any “Spare” CPU to start on the next task.

 

Your app will perform as you said, instances will queue up and the first request will take the same amount of time as it did before, the second request will take twice as long, and so your average will be 50% longer than it was on 2.5.

You might test F2 instances with thread safe. In some cases we have found these to be Significantly faster for Request 1.

 

With F1 and thread safe you would see

 

100Ms for 1^st request
200MS (100+100) for 2^nd request.
Average 150ms

With F1 With out thread safe

2 instances would handle requests in 95ms

 

F2 With Thread Safe

50MS for 1^st Request
100MS for 2^nd Request
Average 75MS

 

This F2 Time is As a Result Faster, and The same Price if not Cheaper than the F1’s with out thread safe.

[Brandon Wirtz]

Oh, and I know Goog isn’t in the business of saving us all money, but using the data in the dashboard, they could build a really great Cost optimizer based on the way your CPU usage looks.

[Kenneth]

Where do you find your API %?

[Brandon Wirtz]

Dashboard, CPU Seconds Used/Second Compare the Blue to the orange.

[Kenneth]

Right, so if they're pretty much on top of each other then python 2.7 multithreading is the way to go.  Cool.

[PK]

Hi Felipe,

+1

my experience is similar with yours. I just switched last Sunday and the latency in python27 is much worse than before, and really not acceptable in my case. I would like to write a lot more and hopefully I will get some time in the next few days but I am currently overwhelmed trying to optimize the site for python27 and calm down customers who call and ask why things are slow. From Appstats it seems that long sequences of db.get() are much slower in python27

The good news is that the rest seems to work pretty much fine both on SDK and production. I look forward to seeing further latency improvements from the Google Team.

Thanks for sharing your experience.

PK

[Daniel]

Hey Brandon.  I'm not clear how to read that graph.  Here's what I'm seeing on my app, how do I interpret this and know if 2.7 is might help me?

[Brandon Wirtz]

Most of your CPU is API calls. So moving to thread safe should help you a lot.  Orange is your API CPU usage, and Blue is your total.  So about 75% of your CPU is API, implying that your app does a lot of waiting on API’s…

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Daniel
Sent: Saturday, March 10, 2012 1:12 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] My expirencie with python 2.7 and others

 

Hey Brandon.  I'm not clear how to read that graph.  Here's what I'm seeing on my app, how do I interpret this and know if 2.7 is might help me?

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/63jINNx8o7cJ.

[Daniel Kramer]

Got it.  I've flipped my app over to 2.7 thread safe.. we'll see how it goes, so far seems to be running without error.  Early indications on the CPU seconds Use/Seconds is still showing a similar ratio between Total CPU and API Calls CPU.. shouldn't I expect those lines to come closer together?

[Brandon Wirtz]

Nope. You are still doing X number of API Cycles for every Y number of CPU cycles. Your Application Profile didn’t change.  

Where you should see the difference Is in number of instances used, and pending latency. (and you app might feel faster)

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Daniel Kramer
Sent: Saturday, March 10, 2012 9:37 PM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] My expirencie with python 2.7 and others

 

Got it.  I've flipped my app over to 2.7 thread safe.. we'll see how it goes, so far seems to be running without error.  Early indications on the CPU seconds Use/Seconds is still showing a similar ratio between Total CPU and API Calls CPU.. shouldn't I expect those lines to come closer together?

 

On Mar 10, 2012, at 9:52 AM, Brandon Wirtz wrote:

Most of your CPU is API calls. So moving to thread safe should help you a lot.  Orange is your API CPU usage, and Blue is your total.  So about 75% of your CPU is API, implying that your app does a lot of waiting on API’s…

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Daniel
Sent: Saturday, March 10, 2012 1:12 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] My expirencie with python 2.7 and others

 

Hey Brandon.  I'm not clear how to read that graph.  Here's what I'm seeing on my app, how do I interpret this and know if 2.7 is might help me?

 

 

 

On Thursday, March 8, 2012 11:36:21 PM UTC-8, Brandon Wirtz wrote:

Dashboard, CPU Seconds Used/Second Compare the Blue to the orange.

-- 
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/63jINNx8o7cJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Chris Ramsdale]

Hey Felippe,

Have you considered bumping your instance class and enabling threadsafe?

We have seen a handful of customers that have had success doing just this.

Also, did you hear back from the premier account folks? If not, send me your app IDs

(offline if you'd like) and I'll get the ball rolling here.

-- Chris

Product Manager, Google App Engine

[Felippe Bueno]

Hi Chris,

I just sent you a private message with the appids.

I tryed to change my instances class to F2 and F4, and I didn't noticed any significant improvements.

I didn't gave up yet. Thanks for your answer.

Brandon,

Thanks for the explanation, things are clearer now.

Cheers

[Daniel]

I see.  I feel stupid, just realized I was not running python27 like I thought as the app never deployed.  I thought python27 was available to everyone but I haven't migrated to HRD. I didn't' realize that was still a requirement.  I haven't migrated as I assumed it would cost me more $$ to run.  I supposed it's possible to migrate, run python27 and bring costs down depending on the app.  Unfortunately it looks like it's not easy to migrate, try it out and switch back if it doesn't work out.. looks like a one way street

On Sunday, March 11, 2012 4:22:38 PM UTC-7, Brandon Wirtz wrote:

Nope. You are still doing X number of API Cycles for every Y number of CPU cycles. Your Application Profile didn’t change.  

Where you should see the difference Is in number of instances used, and pending latency. (and you app might feel faster)

 

 

From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Daniel Kramer
Sent: Saturday, March 10, 2012 9:37 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] My expirencie with python 2.7 and others

 

Got it.  I've flipped my app over to 2.7 thread safe.. we'll see how it goes, so far seems to be running without error.  Early indications on the CPU seconds Use/Seconds is still showing a similar ratio between Total CPU and API Calls CPU.. shouldn't I expect those lines to come closer together?

 

On Mar 10, 2012, at 9:52 AM, Brandon Wirtz wrote:

Most of your CPU is API calls. So moving to thread safe should help you a lot.  Orange is your API CPU usage, and Blue is your total.  So about 75% of your CPU is API, implying that your app does a lot of waiting on API’s…

 

From: google-appengine@googlegroups.com [mailto:google-app...@googlegroups.com] On Behalf Of Daniel
Sent: Saturday, March 10, 2012 1:12 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] My expirencie with python 2.7 and others

 

Hey Brandon.  I'm not clear how to read that graph.  Here's what I'm seeing on my app, how do I interpret this and know if 2.7 is might help me?

 

 

 

On Thursday, March 8, 2012 11:36:21 PM UTC-8, Brandon Wirtz wrote:

Dashboard, CPU Seconds Used/Second Compare the Blue to the orange.

-- 
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/63jINNx8o7cJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsub...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsub...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Greg]

On Mar 13, 5:39 pm, Daniel <danielkra...@gmail.com> wrote:
> down depending on the app.  Unfortunately it looks like it's not easy to
> migrate, try it out and switch back if it doesn't work out.. looks like a
> one way street

True, but I STRONGLY recommend you migrate to HR anyway. That doesn't
cost you anything extra (well, possibly a tiny bit of extra
processing) and gives you freedom from the continuing planned outages.
Anyone with a production app still on MS is crazy. You then have the
freedom to switch to 2.7 threadsafe and back again if you want to.

Cheers
Greg.

[Daniel]

Hmm.. Sounds like I have bad info.  I thought HRD cost more money.. sounds like under the new pricing model it's the same.   I don't remember Google advertising that.  Anyway, great news I'll start the migration